Repository: argoproj/argo-workflows
Pull request #14467: fix: correctly release all mutexes when multiple mutexes are held. FIxes #14461
Author: login: atgane
Labels: name: area/mutex-semaphore
Created: 2025-05-11T04:03:35Z | Merged: 2025-05-22T08:29:03Z

<!-- markdownlint-disable MD041 -- this is rendered within existing HTML, so allow starting without an H1 -->

<!--

### Before you open your PR

- Run `make pre-commit -B` to fix codegen and lint problems (build will fail).
- [Signed-off your commits](https://github.com/apps/dco/) (otherwise the DCO check will fail).
- Used [a conventional commit message](https://www.conventionalcommits.org/en/v1.0.0/).

### When you open your PR

- PR title format should also conform to [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/).
- "Fixes #" is in both the PR title (for release notes) and this description (to automatically link and close the issue).
- Create the PR as draft.
- Once builds are green, mark your PR "Ready for review".

When changes are requested, please address them and then dismiss the review to get it reviewed again.

-->

<!-- Does this PR fix an issue -->

Fix #14461

### Motivation

Fix the issue where some mutexes are not properly released when multiple mutexes are used in a workflow. This ensures all held mutexes are released correctly after execution.

### Modifications

The issue occurred because the logic inside the loop modified the original `MutexHolding` array while iterating over it. To fix this, we create a copy of the `MutexHolding` array and operate on the copy instead. This guarantees that all mutexes are properly released.

### Verification

Added a test case named `WorkflowLevelMutexAcquireAndReleaseWithMultipleMutex` in `sync_manager_test.go` to verify that all mutexes are correctly acquired and released.

### Documentation

No documentation needed

<!--
### Beyond this PR

Thank you for submitting this! Have you ever thought of becoming a Reviewer or Approver on the project?

Argo Workflows is seeking more community involvement and ultimately more [Reviewers and Approvers](https://github.com/argoproj/argoproj/blob/main/community/membership.md) to help keep it viable.
See [Sustainability Effort](https://github.com/argoproj/argo-workflows/blob/main/community/sustainability_effort.md) for more information.

-->


Linked issue #14461: The controller couldn't release the lock correct if the workflow uses multiple mutexes
### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

I have a workflow with four mutexes, lockA,lockB,lockC,lockD. Not all locks were released when the workflow finished with Succeeded status.

```
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: helloworld
  namespace: infra
spec:
  entrypoint: hello
  serviceAccountName: argo-workflow
  synchronization:
    mutexes:
    - name: lockA
    - name: lockB
    - name: lockC
    - name: lockD
  templates:
    - name: hello               # The first "template" in this Workflow, it is referenced by "entrypoint"
      steps:                    # The type of this "template" is "steps"
      - - name: hello
          template: print-message # We reference our second "template" here
          arguments:
            parameters: [{name: message, value: "hello"}]

    - name: print-message       # The second "template" in this Workflow, it is referenced by "hello"
      inputs:
        parameters:
        - name: message
      container:                # The type of this "template" is "container"
        image: busybox
        command: [echo]
        args: ["{{inputs.parameters.message}}"]
```

The following log, was queried from the controller, shows only infra/Mutex/lockA,infra/Mutex/lockC,infra/Mutex/lockD released

```
{"Phase":"","ResourceVersion":"233665594","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:54:57.207Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Workflow processing has been postponed due to concurrency limit","namespace":"infra","time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated phase  -\u003e Pending","namespace":"infra","time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated message  -\u003e Waiting for infra/Mutex/lockB lock. Lock status: 0/1","namespace":"infra","time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Pending","resourceVersion":"233665595","time":"2025-05-08T04:54:57.237Z","workflow":"helloworld94kfj"}
{"Phase":"Pending","ResourceVersion":"233665595","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:54:57.237Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:54:57.238Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Workflow processing has been postponed due to concurrency limit","namespace":"infra","time":"2025-05-08T04:54:57.238Z","workflow":"helloworld94kfj"}
{"Phase":"Pending","ResourceVersion":"233665595","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:07.237Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:07.238Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Workflow processing has been postponed due to concurrency limit","namespace":"infra","time":"2025-05-08T04:55:07.238Z","workflow":"helloworld94kfj"}
{"Phase":"Pending","ResourceVersion":"233665595","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:18.472Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:18.472Z","workflow":"helloworld94kfj"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockA acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockA","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockB acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockB","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockC acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockC","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockD acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockD","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","msg":"Updated phase Pending -\u003e Running","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated message Waiting for infra/Mutex/lockB lock. Lock status: 0/1 -\u003e ","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Steps node helloworld94kfj initialized Running","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"StepGroup node helloworld94kfj-498103269 initialized Running","namespace":"infra","time":"2025-05-08T04:55:18.474Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"infra","time":"2025-05-08T04:55:18.474Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Pod node helloworld94kfj-3188636799 initialized Pending","namespace":"infra","time":"2025-05-08T04:55:18.474Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Created pod: helloworld94kfj[0].hello (helloworld94kfj-print-message-3188636799)","namespace":"infra","time":"2025-05-08T04:55:18.657Z","workflow": helloworld94kfj"}
{"level":"info","msg":"Workflow step group node helloworld94kfj-498103269 not yet completed","namespace":"infra","time":"2025-05-08T04:55:18.657Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:18.657Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:18.658Z","workflow":"helloworld94kfj"}
time="2025-05-08T04:55:18Z" level=info msg="add pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:18Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:18Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Running","resourceVersion":"233666013","time":"2025-05-08T04:55:18.707Z","workflow":"helloworld94kfj"}
{"Phase":"Running","ResourceVersion":"233666013","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:18.708Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:18.708Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow step group node helloworld94kfj-498103269 not yet completed","namespace":"infra","time":"2025-05-08T04:55:18.709Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:18.709Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:18.709Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Running","resourceVersion":"233666015","time":"2025-05-08T04:55:18.754Z","workflow":"helloworld94kfj"}
{"Phase":"Running","ResourceVersion":"233666015","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:18.756Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:18.756Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow step group node helloworld94kfj-498103269 not yet completed","namespace":"infra","time":"2025-05-08T04:55:18.757Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:18.757Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:18.757Z","workflow":"helloworld94kfj"}
time="2025-05-08T04:55:20Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:21Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:24Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:26Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
{"Phase":"Running","ResourceVersion":"233666015","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:28.659Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":1,"time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Step group node helloworld94kfj-498103269 successful","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"node helloworld94kfj-498103269 phase Running -\u003e Succeeded","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"node helloworld94kfj-498103269 finished: 2025-05-08 04:55:28.660924708 +0000 UTC","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow": helloworld94kfj"}
{"level":"info","msg":"Outbound nodes of helloworld94kfj-3188636799 is [helloworld94kfj-3188636799]","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Outbound nodes of helloworld94kfj is [helloworld94kfj-3188636799]","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kf "}
{"level":"info","msg":"node helloworld94kfj phase Running -\u003e Succeeded","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"node helloworld94kfj finished: 2025-05-08 04:55:28.661075846 +0000 UTC","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Marking workflow completed","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","lockType":"mutex","msg":"Lock has been released by infra/helloworld94kfj. Available locks: 1","name":"infra/Mutex/lockA","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockA","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","lockType":"mutex","msg":"Lock has been released by infra/helloworld94kfj. Available locks: 1","name":"infra/Mutex/lockC","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockC","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","lockType":"mutex","msg":"Lock has been released by infra/helloworld94kfj. Available locks: 1","name":"infra/Mutex/lockD","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.661Z"}
{"key":"helloworld94kfj","level":"info","msg":"Released all acquired locks","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Succeeded","resourceVersion":"233666210","time":"2025-05-08T04:55:28.682Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockA","time":"2025-05-08T04:55:28.708Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockC","time":"2025-05-08T04:55:28.708Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.708Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.708Z"}
time="2025-05-08T04:55:28Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=infra podName=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:28Z" level=info msg="cleaning up pod" action=labelPodCompleted key=infra/helloworld94kfj-print-message-3188636799/labelPodCompleted namespace=infra podName=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:28Z" level=info msg="delete pod event" pod=helloworld94kfj-print-message-3188636799
```

### Version(s)

v3.6.4,f6680852be3ba4cc3179444594eb35ad7a40277e236cafdb3cc7ef93eb2e0ef8

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: helloworld
  namespace: infra
spec:
  entrypoint: hello
  serviceAccountName: argo-workflow
  synchronization:
    mutexes:
    - name: lockA
    - name: lockB
    - name: lockC
    - name: lockD
  templates:
    - name: hello               # The first "template" in this Workflow, it is referenced by "entrypoint"
      steps:                    # The type of this "template" is "steps"
      - - name: hello
          template: print-message # We reference our second "template" here
          arguments:
            parameters: [{name: message, value: "hello"}]

    - name: print-message       # The second "template" in this Workflow, it is referenced by "hello"
      inputs:
        parameters:
        - name: message
      container:                # The type of this "template" is "container"
        image: busybox
        command: [echo]
        args: ["{{inputs.parameters.message}}"]
```

### Logs from the workflow controller

```text
{"Phase":"","ResourceVersion":"233665594","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:54:57.207Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Workflow processing has been postponed due to concurrency limit","namespace":"infra","time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated phase  -\u003e Pending","namespace":"infra","time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated message  -\u003e Waiting for infra/Mutex/lockB lock. Lock status: 0/1","namespace":"infra","time":"2025-05-08T04:54:57.218Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Pending","resourceVersion":"233665595","time":"2025-05-08T04:54:57.237Z","workflow":"helloworld94kfj"}
{"Phase":"Pending","ResourceVersion":"233665595","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:54:57.237Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:54:57.238Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Workflow processing has been postponed due to concurrency limit","namespace":"infra","time":"2025-05-08T04:54:57.238Z","workflow":"helloworld94kfj"}
{"Phase":"Pending","ResourceVersion":"233665595","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:07.237Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:07.238Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Workflow processing has been postponed due to concurrency limit","namespace":"infra","time":"2025-05-08T04:55:07.238Z","workflow":"helloworld94kfj"}
{"Phase":"Pending","ResourceVersion":"233665595","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:18.472Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:18.472Z","workflow":"helloworld94kfj"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockA acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockA","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockB acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockB","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockC acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockC","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","lockType":"mutex","msg":"infra/Mutex/lockD acquired by infra/helloworld94kfj. Lock availability: 0/1","name":"infra/Mutex/lockD","time":"2025-05-08T04:55:18.473Z"}
{"level":"info","msg":"Updated phase Pending -\u003e Running","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated message Waiting for infra/Mutex/lockB lock. Lock status: 0/1 -\u003e ","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Steps node helloworld94kfj initialized Running","namespace":"infra","time":"2025-05-08T04:55:18.473Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"StepGroup node helloworld94kfj-498103269 initialized Running","namespace":"infra","time":"2025-05-08T04:55:18.474Z","workflow":"helloworld94kfj"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"infra","time":"2025-05-08T04:55:18.474Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Pod node helloworld94kfj-3188636799 initialized Pending","namespace":"infra","time":"2025-05-08T04:55:18.474Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Created pod: helloworld94kfj[0].hello (helloworld94kfj-print-message-3188636799)","namespace":"infra","time":"2025-05-08T04:55:18.657Z","workflow": helloworld94kfj"}
{"level":"info","msg":"Workflow step group node helloworld94kfj-498103269 not yet completed","namespace":"infra","time":"2025-05-08T04:55:18.657Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:18.657Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:18.658Z","workflow":"helloworld94kfj"}
time="2025-05-08T04:55:18Z" level=info msg="add pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:18Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:18Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Running","resourceVersion":"233666013","time":"2025-05-08T04:55:18.707Z","workflow":"helloworld94kfj"}
{"Phase":"Running","ResourceVersion":"233666013","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:18.708Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:18.708Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow step group node helloworld94kfj-498103269 not yet completed","namespace":"infra","time":"2025-05-08T04:55:18.709Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:18.709Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:18.709Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Running","resourceVersion":"233666015","time":"2025-05-08T04:55:18.754Z","workflow":"helloworld94kfj"}
{"Phase":"Running","ResourceVersion":"233666015","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:18.756Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":0,"time":"2025-05-08T04:55:18.756Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow step group node helloworld94kfj-498103269 not yet completed","namespace":"infra","time":"2025-05-08T04:55:18.757Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:18.757Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:18.757Z","workflow":"helloworld94kfj"}
time="2025-05-08T04:55:20Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:21Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:24Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:26Z" level=info msg="update pod event" pod=helloworld94kfj-print-message-3188636799
{"Phase":"Running","ResourceVersion":"233666015","level":"info","msg":"Processing workflow","namespace":"infra","time":"2025-05-08T04:55:28.659Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Task-result reconciliation","namespace":"infra","numObjs":1,"time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Step group node helloworld94kfj-498103269 successful","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"node helloworld94kfj-498103269 phase Running -\u003e Succeeded","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"node helloworld94kfj-498103269 finished: 2025-05-08 04:55:28.660924708 +0000 UTC","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow": helloworld94kfj"}
{"level":"info","msg":"Outbound nodes of helloworld94kfj-3188636799 is [helloworld94kfj-3188636799]","namespace":"infra","time":"2025-05-08T04:55:28.660Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Outbound nodes of helloworld94kfj is [helloworld94kfj-3188636799]","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kf "}
{"level":"info","msg":"node helloworld94kfj phase Running -\u003e Succeeded","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"node helloworld94kfj finished: 2025-05-08 04:55:28.661075846 +0000 UTC","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"reconcileAgentPod","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Marking workflow completed","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","lockType":"mutex","msg":"Lock has been released by infra/helloworld94kfj. Available locks: 1","name":"infra/Mutex/lockA","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockA","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","lockType":"mutex","msg":"Lock has been released by infra/helloworld94kfj. Available locks: 1","name":"infra/Mutex/lockC","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockC","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","lockType":"mutex","msg":"Lock has been released by infra/helloworld94kfj. Available locks: 1","name":"infra/Mutex/lockD","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.661Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.661Z"}
{"key":"helloworld94kfj","level":"info","msg":"Released all acquired locks","namespace":"infra","time":"2025-05-08T04:55:28.661Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"Workflow update successful","namespace":"infra","phase":"Succeeded","resourceVersion":"233666210","time":"2025-05-08T04:55:28.682Z","workflow":"helloworld94kfj"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockA","time":"2025-05-08T04:55:28.708Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockC","time":"2025-05-08T04:55:28.708Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.708Z"}
{"level":"info","msg":"infra/helloworld94kfj released a lock from infra/Mutex/lockD","time":"2025-05-08T04:55:28.708Z"}
time="2025-05-08T04:55:28Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=infra podName=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:28Z" level=info msg="cleaning up pod" action=labelPodCompleted key=infra/helloworld94kfj-print-message-3188636799/labelPodCompleted namespace=infra podName=helloworld94kfj-print-message-3188636799
time="2025-05-08T04:55:28Z" level=info msg="delete pod event" pod=helloworld94kfj-print-message-3188636799
```

### Logs from in your workflow's wait container

```text
the workflow is in pending status


metadata:
  creationTimestamp: "2025-05-08T04:55:50Z"
  generateName: helloworld
  generation: 2
  labels:
    workflows.argoproj.io/completed: "false"
    workflows.argoproj.io/phase: Pending
  managedFields:
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName: {}
    manager: argo
    operation: Update
    time: "2025-05-08T04:55:50Z"
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:workflows.argoproj.io/completed: {}
          f:workflows.argoproj.io/phase: {}
      f:spec: {}
      f:status: {}
    manager: workflow-controller
    operation: Update
    time: "2025-05-08T04:55:50Z"
  name: helloworldhjmgj
  namespace: infra
  resourceVersion: "233666609"
  uid: c0f2e562-c0af-4576-8ee6-c108f0f8a50b
spec:
  arguments: {}
  entrypoint: hello
  nodeSelector:
    node_pool: infra-node-pool
  serviceAccountName: argo-workflow
  synchronization:
    mutexes:
    - name: lockA
    - name: lockB
    - name: lockC
    - name: lockD
  templates:
  - inputs: {}
    metadata: {}
    name: hello
    outputs: {}
    steps:
    - - arguments:
          parameters:
          - name: message
            value: hello
        name: hello
        template: print-message
  - container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - echo
      image: busybox
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: message
    metadata: {}
    name: print-message
    outputs: {}
status:
  artifactGCStatus:
    notSpecified: true
  artifactRepositoryRef:
    artifactRepository: {}
    default: true
  finishedAt: null
  message: 'Waiting for infra/Mutex/lockB lock. Lock status: 0/1'
  phase: Pending
  progress: 0/0
  startedAt: null
  synchronization:
    mutex:
      waiting:
      - holder: infra/helloworld94kfj
        mutex: infra/Mutex/lockB
```
Linked issue comments: user: login: atgane
created_at: 2025-05-08T16:15:16Z
body: I would like to handle this issue.
author_association: CONTRIBUTOR

Diff:
diff --git a/workflow/sync/sync_manager.go b/workflow/sync/sync_manager.go
index ad6df92e5c21..930bcf6eefa3 100644
--- a/workflow/sync/sync_manager.go
+++ b/workflow/sync/sync_manager.go
@@ -524,7 +524,9 @@ func (sm *Manager) ReleaseAll(wf *wfv1.Workflow) bool {
 	}
 
 	if wf.Status.Synchronization.Mutex != nil {
-		for _, holding := range wf.Status.Synchronization.Mutex.Holding {
+		h := make([]wfv1.MutexHolding, len(wf.Status.Synchronization.Mutex.Holding))
+		copy(h, wf.Status.Synchronization.Mutex.Holding)
+		for _, holding := range h {
 			syncLockHolder := sm.syncLockMap[holding.Mutex]
 			if syncLockHolder == nil {
 				continue
diff --git a/workflow/sync/sync_manager_test.go b/workflow/sync/sync_manager_test.go
index 203ec41eb833..4a2ef001a4be 100644
--- a/workflow/sync/sync_manager_test.go
+++ b/workflow/sync/sync_manager_test.go
@@ -475,6 +475,84 @@ func TestSemaphoreWfLevel(t *testing.T) {
 		syncManager.ReleaseAll(wf3)
 		assert.Empty(t, sema.pending.items)
 	})
+
+	t.Run("WorkflowLevelSemaphoreAcquireAndReleaseWithMultipleSemaphores", func(t *testing.T) {
+		// Create ConfigMap with multiple semaphore limits
+		cm := v1.ConfigMap{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "multiple-sema-config",
+				Namespace: "default",
+			},
+			Data: map[string]string{
+				"sem1": "1",
+				"sem2": "1",
+				"sem3": "1",
+			},
+		}
+		_, err := kube.CoreV1().ConfigMaps("default").Create(ctx, &cm, metav1.CreateOptions{})
+		require.NoError(t, err)
+
+		syncManager := NewLockManager(ctx, kube, "", nil, syncLimitFunc, func(key string) {
+			// nextKey = key
+		}, WorkflowExistenceFunc)
+
+		// Create two workflows that both need all semaphores
+		wf1 := wfv1.MustUnmarshalWorkflow(wfWithSemaphore)
+		wf1.Name = "wf1"
+		wf1.Spec.Synchronization.Semaphores = []*wfv1.SemaphoreRef{
+			{
+				ConfigMapKeyRef: &v1.ConfigMapKeySelector{
+					LocalObjectReference: v1.LocalObjectReference{Name: "multiple-sema-config"},
+					Key:                  "sem1",
+				},
+			},
+			{
+				ConfigMapKeyRef: &v1.ConfigMapKeySelector{
+					LocalObjectReference: v1.LocalObjectReference{Name: "multiple-sema-config"},
+					Key:                  "sem2",
+				},
+			},
+			{
+				ConfigMapKeyRef: &v1.ConfigMapKeySelector{
+					LocalObjectReference: v1.LocalObjectReference{Name: "multiple-sema-config"},
+					Key:                  "sem3",
+				},
+			},
+		}
+
+		wf2 := wf1.DeepCopy()
+		wf2.Name = "wf2"
+
+		// First workflow should acquire all semaphores
+		status, wfUpdate, msg, failedLockName, err := syncManager.TryAcquire(ctx, wf1, "", wf1.Spec.Synchronization)
+		require.NoError(t, err)
+		assert.Empty(t, msg)
+		assert.Empty(t, failedLockName)
+		assert.True(t, status)
+		assert.True(t, wfUpdate)
+		require.NotNil(t, wf1.Status.Synchronization)
+		require.NotNil(t, wf1.Status.Synchronization.Semaphore)
+		require.NotNil(t, wf1.Status.Synchronization.Semaphore.Holding)
+		assert.Len(t, wf1.Status.Synchronization.Semaphore.Holding, 3)
+
+		// Release all semaphores from first workflow
+		syncManager.ReleaseAll(wf1)
+
+		// Second workflow should now be able to acquire all semaphores
+		status, wfUpdate, msg, failedLockName, err = syncManager.TryAcquire(ctx, wf2, "", wf2.Spec.Synchronization)
+		require.NoError(t, err)
+		assert.Empty(t, msg)
+		assert.Empty(t, failedLockName)
+		assert.True(t, status)
+		assert.True(t, wfUpdate)
+		require.NotNil(t, wf2.Status.Synchronization)
+		require.NotNil(t, wf2.Status.Synchronization.Semaphore)
+		require.NotNil(t, wf2.Status.Synchronization.Semaphore.Holding)
+		assert.Len(t, wf2.Status.Synchronization.Semaphore.Holding, 3)
+
+		// Clean up
+		syncManager.ReleaseAll(wf2)
+	})
 }
 
 func TestResizeSemaphoreSize(t *testing.T) {
@@ -906,6 +984,42 @@ func TestMutexWfLevel(t *testing.T) {
 		syncManager.ReleaseAll(wf2)
 		assert.Empty(t, mutex.pending.items)
 	})
+
+	t.Run("WorkflowLevelMutexAcquireAndReleaseWithMultipleMutex", func(t *testing.T) {
+		syncManager := NewLockManager(ctx, kube, "", nil, syncLimitFunc, func(key string) {
+			// nextKey = key
+		}, WorkflowExistenceFunc)
+		wf := wfv1.MustUnmarshalWorkflow(wfWithMutex)
+		mutexes := make([]*wfv1.Mutex, 0, 10)
+		for i := range 10 {
+			mutexes = append(mutexes, &wfv1.Mutex{Name: fmt.Sprintf("mutex%d", i)})
+		}
+		wf.Spec.Synchronization.Mutexes = mutexes
+		wf1 := wf.DeepCopy()
+		wf1.Name = "two"
+
+		status, wfUpdate, msg, failedLockName, err := syncManager.TryAcquire(ctx, wf, "", wf.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf.Status.Synchronization)
+		require.NotNil(t, wf.Status.Synchronization.Mutex)
+		require.NotNil(t, wf.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf)
+
+		status, wfUpdate, msg, failedLockName, err = syncManager.TryAcquire(ctx, wf1, "", wf1.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf1.Status.Synchronization)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf1)
+	})
 }
 
 func TestCheckWorkflowExistence(t *testing.T) {


Reviews:
user: login: Joibel
body: Thanks for finding this
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-19T07:55:16Z
user: login: atgane
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T13:56:44Z
user: login: Joibel
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-20T08:43:06Z
user: login: Joibel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T08:44:42Z
user: login: atgane
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T13:42:55Z
user: login: Joibel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T10:45:12Z
user: login: atgane
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T14:23:23Z
user: login: Joibel
body: Awesome, thanks for your contribution
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-22T08:28:55Z

Review comments:
diff_hunk: @@ -524,7 +524,9 @@ func (sm *Manager) ReleaseAll(wf *wfv1.Workflow) bool {
 	}
 
 	if wf.Status.Synchronization.Mutex != nil {
-		for _, holding := range wf.Status.Synchronization.Mutex.Holding {
+		h := make([]wfv1.MutexHolding, len(wf.Status.Synchronization.Mutex.Holding))
path: workflow/sync/sync_manager.go
user: login: Joibel
body: Shouldn't this fix also be applied on line 499, for semaphores?
created_at: 2025-05-19T07:55:05Z
author_association: MEMBER
diff_hunk: @@ -524,7 +524,9 @@ func (sm *Manager) ReleaseAll(wf *wfv1.Workflow) bool {
 	}
 
 	if wf.Status.Synchronization.Mutex != nil {
-		for _, holding := range wf.Status.Synchronization.Mutex.Holding {
+		h := make([]wfv1.MutexHolding, len(wf.Status.Synchronization.Mutex.Holding))
path: workflow/sync/sync_manager.go
user: login: atgane
body: Line 499 doesn't directly modify the `wf.Status.Synchronization.Semaphore.Holding` array, so there's no need to include a change for it. On the other hand, line 505 might cause issues if `holding.Holders` contains multiple elements.

That said, I'm not sure if there's a scenario where `holding.Holders` actually has more than one element. In all of my tests, it has always contained only one, and I've never encountered any related issues.

I haven’t seen a concrete issue yet, but would it be okay if I bring it up again later if I do? I haven’t found a relevant issue to include in that PR for now.
created_at: 2025-05-19T13:56:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -906,6 +906,42 @@ func TestMutexWfLevel(t *testing.T) {
 		syncManager.ReleaseAll(wf2)
 		assert.Empty(t, mutex.pending.items)
 	})
+
+	t.Run("WorkflowLevelMutexAcquireAndReleaseWithMultipleMutex", func(t *testing.T) {
+		syncManager := NewLockManager(ctx, kube, "", nil, syncLimitFunc, func(key string) {
+			// nextKey = key
+		}, WorkflowExistenceFunc)
+		wf := wfv1.MustUnmarshalWorkflow(wfWithMutex)
+		mutexes := make([]*wfv1.Mutex, 0, 10)
+		for i := range 10 {
+			mutexes = append(mutexes, &wfv1.Mutex{Name: fmt.Sprintf("mutex%d", i)})
+		}
+		wf.Spec.Synchronization.Mutexes = mutexes
+		wf1 := wf.DeepCopy()
+		wf1.Name = "two"
+
+		status, wfUpdate, msg, failedLockName, err := syncManager.TryAcquire(ctx, wf, "", wf.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf.Status.Synchronization)
+		require.NotNil(t, wf.Status.Synchronization.Mutex)
+		require.NotNil(t, wf.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf)
+
+		status, wfUpdate, msg, failedLockName, err = syncManager.TryAcquire(ctx, wf1, "", wf1.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf1.Status.Synchronization)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf1)
+	})
 }
path: workflow/sync/sync_manager_test.go
user: login: Joibel
body: Could we prove that semaphores also work here

```
	t.Run("WorkflowLevelSemaphoreAcquireAndReleaseWithMultipleSemaphores", func(t *testing.T) {
		// Create ConfigMap with multiple semaphore limits
		cm := v1.ConfigMap{
			ObjectMeta: metav1.ObjectMeta{
				Name:      "my-config",
				Namespace: "default",
			},
			Data: map[string]string{
				"sem1": "1",
				"sem2": "1",
				"sem3": "1",
			},
		}
		_, err := kube.CoreV1().ConfigMaps("default").Create(ctx, &cm, metav1.CreateOptions{})
		require.NoError(t, err)

		syncManager := NewLockManager(ctx, kube, "", nil, syncLimitFunc, func(key string) {
			// nextKey = key
		}, WorkflowExistenceFunc)

		// Create two workflows that both need all semaphores
		wf1 := wfv1.MustUnmarshalWorkflow(wfWithSemaphore)
		wf1.Name = "wf1"
		wf1.Spec.Synchronization.Semaphores = []*wfv1.SemaphoreRef{
			{
				ConfigMapKeyRef: &v1.ConfigMapKeySelector{
					LocalObjectReference: v1.LocalObjectReference{Name: "my-config"},
					Key:                  "sem1",
				},
			},
			{
				ConfigMapKeyRef: &v1.ConfigMapKeySelector{
					LocalObjectReference: v1.LocalObjectReference{Name: "my-config"},
					Key:                  "sem2",
				},
			},
			{
				ConfigMapKeyRef: &v1.ConfigMapKeySelector{
					LocalObjectReference: v1.LocalObjectReference{Name: "my-config"},
					Key:                  "sem3",
				},
			},
		}

		wf2 := wf1.DeepCopy()
		wf2.Name = "wf2"

		// First workflow should acquire all semaphores
		status, wfUpdate, msg, failedLockName, err := syncManager.TryAcquire(ctx, wf1, "", wf1.Spec.Synchronization)
		require.NoError(t, err)
		assert.Empty(t, msg)
		assert.Empty(t, failedLockName)
		assert.True(t, status)
		assert.True(t, wfUpdate)
		require.NotNil(t, wf1.Status.Synchronization)
		require.NotNil(t, wf1.Status.Synchronization.Semaphore)
		require.NotNil(t, wf1.Status.Synchronization.Semaphore.Holding)
		assert.Len(t, wf1.Status.Synchronization.Semaphore.Holding, 3)

		// Release all semaphores from first workflow
		syncManager.ReleaseAll(wf1)

		// Second workflow should now be able to acquire all semaphores
		status, wfUpdate, msg, failedLockName, err = syncManager.TryAcquire(ctx, wf2, "", wf2.Spec.Synchronization)
		require.NoError(t, err)
		assert.Empty(t, msg)
		assert.Empty(t, failedLockName)
		assert.True(t, status)
		assert.True(t, wfUpdate)
		require.NotNil(t, wf2.Status.Synchronization)
		require.NotNil(t, wf2.Status.Synchronization.Semaphore)
		require.NotNil(t, wf2.Status.Synchronization.Semaphore.Holding)
		assert.Len(t, wf2.Status.Synchronization.Semaphore.Holding, 3)

		// Clean up
		syncManager.ReleaseAll(wf2)
	})
```
created_at: 2025-05-20T08:42:56Z
author_association: MEMBER
diff_hunk: @@ -524,7 +524,9 @@ func (sm *Manager) ReleaseAll(wf *wfv1.Workflow) bool {
 	}
 
 	if wf.Status.Synchronization.Mutex != nil {
-		for _, holding := range wf.Status.Synchronization.Mutex.Holding {
+		h := make([]wfv1.MutexHolding, len(wf.Status.Synchronization.Mutex.Holding))
path: workflow/sync/sync_manager.go
user: login: Joibel
body: Ok, can we just add something like the test I've suggested to prove the equivalent works.
created_at: 2025-05-20T08:44:42Z
author_association: MEMBER
diff_hunk: @@ -906,6 +906,42 @@ func TestMutexWfLevel(t *testing.T) {
 		syncManager.ReleaseAll(wf2)
 		assert.Empty(t, mutex.pending.items)
 	})
+
+	t.Run("WorkflowLevelMutexAcquireAndReleaseWithMultipleMutex", func(t *testing.T) {
+		syncManager := NewLockManager(ctx, kube, "", nil, syncLimitFunc, func(key string) {
+			// nextKey = key
+		}, WorkflowExistenceFunc)
+		wf := wfv1.MustUnmarshalWorkflow(wfWithMutex)
+		mutexes := make([]*wfv1.Mutex, 0, 10)
+		for i := range 10 {
+			mutexes = append(mutexes, &wfv1.Mutex{Name: fmt.Sprintf("mutex%d", i)})
+		}
+		wf.Spec.Synchronization.Mutexes = mutexes
+		wf1 := wf.DeepCopy()
+		wf1.Name = "two"
+
+		status, wfUpdate, msg, failedLockName, err := syncManager.TryAcquire(ctx, wf, "", wf.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf.Status.Synchronization)
+		require.NotNil(t, wf.Status.Synchronization.Mutex)
+		require.NotNil(t, wf.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf)
+
+		status, wfUpdate, msg, failedLockName, err = syncManager.TryAcquire(ctx, wf1, "", wf1.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf1.Status.Synchronization)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf1)
+	})
 }
path: workflow/sync/sync_manager_test.go
user: login: atgane
body: I tried running the test case you provided, but it worked fine.

Do you happen to have any logs that show the semaphore not releasing correctly in that case? In my logs, it works correctly.

```
time="2025-05-20T22:39:09+09:00" level=info msg="Sync manager ttl" syncLimitCacheTTL=0s
time="2025-05-20T22:39:09+09:00" level=info msg="Sync manager initialized" dbConfigured=false
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem1"
time="2025-05-20T22:39:09+09:00" level=info msg="TryAcquire on default/ConfigMap/my-config/sem1"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem1"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem2"
time="2025-05-20T22:39:09+09:00" level=info msg="TryAcquire on default/ConfigMap/my-config/sem2"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem2"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem3"
time="2025-05-20T22:39:09+09:00" level=info msg="TryAcquire on default/ConfigMap/my-config/sem3"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem3"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem1"
time="2025-05-20T22:39:09+09:00" level=info msg="default/ConfigMap/my-config/sem1 semaphore resized from 0 to 1" lockType=semaphore name=default/ConfigMap/my-config/sem1
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem2"
time="2025-05-20T22:39:09+09:00" level=info msg="default/ConfigMap/my-config/sem2 semaphore resized from 0 to 1" lockType=semaphore name=default/ConfigMap/my-config/sem2
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem3"
time="2025-05-20T22:39:09+09:00" level=info msg="default/ConfigMap/my-config/sem3 semaphore resized from 0 to 1" lockType=semaphore name=default/ConfigMap/my-config/sem3
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem1"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem2"
time="2025-05-20T22:39:09+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem3"
time="2025-05-20T22:39:09+09:00" level=info msg="default/ConfigMap/my-config/sem1 acquired by default/wf1. Lock availability: 0/1" lockType=semaphore name=default/ConfigMap/my-config/sem1
time="2025-05-20T22:41:00+09:00" level=info msg="default/ConfigMap/my-config/sem2 acquired by default/wf1. Lock availability: 0/1" lockType=semaphore name=default/ConfigMap/my-config/sem2
time="2025-05-20T22:41:00+09:00" level=info msg="default/ConfigMap/my-config/sem3 acquired by default/wf1. Lock availability: 0/1" lockType=semaphore name=default/ConfigMap/my-config/sem3
time="2025-05-20T22:41:01+09:00" level=info msg="Lock has been released by default/wf1. Available locks: 1" lockType=semaphore name=default/ConfigMap/my-config/sem1
time="2025-05-20T22:41:01+09:00" level=info msg="default/wf1 released a lock from default/ConfigMap/my-config/sem1"
time="2025-05-20T22:41:02+09:00" level=info msg="Lock has been released by default/wf1. Available locks: 1" lockType=semaphore name=default/ConfigMap/my-config/sem2
time="2025-05-20T22:41:02+09:00" level=info msg="default/wf1 released a lock from default/ConfigMap/my-config/sem2"
time="2025-05-20T22:41:02+09:00" level=info msg="Lock has been released by default/wf1. Available locks: 1" lockType=semaphore name=default/ConfigMap/my-config/sem3
time="2025-05-20T22:41:02+09:00" level=info msg="default/wf1 released a lock from default/ConfigMap/my-config/sem3"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem1"
time="2025-05-20T22:41:02+09:00" level=info msg="TryAcquire on default/ConfigMap/my-config/sem1"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem1"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem2"
time="2025-05-20T22:41:02+09:00" level=info msg="TryAcquire on default/ConfigMap/my-config/sem2"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem2"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem3"
time="2025-05-20T22:41:02+09:00" level=info msg="TryAcquire on default/ConfigMap/my-config/sem3"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem3"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem1"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem2"
time="2025-05-20T22:41:02+09:00" level=info msg="DecodeLockName default/ConfigMap/my-config/sem3"
time="2025-05-20T22:41:02+09:00" level=info msg="default/ConfigMap/my-config/sem1 acquired by default/wf2. Lock availability: 0/1" lockType=semaphore name=default/ConfigMap/my-config/sem1
time="2025-05-20T22:41:04+09:00" level=info msg="default/ConfigMap/my-config/sem2 acquired by default/wf2. Lock availability: 0/1" lockType=semaphore name=default/ConfigMap/my-config/sem2
time="2025-05-20T22:41:05+09:00" level=info msg="default/ConfigMap/my-config/sem3 acquired by default/wf2. Lock availability: 0/1" lockType=semaphore name=default/ConfigMap/my-config/sem3
time="2025-05-20T22:41:06+09:00" level=info msg="Lock has been released by default/wf2. Available locks: 1" lockType=semaphore name=default/ConfigMap/my-config/sem1
time="2025-05-20T22:41:06+09:00" level=info msg="default/wf2 released a lock from default/ConfigMap/my-config/sem1"
time="2025-05-20T22:41:06+09:00" level=info msg="Lock has been released by default/wf2. Available locks: 1" lockType=semaphore name=default/ConfigMap/my-config/sem2
time="2025-05-20T22:41:06+09:00" level=info msg="default/wf2 released a lock from default/ConfigMap/my-config/sem2"
time="2025-05-20T22:41:07+09:00" level=info msg="Lock has been released by default/wf2. Available locks: 1" lockType=semaphore name=default/ConfigMap/my-config/sem3
time="2025-05-20T22:41:07+09:00" level=info msg="default/wf2 released a lock from default/ConfigMap/my-config/sem3"
```
created_at: 2025-05-20T13:42:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -906,6 +906,42 @@ func TestMutexWfLevel(t *testing.T) {
 		syncManager.ReleaseAll(wf2)
 		assert.Empty(t, mutex.pending.items)
 	})
+
+	t.Run("WorkflowLevelMutexAcquireAndReleaseWithMultipleMutex", func(t *testing.T) {
+		syncManager := NewLockManager(ctx, kube, "", nil, syncLimitFunc, func(key string) {
+			// nextKey = key
+		}, WorkflowExistenceFunc)
+		wf := wfv1.MustUnmarshalWorkflow(wfWithMutex)
+		mutexes := make([]*wfv1.Mutex, 0, 10)
+		for i := range 10 {
+			mutexes = append(mutexes, &wfv1.Mutex{Name: fmt.Sprintf("mutex%d", i)})
+		}
+		wf.Spec.Synchronization.Mutexes = mutexes
+		wf1 := wf.DeepCopy()
+		wf1.Name = "two"
+
+		status, wfUpdate, msg, failedLockName, err := syncManager.TryAcquire(ctx, wf, "", wf.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf.Status.Synchronization)
+		require.NotNil(t, wf.Status.Synchronization.Mutex)
+		require.NotNil(t, wf.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf)
+
+		status, wfUpdate, msg, failedLockName, err = syncManager.TryAcquire(ctx, wf1, "", wf1.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf1.Status.Synchronization)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf1)
+	})
 }
path: workflow/sync/sync_manager_test.go
user: login: Joibel
body: I agree it passes, but I'd still like the test added in this PR. It would prevent this as a potential future regression.
created_at: 2025-05-21T10:45:12Z
author_association: MEMBER
diff_hunk: @@ -906,6 +906,42 @@ func TestMutexWfLevel(t *testing.T) {
 		syncManager.ReleaseAll(wf2)
 		assert.Empty(t, mutex.pending.items)
 	})
+
+	t.Run("WorkflowLevelMutexAcquireAndReleaseWithMultipleMutex", func(t *testing.T) {
+		syncManager := NewLockManager(ctx, kube, "", nil, syncLimitFunc, func(key string) {
+			// nextKey = key
+		}, WorkflowExistenceFunc)
+		wf := wfv1.MustUnmarshalWorkflow(wfWithMutex)
+		mutexes := make([]*wfv1.Mutex, 0, 10)
+		for i := range 10 {
+			mutexes = append(mutexes, &wfv1.Mutex{Name: fmt.Sprintf("mutex%d", i)})
+		}
+		wf.Spec.Synchronization.Mutexes = mutexes
+		wf1 := wf.DeepCopy()
+		wf1.Name = "two"
+
+		status, wfUpdate, msg, failedLockName, err := syncManager.TryAcquire(ctx, wf, "", wf.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf.Status.Synchronization)
+		require.NotNil(t, wf.Status.Synchronization.Mutex)
+		require.NotNil(t, wf.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf)
+
+		status, wfUpdate, msg, failedLockName, err = syncManager.TryAcquire(ctx, wf1, "", wf1.Spec.Synchronization)
+		require.NoError(t, err)
+		require.Empty(t, msg)
+		require.Empty(t, failedLockName)
+		require.True(t, status)
+		require.True(t, wfUpdate)
+		require.NotNil(t, wf1.Status.Synchronization)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex)
+		require.NotNil(t, wf1.Status.Synchronization.Mutex.Holding)
+		syncManager.ReleaseAll(wf1)
+	})
 }
path: workflow/sync/sync_manager_test.go
user: login: atgane
body: Sounds like good intentions. I'll add a test and push it. 

Please check the location of the test code.
created_at: 2025-05-21T14:23:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -524,7 +524,9 @@ func (sm *Manager) ReleaseAll(wf *wfv1.Workflow) bool {
 	}
 
 	if wf.Status.Synchronization.Mutex != nil {
-		for _, holding := range wf.Status.Synchronization.Mutex.Holding {
+		h := make([]wfv1.MutexHolding, len(wf.Status.Synchronization.Mutex.Holding))
path: workflow/sync/sync_manager.go
user: login: Joibel
body: Thanks
created_at: 2025-05-22T08:28:29Z
author_association: MEMBER

PR comments:
user: login: Joibel
created_at: 2025-05-20T07:58:11Z
body: /cherry-pick release-3.6
author_association: MEMBER
user: login: atgane
created_at: 2025-05-20T14:03:00Z
body: @Joibel 

The reason why semaphores don't cause problems seems to be due to their different handling from mutexes.

The part of the semaphore where you can run into the same problem as mutexes is the green boxed part below.

```go
// https://github.com/argoproj/argo-workflows/blob/13c9b37bf7a311a8ffbb8a608f00b56c05c5fcbf/workflow/sync/sync_manager.go#L507
			for _, holderKey := range holding.Holders {
				syncLockHolder.release(holderKey)
				wf.Status.Synchronization.Semaphore.LockReleased(holderKey, holding.Semaphore) // ✅
				log.Infof("%s released a lock from %s", holderKey, holding.Semaphore)
			}
```

The function is dynamically changing the `semaphoreHolding.Holders` array inside the iterator.

```go
// https://github.com/argoproj/argo-workflows/blob/13c9b37bf7a311a8ffbb8a608f00b56c05c5fcbf/pkg/apis/workflow/v1alpha1/workflow_types.go#L3860C1-L3871C2
func (ss *SemaphoreStatus) LockReleased(holderKey, lockKey string) bool {
	i, semaphoreHolding := ss.GetHolding(lockKey)
	holdingName := holderKey

	if i >= 0 {
		semaphoreHolding.Holders = slices.DeleteFunc(semaphoreHolding.Holders,
			func(x string) bool { return x == holdingName })
		ss.Holding[i] = semaphoreHolding
		return true
	}
	return false
}
```

If so, a large number of elements in the `semaphoreHolding.Holders` array can cause strange behavior.

However, the part where the Holders array grows is the part below, where the elements of the array grow:
1. There should be no duplicate `lockKeys` in the same workflow.
2. The `semaphoreHolding.Holders` array must not contain a holdingName.

```go
// https://github.com/argoproj/argo-workflows/blob/13c9b37bf7a311a8ffbb8a608f00b56c05c5fcbf/pkg/apis/workflow/v1alpha1/workflow_types.go#L3846
func (ss *SemaphoreStatus) LockAcquired(holderKey, lockKey string, currentHolders []string) bool {
	i, semaphoreHolding := ss.GetHolding(lockKey)
	holdingName := holderKey
	if i < 0 {
		ss.Holding = append(ss.Holding, SemaphoreHolding{Semaphore: lockKey, Holders: []string{holdingName}})
		return true
	} else if !slices.Contains(semaphoreHolding.Holders, holdingName) {
		semaphoreHolding.Holders = append(semaphoreHolding.Holders, holdingName) // ✅
		ss.Holding[i] = semaphoreHolding
		return true
	}
	return false
}
```

Condition 1 is a defended condition and is returning an error if the lockKey is the same in the same workflow.

So this part is probably not a problem.
author_association: CONTRIBUTOR
user: login: Joibel
created_at: 2025-05-21T15:30:20Z
body: `main` is broken from some recent merges, I'll get that fixed so this CI passes.
author_association: MEMBER