Repository: oras-project/oras-go
Pull request #921: fix: fix concurrency bugs causing panics and race conditions in `internal/syncutil`
Author: login: Wwwsylvia
Created: 2025-03-17T11:07:12Z | Merged: 2025-03-19T10:32:28Z

1. Remove the use of `atomic.Value`
2. Cancel all go-routines before releasing semaphore permits
3. Wait for schedule go-routines to release the semaphore permits before returning
4. Add unit tests

Fixes #916, fixes #908

Linked issue #916: Incorrect use of `atomic.Value` in the `syncutil.Go` utility function causes panic
While using ORAS-Go to push a layout with some slightly-incorrect manifest contents I encountered this panic:

```
panic: sync/atomic: compare and swap of inconsistently typed value into Value

goroutine 83 [running]:
sync/atomic.(*Value).CompareAndSwap(0xc00034a700, {0x0, 0x0}, {0x99fe80, 0xc0004a2660})
        sync/atomic/value.go:174 +0x150
oras.land/oras-go/v2/internal/syncutil.Go[...].func1.func2()
        oras.land/oras-go/v2@v2.5.1-0.20250221033735-cb6d75be7dd4/internal/syncutil/limit.go:86 +0xe6
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/sync@v0.11.0/errgroup/errgroup.go:78 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 46
        golang.org/x/sync@v0.11.0/errgroup/errgroup.go:75 +0x93
```

Unfortunately I can't share with you the incorrect content since it was just throwaway stuff in a job where I didn't preserve the input artifacts, but I found the relevant code here:

https://github.com/oras-project/oras-go/blob/853e0125ccad32ff691e4ed70e156c7619021bfd/internal/syncutil/limit.go#L73-L92

I think the `egErr.CompareAndSwap` call on line 86 is incorrect, because `err` here has an interface type (`error`) and so whichever concrete error type is stored there first freezes the type expected for all future calls, per the [`Value.CompareAndSwap`](https://pkg.go.dev/sync/atomic#Value.CompareAndSwap) documentation:

> All calls to CompareAndSwap for a given Value must use values of the same concrete type. CompareAndSwap of an inconsistent type panics, as does CompareAndSwap(old, nil).

If `egErr.CompareAndSwap` gets subsequently called again with a different concrete error type from one of the other goroutines then I expect that this would generate the panic I observed.

I think one potential way to make this more robust would be for this function to declare a local struct type to act as the consistent concrete type for `egErr`, with the dynamically-typed `error` inside it:

```go
type WrappedError struct {
    err error
}
# ...

egErr.CompareAndSwap(nil, WrappedErr{err})
```

Then when loading the error again:

```go
if egErrWrap, ok := egErr.Load().(WrappedError) ok && egErrWrap.err != nil {
	return egErrWrap.err
}
```

The concrete type in `egErr` would therefore always be exactly `WrappedError`, and so this panic would then be impossible.

Again I'm sorry I can't offer a specific reproduction case for this, but I hope the above is useful. I'm not blocked by this because I've now corrected my manifest content so that it doesn't cause these errors.

Linked issue comments: user: login: apparentlymart
created_at: 2025-03-06T01:01:41Z
body: I encountered the same panic again a little while later and so I added some print statements to get a better idea of what was going on. It turns out that it occurs when I try to push a manifest using the standard empty JSON config descriptor into a registry that doesn't have that blob already -- this time using the `oras cp --from-oci-layout` command rather than ORAS-Go directly -- causing the following sequence of stores to `egErr`:

* `*fmt.wrapError: sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a: application/vnd.oci.empty.v1+json: not found`
* `*url.Error: Post "http://127.0.0.1:5000/v2/test-thing/blobs/uploads/": sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a: application/vnd.oci.empty.v1+json: not found`
* `*fmt.wrapError: sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a: application/vnd.oci.empty.v1+json: not found`

The second store is causing the panic because `*url.Error` is inconsistent with the previous `*fmt.wrapError`.

I captured all three of these using a version of this function modified as I suggested in my original writeup, which fixed the problem and so also allowed the ORAS CLI to return one of the errors without crashing:

```go
// Go concurrently invokes fn on items.
func Go[T any](ctx context.Context, limiter *semaphore.Weighted, fn GoFunc[T], items ...T) error {
	type WrapErr struct {
		err error
	}

	eg, egCtx := errgroup.WithContext(ctx)
	var egErr atomic.Value
	for _, item := range items {
		region := LimitRegion(egCtx, limiter)
		if err := region.Start(); err != nil {
			if egErrWrap, ok := egErr.Load().(WrapErr); ok && egErrWrap.err != nil {
				return egErrWrap.err
			}
			return err
		}
		eg.Go(func(t T) func() error {
			return func() error {
				defer region.End()
				err := fn(egCtx, region, t)
				if err != nil {
					fmt.Fprintf(os.Stderr, "%T: %s\n", err, err)
					egErr.CompareAndSwap(nil, WrapErr{err})
					return err
				}
				return nil
			}
		}(item))
	}
	return eg.Wait()
}
```

In case it's important for reproduction, my local OCI layout directory _also_ lacks the blob for the empty JSON object. I suspect that's what the first of the three errors was complaining about, but I've not confirmed that.

author_association: NONE
user: login: Wwwsylvia
created_at: 2025-03-07T11:06:27Z
body: Hi @apparentlymart , thanks for opening the issue. We will look into it once we have some bandwidth.
author_association: MEMBER

Diff:
diff --git a/internal/syncutil/limit.go b/internal/syncutil/limit.go
index f2caabf72..e429f24f4 100644
--- a/internal/syncutil/limit.go
+++ b/internal/syncutil/limit.go
@@ -17,7 +17,6 @@ package syncutil
 
 import (
 	"context"
-	"sync/atomic"
 
 	"golang.org/x/sync/errgroup"
 	"golang.org/x/sync/semaphore"
@@ -68,27 +67,41 @@ type GoFunc[T any] func(ctx context.Context, region *LimitedRegion, t T) error
 
 // Go concurrently invokes fn on items.
 func Go[T any](ctx context.Context, limiter *semaphore.Weighted, fn GoFunc[T], items ...T) error {
+	ctx, cancel := context.WithCancelCause(ctx)
+	defer cancel(nil)
+
 	eg, egCtx := errgroup.WithContext(ctx)
-	var egErr atomic.Value
 	for _, item := range items {
 		region := LimitRegion(egCtx, limiter)
 		if err := region.Start(); err != nil {
-			if egErr, ok := egErr.Load().(error); ok && egErr != nil {
-				return egErr
-			}
-			return err
+			cancel(err)
+			// break loop instead of returning to allow previously scheduled
+			// goroutines to finish their deferred region.End() calls
+			break
 		}
-		eg.Go(func(t T) func() error {
+
+		eg.Go(func(t T, lr *LimitedRegion) func() error {
 			return func() error {
-				defer region.End()
-				err := fn(egCtx, region, t)
-				if err != nil {
-					egErr.CompareAndSwap(nil, err)
+				defer lr.End()
+
+				select {
+				case <-egCtx.Done():
+					// skip the task if the context is already cancelled
+					return nil
+				default:
+				}
+
+				if err := fn(egCtx, lr, t); err != nil {
+					cancel(err)
 					return err
 				}
 				return nil
 			}
-		}(item))
+		}(item, region))
+	}
+
+	if err := eg.Wait(); err != nil {
+		cancel(err)
 	}
-	return eg.Wait()
+	return context.Cause(ctx)
 }
diff --git a/internal/syncutil/limit_test.go b/internal/syncutil/limit_test.go
new file mode 100644
index 000000000..5f644b2e1
--- /dev/null
+++ b/internal/syncutil/limit_test.go
@@ -0,0 +1,88 @@
+/*
+Copyright The ORAS Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package syncutil
+
+import (
+	"context"
+	"errors"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"golang.org/x/sync/semaphore"
+)
+
+func TestLimitedRegion_Success(t *testing.T) {
+	limiter := semaphore.NewWeighted(2)
+	ctx := context.Background()
+	var counter int32
+
+	err := Go(ctx, limiter, func(ctx context.Context, region *LimitedRegion, i int) error {
+		// just sleeps a little and increments counter to simulate task
+		time.Sleep(10 * time.Millisecond)
+		atomic.AddInt32(&counter, 1)
+		return nil
+	}, 1, 2, 3, 4, 5)
+
+	if err != nil {
+		t.Fatalf("expected no error, got %v", err)
+	}
+
+	// after everything finishes, we expect counter to be 5
+	if want := 5; atomic.LoadInt32(&counter) != int32(want) {
+		t.Errorf("expected counter == %v, got %v", want, counter)
+	}
+
+	// when all work is done the semaphore should have all permits available
+	if !limiter.TryAcquire(2) {
+		t.Error("semaphore permits were not fully released at the end")
+	}
+	limiter.Release(2)
+}
+
+func TestLimitedRegion_Cancellation(t *testing.T) {
+	limiter := semaphore.NewWeighted(2)
+	ctx := context.Background()
+	var counter int32
+
+	errTest := errors.New("test error")
+	err := Go(ctx, limiter, func(ctx context.Context, region *LimitedRegion, i int) error {
+		if i < 0 {
+			// trigger an error on negative values
+			return errTest
+		}
+		// just sleeps a little and increments counter to simulate task
+		time.Sleep(50 * time.Millisecond)
+		atomic.AddInt32(&counter, 1)
+		return nil
+	}, 1, -1, 2, 0, -2)
+
+	// we expect the returned error to be errTest.
+	if !errors.Is(err, errTest) {
+		t.Fatalf("expected error %v; got %v", errTest, err)
+	}
+
+	// after everything finishes, we expect counter to be smaller than 5.
+	if max := 5; atomic.LoadInt32(&counter) >= int32(max) {
+		t.Errorf("expected counter < %v, got %v", max, counter)
+	}
+
+	// when all work is done the semaphore should have all permits available
+	if !limiter.TryAcquire(2) {
+		t.Error("semaphore permit was not released after error cancellation")
+	}
+	limiter.Release(2)
+}
diff --git a/internal/syncutil/limitgroup.go b/internal/syncutil/limitgroup.go
index 1071bedc9..4ef087dcc 100644
--- a/internal/syncutil/limitgroup.go
+++ b/internal/syncutil/limitgroup.go
@@ -21,7 +21,7 @@ import (
 	"golang.org/x/sync/errgroup"
 )
 
-// A LimitedGroup is a collection of goroutines working on subtasks that are part of
+// LimitedGroup is a collection of goroutines working on subtasks that are part of
 // the same overall task.
 type LimitedGroup struct {
 	grp *errgroup.Group
diff --git a/internal/syncutil/limitgroup_test.go b/internal/syncutil/limitgroup_test.go
new file mode 100644
index 000000000..31de350c7
--- /dev/null
+++ b/internal/syncutil/limitgroup_test.go
@@ -0,0 +1,120 @@
+/*
+Copyright The ORAS Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package syncutil
+
+import (
+	"context"
+	"errors"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+func TestLimitedGroup_Success(t *testing.T) {
+	ctx := context.Background()
+	numTasks := 5
+
+	// Create a limited group with a concurrency limit of 2.
+	lg, _ := LimitGroup(ctx, 2)
+	var counter int32
+
+	for range numTasks {
+		lg.Go(func() error {
+			// simulate some work.
+			time.Sleep(10 * time.Millisecond)
+			atomic.AddInt32(&counter, 1)
+			return nil
+		})
+	}
+
+	if err := lg.Wait(); err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+
+	if got := atomic.LoadInt32(&counter); got != int32(numTasks) {
+		t.Errorf("expected counter %d, got %d", numTasks, got)
+	}
+}
+
+func TestLimitedGroup_Error(t *testing.T) {
+	ctx := context.Background()
+	lg, _ := LimitGroup(ctx, 2)
+	errTest := errors.New("test error")
+	var executed int32
+
+	lg.Go(func() error {
+		// delay a bit so that other tasks are scheduled.
+		time.Sleep(20 * time.Millisecond)
+		atomic.AddInt32(&executed, 1)
+		return errTest
+	})
+
+	// simulates a slower, normal task.
+	lg.Go(func() error {
+		// wait until cancellation is (hopefully) in effect.
+		time.Sleep(50 * time.Millisecond)
+		atomic.AddInt32(&executed, 1)
+		return nil
+	})
+
+	err := lg.Wait()
+	if !errors.Is(err, errTest) {
+		t.Fatalf("expected error %v, got %v", errTest, err)
+	}
+
+	if atomic.LoadInt32(&executed) < 1 {
+		t.Errorf("expected at least one task executed, got %d", executed)
+	}
+}
+
+func TestLimitedGroup_Limit(t *testing.T) {
+	ctx := context.Background()
+	limit := 2
+	lg, _ := LimitGroup(ctx, limit)
+	var concurrent, maxConcurrent int32
+	numTasks := 10
+
+	for range numTasks {
+		lg.Go(func() error {
+			// increment the count of concurrently active tasks.
+			cur := atomic.AddInt32(&concurrent, 1)
+			// update the max concurrent tasks if needed.
+			for {
+				prevMax := atomic.LoadInt32(&maxConcurrent)
+				if cur > prevMax {
+					if atomic.CompareAndSwapInt32(&maxConcurrent, prevMax, cur) {
+						break
+					}
+				} else {
+					break
+				}
+			}
+
+			// simulate a short task.
+			time.Sleep(20 * time.Millisecond)
+			atomic.AddInt32(&concurrent, -1)
+			return nil
+		})
+	}
+
+	if err := lg.Wait(); err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+
+	if maxConcurrent > int32(limit) {
+		t.Errorf("expected max concurrent tasks <= %d, got %d", limit, maxConcurrent)
+	}
+}


Reviews:
user: login: shizhMSFT
body: For the code coverage, let's discuss offline.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T13:50:00Z
user: login: Wwwsylvia
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T04:36:17Z
user: login: shizhMSFT
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T10:29:57Z

Review comments:
diff_hunk: @@ -68,27 +68,57 @@ type GoFunc[T any] func(ctx context.Context, region *LimitedRegion, t T) error
 
 // Go concurrently invokes fn on items.
 func Go[T any](ctx context.Context, limiter *semaphore.Weighted, fn GoFunc[T], items ...T) error {
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+
 	eg, egCtx := errgroup.WithContext(ctx)
-	var egErr atomic.Value
+	var once sync.Once
+	var firstErr error
+
 	for _, item := range items {
 		region := LimitRegion(egCtx, limiter)
 		if err := region.Start(); err != nil {
-			if egErr, ok := egErr.Load().(error); ok && egErr != nil {
-				return egErr
-			}
-			return err
+			once.Do(func() {
+				// record the first error occurred
+				firstErr = err
+			})
+			// when an error occurs, cancel other tasks
+			cancel()
+			// continue loop instead of returning to allow already scheduled
+			// goroutines to finish their deferred region.End() calls
+			continue
path: internal/syncutil/limit.go
user: login: shizhMSFT
body: Why don't we just `break`?
created_at: 2025-03-18T13:44:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -68,27 +68,57 @@ type GoFunc[T any] func(ctx context.Context, region *LimitedRegion, t T) error
 
 // Go concurrently invokes fn on items.
 func Go[T any](ctx context.Context, limiter *semaphore.Weighted, fn GoFunc[T], items ...T) error {
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
+
 	eg, egCtx := errgroup.WithContext(ctx)
-	var egErr atomic.Value
+	var once sync.Once
+	var firstErr error
+
 	for _, item := range items {
 		region := LimitRegion(egCtx, limiter)
 		if err := region.Start(); err != nil {
-			if egErr, ok := egErr.Load().(error); ok && egErr != nil {
-				return egErr
-			}
-			return err
+			once.Do(func() {
+				// record the first error occurred
+				firstErr = err
+			})
+			// when an error occurs, cancel other tasks
+			cancel()
+			// continue loop instead of returning to allow already scheduled
+			// goroutines to finish their deferred region.End() calls
+			continue
path: internal/syncutil/limit.go
user: login: Wwwsylvia
body: Oh good catch! We should break
created_at: 2025-03-19T04:36:17Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-03-17T11:08:22Z
body: ## [Codecov](https://app.codecov.io/gh/oras-project/oras-go/pull/921?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oras-project) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 80.98%. Comparing base [(`fdf2c51`)](https://app.codecov.io/gh/oras-project/oras-go/commit/fdf2c5161537db3ba9b57efe4141526e488610af?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oras-project) to head [(`26ee98a`)](https://app.codecov.io/gh/oras-project/oras-go/commit/26ee98a867dd3e1d0876659c7bdb4c7962f596ca?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oras-project).
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #921      +/-   ##
==========================================
+ Coverage   80.24%   80.98%   +0.74%     
==========================================
  Files          63       63              
  Lines        6053     6063      +10     
==========================================
+ Hits         4857     4910      +53     
+ Misses        858      811      -47     
- Partials      338      342       +4     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/oras-project/oras-go/pull/921?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oras-project).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oras-project).

<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: Wwwsylvia
created_at: 2025-03-17T11:12:32Z
body: Mmmm the missing lines are tricky to cover
author_association: MEMBER