Repository: cockroachdb/pebble

Issue:
master: nightly linux-msan failed
The nightly linux-msan test run failed on 4cb1291805fb54e025da6408ff7d3dcb1bfcb8e8 (go1.23). Please review the run [here](https://github.com/cockroachdb/pebble/actions/runs/18816397870).

Jira issue: PEBBLE-1237

Discussion:
@github-actions: The nightly darwin test run failed on eda3cd30fad5cddb7749d87fe955a693740d83ba (go1.23). Please review the run [here](https://github.com/cockroachdb/pebble/actions/runs/18838050817).

[radu] this does not belong here. It is from crl-release-25.2 and is a different target.@github-actions: The nightly linux-msan test run failed on 4cb1291805fb54e025da6408ff7d3dcb1bfcb8e8 (go1.23). Please review the run [here](https://github.com/cockroachdb/pebble/actions/runs/18871119089).@RaduBerinde: I have been trying to reproduce at various shas with a special CI action.

I "bisected" to the delete pacer rewrite e56c4c9140dc7db01e1341cf1bd39662a52ba06a (failure here: https://github.com/RaduBerinde/pebble/actions/runs/18911102158)

The change before (c2e588ae23f1ca5181eff7cd22e3b2cd60823ef2) had many successful runs (the one failure is a timeout in a test):
https://github.com/RaduBerinde/pebble/actions/runs/18899140913
https://github.com/RaduBerinde/pebble/actions/runs/18911097600
https://github.com/RaduBerinde/pebble/actions/runs/18912496300

The delete pacer rewrite does not involve any unsafe code. I have been looking at the diff and can't really explain this, other than it somehow makes an existing bug show up for some reason..

I will keep investigating. There are two different failure modes. One always happens during a compaction in a specific checkpoint test, but does not reproduce if I just run that one test many times.. (perhaps pointing to a timing issue)@RaduBerinde: Leaving some notes. I am able to reproduce a msan uninitialized failure when using the path of an obsolete file when it comes time to delete it.

Eventually I was able to reproduce on a gceworker with Ubuntu 24.04, with this docker container:
```
docker run --security-opt seccomp:unconfined --rm -it   -v "$PWD":/work -w /work   ghcr.io/christopherhx/runner-images:ubuntu24-runner-large-latest   bas
```

And this command
```
GOMAXPROCS=4 CC=clang go test -msan -tags 'invariants slowbuild' ./internal/metamorphic -run TestMetaCockroachKVs/execution/standard-012 -timeout 0 -v --exec 'stress -p 80 -maxruns 5000
```

I tried many variations of debugging messages and techniques. I am using this code to check if the data is poisoned and get more info (like stacks) before the failure:
```go
/*
#include <sanitizer/msan_interface.h>
static inline long msan_test(const void* p, size_t n) {
  return __msan_test_shadow(p, n);   // -1 if clean, else offset of first poisoned byte
}
*/
import "C"
import "unsafe"

func TestString(s string) (firstPoison int) {
	if len(s) == 0 {
		return -1
	}
	p := unsafe.StringData(s)
	return int(C.msan_test(unsafe.Pointer(p), C.size_t(len(s))))
}
```

In most cases, the file path is unchanged even though msan detected some kind of copy from unininitialized data. I can safely print a `string.Clone` of it but printing the string directly causes the failure.
But in a single instance I saw the path show up as:
```
        path clone: �o]+��͵�🪳🪳
```
This is exactly the `pebbleDBMagic` at the end of ssts.@github-actions: The nightly linux-msan test run failed on e8d7ecb385594ba11176e186a933e8c754220511 (go1.25). Please review the run [here](https://github.com/cockroachdb/pebble/actions/runs/18936943972).@github-actions: The nightly linux-msan test run failed on e8d7ecb385594ba11176e186a933e8c754220511 (go1.23). Please review the run [here](https://github.com/cockroachdb/pebble/actions/runs/18995127609).@github-actions: The nightly linux-msan test run failed on e8d7ecb385594ba11176e186a933e8c754220511 (go1.25). Please review the run [here](https://github.com/cockroachdb/pebble/actions/runs/18995127609).

Solution patch:
diff --git a/.github/workflows/nightlies.yaml b/.github/workflows/nightlies.yaml
index 447f4ea5..65fa4701 100644
--- a/.github/workflows/nightlies.yaml
+++ b/.github/workflows/nightlies.yaml
@@ -10,7 +10,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: [ '1.23', '1.25' ]
+        go: [ '1.25' ]
     uses: ./.github/workflows/tests.yaml
     with:
       sha: ${{ github.sha }}
@@ -21,7 +21,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: [ '1.23', '1.25' ]
+        go: [ '1.25' ]
     uses: ./.github/workflows/s390x.yaml
     with:
       sha: ${{ github.sha }}
@@ -32,7 +32,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: [ '1.23', '1.25' ]
+        go: [ '1.25' ]
     uses: ./.github/workflows/stress.yaml
     with:
       sha: ${{ github.sha }}
@@ -43,7 +43,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: [ '1.23', '1.25' ]
+        go: [ '1.25' ]
     uses: ./.github/workflows/instrumented.yaml
     with:
       sha: ${{ github.sha }}
diff --git a/Makefile b/Makefile
index 276cab69..4be93e5d 100644
--- a/Makefile
+++ b/Makefile
@@ -84,7 +84,7 @@ stress-crossversion:
 
 .PHONY: test-s390x-qemu
 test-s390x-qemu: TAGS += slowbuild
-test-s390x-qemu: S390X_GOVERSION := 1.23
+test-s390x-qemu: S390X_GOVERSION := 1.25
 test-s390x-qemu:
 	@echo "Running tests on s390x using QEMU"
 	@echo "Requires a recent linux with docker and qemu-user-static installed"
diff --git a/go.mod b/go.mod
index e070241f..938706cc 100644
--- a/go.mod
+++ b/go.mod
@@ -57,6 +57,4 @@ require (
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
 
-go 1.23.0
-
-toolchain go1.23.6
+go 1.25.3
diff --git a/internal/deletepacer/delete_pacer.go b/internal/deletepacer/delete_pacer.go
index 41fd0f69..3dd2a3b8 100644
--- a/internal/deletepacer/delete_pacer.go
+++ b/internal/deletepacer/delete_pacer.go
@@ -7,11 +7,11 @@ package deletepacer
 import (
 	"context"
 	"runtime/pprof"
+	"slices"
 	"sync"
 	"time"
 
 	"github.com/cockroachdb/crlib/crtime"
-	"github.com/cockroachdb/crlib/fifo"
 	"github.com/cockroachdb/pebble/internal/base"
 	"github.com/cockroachdb/pebble/internal/invariants"
 	"github.com/cockroachdb/pebble/metrics"
@@ -54,7 +54,7 @@ type DeletePacer struct {
 	mu struct {
 		sync.Mutex
 
-		queue             fifo.Queue[queueEntry]
+		queue             []queueEntry
 		queuedPacingBytes uint64
 		// queuedHistory keeps track of pacing bytes added to the queue within the
 		// last 5 minutes.
@@ -100,7 +100,6 @@ func Open(
 		deleteFn:        deleteFn,
 		notifyCh:        make(chan struct{}, 1),
 	}
-	dp.mu.queue = fifo.MakeQueue(&queueBackingPool)
 	dp.mu.queuedHistory.Init(crtime.NowMono(), RecentRateWindow)
 	dp.mu.deletedCond.L = &dp.mu.Mutex
 	dp.waitGroup.Add(1)
@@ -112,8 +111,6 @@ func Open(
 	return dp
 }
 
-var queueBackingPool = fifo.MakeQueueBackingPool[queueEntry]()
-
 type queueEntry struct {
 	ObsoleteFile
 	JobID int
@@ -159,17 +156,17 @@ func (dp *DeletePacer) mainLoop() {
 	dp.mu.Lock()
 	defer dp.mu.Unlock()
 	for {
-		if dp.mu.closed && dp.mu.queue.Len() == 0 {
+		if dp.mu.closed && len(dp.mu.queue) == 0 {
 			return
 		}
 		now := crtime.NowMono()
 		disablePacing := dp.mu.closed
-		if dp.mu.queue.Len() > maxQueueSize {
+		if len(dp.mu.queue) > maxQueueSize {
 			// The queue is getting out of hand; disable pacing.
 			disablePacing = true
 			if lastMaxQueueLog == 0 || now.Sub(lastMaxQueueLog) > time.Minute {
 				lastMaxQueueLog = now
-				dp.logger.Errorf("excessive delete pacer queue size %d; pacing temporarily disabled", dp.mu.queue.Len())
+				dp.logger.Errorf("excessive delete pacer queue size %d; pacing temporarily disabled", len(dp.mu.queue))
 			}
 		}
 
@@ -180,7 +177,7 @@ func (dp *DeletePacer) mainLoop() {
 		//   2. Wait for pacing debt to clear;
 		//   3. Otherwise, delete next file.
 		switch {
-		case dp.mu.queue.Len() == 0:
+		case len(dp.mu.queue) == 0:
 			// Nothing to do.
 			dp.mu.Unlock()
 			<-dp.notifyCh
@@ -203,8 +200,8 @@ func (dp *DeletePacer) mainLoop() {
 
 		default:
 			// Delete a file.
-			file := *dp.mu.queue.PeekFront()
-			dp.mu.queue.PopFront()
+			file := dp.mu.queue[0]
+			dp.mu.queue = dp.mu.queue[1:]
 			if b := file.pacingBytes(); b != 0 {
 				dp.mu.queuedPacingBytes = invariants.SafeSub(dp.mu.queuedPacingBytes, b)
 				rateCalc.AddDebt(b)
@@ -238,7 +235,12 @@ func (dp *DeletePacer) Enqueue(jobID int, files ...ObsoleteFile) {
 			dp.mu.queuedHistory.Add(now, b)
 		}
 		dp.mu.metrics.InQueue.Inc(file.FileType, file.FileSize, file.Placement)
-		dp.mu.queue.PushBack(queueEntry{
+		if len(dp.mu.queue) == cap(dp.mu.queue) {
+			// If we have to allocate, make sure we don't have to allocate again for a
+			// while.
+			dp.mu.queue = slices.Grow(dp.mu.queue, 1000)
+		}
+		dp.mu.queue = append(dp.mu.queue, queueEntry{
 			ObsoleteFile: file,
 			JobID:        jobID,
 		})
diff --git a/internal/devtools/go.mod b/internal/devtools/go.mod
index 0911d449..0e0dd28a 100644
--- a/internal/devtools/go.mod
+++ b/internal/devtools/go.mod
@@ -1,6 +1,6 @@
 module github.com/cockroachdb/pebble/internal/devtools
 
-go 1.23.6
+go 1.25.3
 
 require (
 	github.com/cockroachdb/cockroach v0.0.0-20250225003441-0fd08b1c5cc1



Test patch:
diff --git a/internal/deletepacer/delete_pacer_test.go b/internal/deletepacer/delete_pacer_test.go
index 9361da66..908e7751 100644
--- a/internal/deletepacer/delete_pacer_test.go
+++ b/internal/deletepacer/delete_pacer_test.go
@@ -248,7 +248,7 @@ func TestFallingBehind(t *testing.T) {
 	queueSize := func() int {
 		dp.mu.Lock()
 		defer dp.mu.Unlock()
-		return dp.mu.queue.Len()
+		return len(dp.mu.queue)
 	}
 	// At 1MB, each job will take 100ms each. Note that the rate increase based on
 	// history won't make much difference, since the enqueued size is averaged