Repository: cloudnative-pg/cloudnative-pg

Issue:
[Bug]: Former master cannot perform pg_rewind after failover if plugin is used for WAL archiving
### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

_No response_

### Version

1.27 (latest patch)

### What version of Kubernetes are you using?

1.34

### What is your Kubernetes environment?

Self-managed: kind (evaluation)

### How did you install the operator?

YAML manifest

### What happened?

Hi! 
While testing operator behavior on sudden network partitioning I've encountered situation when former primary fails to perform `pg_rewind` after network connection is returned, so that former primary is unable to catch up and become replica.

That only occurs when plugins are used for WAL archiving/restoration.

Precise steps for reproduction:

1. Create 3-node cluster with synchronous replication in KinD
2. Create pgbench deployment to create constant load on cluster
3. Disconnect kind node with current primary (ex. `docker network disconnect kind cnpg-worker3` )
4. Wait until failover is performed
5. Connect kind node with former primary back (ex. `docker network connect kind cnpg-worker3`)
6. See that former primary fails to perform pg_rewind with `--restore-target-wal` enabled due to unable to fetch WAL files

`pg_rewind` calls `manager wal-restore` with FULL path to target file:
```
/controller/manager wal-restore --log-destination /controller/log/postgres.json 00000001000000010000005F /var/lib/postgresql/data/pgdata/pg_wal/00000001000000010000005F
```

which we do not handle correctly at https://github.com/cloudnative-pg/cloudnative-pg/blob/9daa681325034a6b6548e347bbc4c0b54ea668db/internal/cmd/manager/walrestore/cmd.go#L126

making GPRC-call to plugin with duplicating PGDATA part of restore filepath:
```
/var/lib/postgresql/data/pgdata/var/lib/postgresql/data/pgdata/pg_wal/00000001000000010000005F
```


### Cluster resource

```shell
Cluster:


apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  instances: 3
  plugins:
  - name: cnpg-extensions.yandex.cloud      # Should always be "cnpg-extensions.yandex.cloud" for wal-g plugin
    isWALArchiver: true                     # Set to true to enable continuous WAL archiving
    parameters:
      backupConfig: example-backup-config   # Name of BackupConfig used to create backups
  postgresql:
    parameters:
      max_connections: "250"
      shared_buffers: 4GB
      effective_cache_size: 12GB
      maintenance_work_mem: 1GB
      checkpoint_completion_target: "0.9"
      wal_buffers: 16MB
      default_statistics_target: "100"
      random_page_cost: "1.1"
      effective_io_concurrency: "200"
      work_mem: 16513kB
      huge_pages: "off"
      min_wal_size: 1GB
      max_wal_size: 4GB
      max_worker_processes: "4"
      max_parallel_workers_per_gather: "2"
      max_parallel_workers: "4"
      max_parallel_maintenance_workers: "2"
    synchronous:
      method: any
      number: 1
      dataDurability: required
  storage:
    size: 1Gi


PGBench deployment:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbench-runner-cnpg
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pgbench-runner-cnpg
  template:
    metadata:
      labels:
        app: pgbench-runner-cnpg
    spec:
      restartPolicy: Always  # important to avoid infinite restart of completed pods
      containers:
        - name: pgbench
          image: postgres:17
          resources:
            requests:
              cpu: "10m"
              memory: "128Mi"
            limits: {}
          env:
            - name: PGHOST
              valueFrom:
                secretKeyRef:
                  name: cluster-example-app
                  key: host
            - name: PGPORT
              valueFrom:
                secretKeyRef:
                  name: cluster-example-app
                  key: port
            - name: PGUSER
              valueFrom:
                secretKeyRef:
                  name: cluster-example-app
                  key: user
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: cluster-example-app
                  key: password
            - name: PGDATABASE
              valueFrom:
                secretKeyRef:
                  name: cluster-example-app
                  key: dbname
          command: ["sh", "-c"]
          args:
            - |
              echo "Initializing benchmark database if needed...";
              pgbench -i -s 60 -d $PGDATABASE || true;
              echo "Running pgbench...";
              pgbench -j 6 -c 60 -b tpcb-like --time 600 -d $PGDATABASE
```

### Relevant log output

```shell
Postgres instance logs:


{"level":"info","ts":"2025-11-05T15:32:38.4179151Z","msg":"Starting up pg_rewind","logger":"instance-manager","logging_pod":"cluster-separate-wal-storage-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-separate-wal-storage","namespace":"default"},"namespace":"default","name":"cluster-separate-wal-storage","reconcileID":"9063eeb8-8b27-45ec-a1cf-45305a5d63ff","logging_pod":"cluster-separate-wal-storage-1","pgdata":"/var/lib/postgresql/data/pgdata","options":["-P","--source-server","host=cluster-separate-wal-storage-rw user=streaming_replica port=5432 sslkey=/controller/certificates/streaming_replica.key sslcert=/controller/certificates/streaming_replica.crt sslrootcert=/controller/certificates/server-ca.crt application_name=cluster-separate-wal-storage-1 sslmode=verify-ca dbname=postgres","--target-pgdata","/var/lib/postgresql/data/pgdata","--restore-target-wal"]}

{"level":"info","ts":"2025-11-05T15:32:38.46084317Z","logger":"pg_rewind","msg":"pg_rewind: connected to server","pipe":"stderr","logging_pod":"cluster-separate-wal-storage-1"}

{"level":"info","ts":"2025-11-05T15:32:38.463751188Z","logger":"pg_rewind","msg":"pg_rewind: servers diverged at WAL location 1/7F1322D0 on timeline 1","pipe":"stderr","logging_pod":"cluster-separate-wal-storage-1"}

{"level":"info","ts":"2025-11-05T15:32:39.396083475Z","logger":"pg_rewind","msg":"pg_rewind: error: could not restore file \"00000001000000010000005F\" from archive","pipe":"stderr","logging_pod":"cluster-separate-wal-storage-1"}

{"level":"info","ts":"2025-11-05T15:32:39.396124895Z","logger":"pg_rewind","msg":"pg_rewind: error: could not find previous WAL record at 1/5FFFE728","pipe":"stderr","logging_pod":"cluster-separate-wal-storage-1"}

{"level":"error","ts":"2025-11-05T15:32:39.398028104Z","msg":"Failed to execute pg_rewind","logger":"instance-manager","logging_pod":"cluster-separate-wal-storage-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-separate-wal-storage","namespace":"default"},"namespace":"default","name":"cluster-separate-wal-storage","reconcileID":"9063eeb8-8b27-45ec-a1cf-45305a5d63ff","logging_pod":"cluster-separate-wal-storage-1","options":["-P","--source-server","host=cluster-separate-wal-storage-rw user=streaming_replica port=5432 sslkey=/controller/certificates/streaming_replica.key sslcert=/controller/certificates/streaming_replica.crt sslrootcert=/controller/certificates/server-ca.crt application_name=cluster-separate-wal-storage-1 sslmode=verify-ca dbname=postgres","--target-pgdata","/var/lib/postgresql/data/pgdata","--restore-target-wal"],"error":"exit status 1","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*Instance).Rewind\n\tpkg/management/postgres/instance.go:1165\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).verifyPgDataCoherenceForPrimary\n\tinternal/management/controller/instance_startup.go:140\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).initialize\n\tinternal/management/controller/instance_controller.go:457\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).Reconcile\n\tinternal/management/controller/instance_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller...[]).Start.func2.1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}



Plugin logs (using cnpg-plugin-wal-g):

{"level":"error","ts":"2025-11-05T15:32:39Z","logger":"plugin_wal","msg":"Failed to run wal-g fetch 00000001000000010000005F /var/lib/postgresql/data/pgdata/var/lib/postgresql/data/pgdata/pg_wal/00000001000000010000005F","method":"Restore","stdout":"","stderr":"ERROR: 2025/11/05 15:32:39.100804 open /var/lib/postgresql/data/pgdata/var/lib/postgresql/data/pgdata/pg_wal/00000001000000010000005F: no such file or directory\n","error":"subprocess run cmd result exit status 1"}
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Hints from issue comments:
@Endevir: This issue is quite similar to this one: https://github.com/cloudnative-pg/cloudnative-pg/issues/6965 

Solution patch:
diff --git a/internal/cmd/manager/walrestore/cmd.go b/internal/cmd/manager/walrestore/cmd.go
index 080b516d..c77259fd 100644
--- a/internal/cmd/manager/walrestore/cmd.go
+++ b/internal/cmd/manager/walrestore/cmd.go
@@ -25,7 +25,6 @@ import (
 	"errors"
 	"fmt"
 	"os"
-	"path"
 	"strings"
 	"time"
 
@@ -123,7 +122,7 @@ func run(ctx context.Context, pgData string, podName string, args []string) erro
 		return fmt.Errorf("failed to get cluster: %w", err)
 	}
 
-	walFound, err := restoreWALViaPlugins(ctx, cluster, walName, path.Join(pgData, destinationPath))
+	walFound, err := restoreWALViaPlugins(ctx, cluster, walName, pgData, destinationPath)
 	if err != nil {
 		// With the current implementation, this happens when both of the following conditions are met:
 		//
@@ -262,6 +261,7 @@ func restoreWALViaPlugins(
 	ctx context.Context,
 	cluster *apiv1.Cluster,
 	walName string,
+	pgData string,
 	destinationPathName string,
 ) (bool, error) {
 	contextLogger := log.FromContext(ctx)
@@ -282,7 +282,7 @@ func restoreWALViaPlugins(
 	}
 	defer client.Close(ctx)
 
-	return client.RestoreWAL(ctx, cluster, walName, destinationPathName)
+	return client.RestoreWAL(ctx, cluster, walName, postgres.BuildWALPath(pgData, destinationPathName))
 }
 
 // checkEndOfWALStreamFlag returns ErrEndOfWALStreamReached if the flag is set in the restorer
diff --git a/pkg/management/postgres/archiver/archiver.go b/pkg/management/postgres/archiver/archiver.go
index d9750949..d4c2119b 100644
--- a/pkg/management/postgres/archiver/archiver.go
+++ b/pkg/management/postgres/archiver/archiver.go
@@ -282,11 +282,6 @@ func archiveWALViaPlugins(
 ) error {
 	contextLogger := log.FromContext(ctx)
 
-	// check if the `walName` is an absolute path or just the filename
-	if !filepath.IsAbs(walName) {
-		walName = filepath.Join(pgData, walName)
-	}
-
 	plugins := repository.New()
 	defer plugins.Close()
 
@@ -306,7 +301,7 @@ func archiveWALViaPlugins(
 		}
 	}
 
-	return client.ArchiveWAL(ctx, cluster, walName)
+	return client.ArchiveWAL(ctx, cluster, postgres.BuildWALPath(pgData, walName))
 }
 
 // isCheckWalArchiveFlagFilePresent returns true if the file CheckEmptyWalArchiveFile is present in the PGDATA directory
diff --git a/pkg/postgres/wal.go b/pkg/postgres/wal.go
index bf9de310..3f8fd23b 100644
--- a/pkg/postgres/wal.go
+++ b/pkg/postgres/wal.go
@@ -23,6 +23,7 @@ import (
 	"errors"
 	"fmt"
 	"path"
+	"path/filepath"
 	"regexp"
 	"strconv"
 )
@@ -193,3 +194,14 @@ func (segment Segment) NextSegments(size int, postgresVersion *int, segmentSize
 
 	return result
 }
+
+// BuildWALPath constructs the full destination path for WAL operations.
+// If walPath is already absolute, it returns it as-is.
+// If walPath is relative, it joins it with pgData.
+// This prevents path duplication when PostgreSQL or pg_rewind passes absolute paths.
+func BuildWALPath(pgData, walPath string) string {
+	if !filepath.IsAbs(walPath) {
+		return filepath.Join(pgData, walPath)
+	}
+	return walPath
+}



Test patch:
diff --git a/pkg/postgres/wal_test.go b/pkg/postgres/wal_test.go
index 92a7d17c..d7048412 100644
--- a/pkg/postgres/wal_test.go
+++ b/pkg/postgres/wal_test.go
@@ -247,3 +247,66 @@ var _ = Describe("WAL files checking", func() {
 		}
 	})
 })
+
+var _ = Describe("BuildWALPath", func() {
+	const pgData = "/var/lib/postgresql/data/pgdata"
+
+	Context("when walPath is a relative path", func() {
+		It("should join pgData with the relative path", func() {
+			relativePath := "pg_wal/000000010000000000000001"
+			expectedPath := "/var/lib/postgresql/data/pgdata/pg_wal/000000010000000000000001"
+
+			result := BuildWALPath(pgData, relativePath)
+
+			Expect(result).To(Equal(expectedPath))
+		})
+
+		It("should handle just a filename", func() {
+			filename := "000000010000000000000001"
+			expectedPath := "/var/lib/postgresql/data/pgdata/000000010000000000000001"
+
+			result := BuildWALPath(pgData, filename)
+
+			Expect(result).To(Equal(expectedPath))
+		})
+
+		It("should handle subdirectory with filename", func() {
+			relativePath := "pg_wal/archive_status/000000010000000000000001.ready"
+			expectedPath := "/var/lib/postgresql/data/pgdata/pg_wal/archive_status/000000010000000000000001.ready"
+
+			result := BuildWALPath(pgData, relativePath)
+
+			Expect(result).To(Equal(expectedPath))
+		})
+	})
+
+	Context("when walPath is an absolute path", func() {
+		It("should use the absolute path as-is without joining with pgData", func() {
+			absolutePath := "/var/lib/postgresql/data/pgdata/pg_wal/000000010000000000000001"
+
+			result := BuildWALPath(pgData, absolutePath)
+
+			Expect(result).To(Equal(absolutePath))
+		})
+
+		It("should not duplicate pgData when path is already absolute (the bug scenario from #9067)", func() {
+			// This is the scenario from issue #9067: pg_rewind passes an absolute path
+			absolutePath := "/var/lib/postgresql/data/pgdata/pg_wal/00000001000000010000005F"
+
+			result := BuildWALPath(pgData, absolutePath)
+
+			// Verify the bug is fixed: result should be the absolute path, not duplicated
+			Expect(result).To(Equal(absolutePath))
+			// Verify it doesn't create the buggy duplicated path
+			Expect(result).NotTo(ContainSubstring("/var/lib/postgresql/data/pgdata/var/lib/postgresql/data/pgdata"))
+		})
+
+		It("should handle absolute paths from different roots", func() {
+			differentAbsolutePath := "/tmp/wal/000000010000000000000001"
+
+			result := BuildWALPath(pgData, differentAbsolutePath)
+
+			Expect(result).To(Equal(differentAbsolutePath))
+		})
+	})
+})