Repository: cloudnative-pg/cloudnative-pg

Issue:
[Bug]: Errors scaling up after an in-place major upgrade when VolumeSnapshots backup are available
### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

_No response_

### Version

1.26 (latest patch)

### What version of Kubernetes are you using?

1.33

### What is your Kubernetes environment?

Self-managed: kind (evaluation)

### How did you install the operator?

YAML manifest

### What happened?

When valid volume snapshot backups are present, after a successful major version upgrade, the creation of the replicas tries to use the snapshots taken before the major upgrade, leading to a failure:

```
{
  "level": "info",
  "ts": "2025-05-29T20:16:42.497601464Z",
  "logger": "postgres",
  "msg": "2025-05-29 20:16:42 UTC FATAL:  database files are incompatible with server",
  "pipe": "stderr",
  "logging_pod": "cluster-example-4"
}
{
  "level": "info",
  "ts": "2025-05-29T20:16:42.497639304Z",
  "logger": "postgres",
  "msg": "2025-05-29 20:16:42 UTC DETAIL:  The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.5.",
  "pipe": "stderr",
  "logging_pod": "cluster-example-4"
}
```

### Cluster resource

```shell

```

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Discussion:
@dosubot: <!-- Answer -->
There is currently no existing issue in the CloudNativePG repository that matches the problem you described, where scaling up after an in-place major upgrade with available VolumeSnapshot backups causes new replicas to fail due to database file incompatibility.

Based on the code and documentation, CloudNativePG uses the most recent completed VolumeSnapshot backup as the source when creating new replica PVCs. If a snapshot taken before the major upgrade is still available, the operator may select it, resulting in a data directory initialized by the old PostgreSQL version (e.g., 16) being used with the new version (e.g., 17.5), which leads to the "database files are incompatible with server" error you observed. This is consistent with the logic in the code that selects candidate storage sources for new replicas, as it does not appear to filter out backups from previous major versions when scaling up after an upgrade [[source](https://github.com/cloudnative-pg/cloudnative-pg/blob/f361b72a31340eda65ee2d7081235bfc9b388af0/pkg/reconciler/persistentvolumeclaim/storagesource.go)].

The documentation confirms that CloudNativePG leverages Kubernetes VolumeSnapshots for backup and recovery, but it is up to the user to ensure that only compatible snapshots are available for restore operations [[source](https://cloudnative-pg.io/documentation/1.25/backup_volumesnapshot)].

This behavior suggests a gap in the current implementation: after a major upgrade, old VolumeSnapshot backups remain available and may be incorrectly reused for new replicas, causing the incompatibility error. As a workaround, you should manually remove or archive any VolumeSnapshot backups created before the major upgrade to prevent the operator from selecting them when scaling up. This will ensure that only snapshots compatible with the current PostgreSQL version are used.

If you need to automate this, consider scripting the cleanup of old snapshots as part of your upgrade process. If you believe this should be handled automatically by the operator, it would be reasonable to open a feature request or bug report describing this scenario in detail.


<!-- Dosu Comment Footer -->
*To reply, just mention my name [@dosu](https://go.dosu.dev/dosubot).*

---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/3f0ec13d-1fef-4250-a196-c080d31ec6b1?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/3f0ec13d-1fef-4250-a196-c080d31ec6b1?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/3f0ec13d-1fef-4250-a196-c080d31ec6b1?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/3f0ec13d-1fef-4250-a196-c080d31ec6b1?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/3f0ec13d-1fef-4250-a196-c080d31ec6b1?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/3f0ec13d-1fef-4250-a196-c080d31ec6b1?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/3f0ec13d-1fef-4250-a196-c080d31ec6b1?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/cloudnative-pg/cloudnative-pg/issues/7705)@dragoangel: have same issue on 1.27.0 😓

Solution patch:
diff --git a/pkg/reconciler/persistentvolumeclaim/storagesource.go b/pkg/reconciler/persistentvolumeclaim/storagesource.go
index d3706f0a..43c448d8 100644
--- a/pkg/reconciler/persistentvolumeclaim/storagesource.go
+++ b/pkg/reconciler/persistentvolumeclaim/storagesource.go
@@ -25,7 +25,6 @@ import (
 	"github.com/cloudnative-pg/machinery/pkg/log"
 	volumesnapshot "github.com/kubernetes-csi/external-snapshotter/client/v8/apis/volumesnapshot/v1"
 	corev1 "k8s.io/api/core/v1"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/utils/ptr"
 
 	apiv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1"
@@ -95,7 +94,7 @@ func GetCandidateStorageSourceForReplica(
 
 	if result := getCandidateSourceFromBackupList(
 		ctx,
-		cluster.CreationTimestamp,
+		cluster,
 		backupList,
 	); result != nil {
 		return result
@@ -117,25 +116,57 @@ func GetCandidateStorageSourceForReplica(
 // given a backup list
 func getCandidateSourceFromBackupList(
 	ctx context.Context,
-	clusterCreationTime metav1.Time,
+	cluster *apiv1.Cluster,
 	backupList apiv1.BackupList,
 ) *StorageSource {
 	contextLogger := log.FromContext(ctx)
 
+	majorVersion, err := cluster.GetPostgresqlMajorVersion()
+	if err != nil {
+		contextLogger.Warning(
+			"unable to determine cluster major version; skipping backup as a recovery source",
+			"error", err.Error(),
+		)
+		return nil
+	}
+
+	isCorrectMajorVersion := func(backup *apiv1.Backup) bool {
+		// If we don't have image info, we can't determine the cluster version reliably; skip enforcement
+		if cluster.Status.PGDataImageInfo == nil {
+			return true
+		}
+
+		backupMajorVersion := backup.Status.MajorVersion
+		if backupMajorVersion == 0 {
+			contextLogger.Warning(
+				"majorVersion on backup status is not populated, cannot use it as a recovery source.",
+			)
+			return false
+		}
+
+		return majorVersion == backupMajorVersion
+	}
+
 	backupList.SortByReverseCreationTime()
 	for idx := range backupList.Items {
 		backup := &backupList.Items[idx]
-		contextLogger := contextLogger.WithValues()
 
 		if !backup.IsCompletedVolumeSnapshot() {
 			contextLogger.Trace("skipping backup, not a valid storage source candidate")
 			continue
 		}
 
-		if backup.CreationTimestamp.Before(&clusterCreationTime) {
+		if backup.CreationTimestamp.Before(&cluster.CreationTimestamp) {
+			contextLogger.Info(
+				"skipping backup as a potential recovery storage source candidate because it was created before the Cluster object",
+			)
+			continue
+		}
+
+		if !isCorrectMajorVersion(backup) {
 			contextLogger.Info(
-				"skipping backup as a potential recovery storage source candidate " +
-					"because if was created before the Cluster object")
+				"skipping backup as a potential recovery storage source candidate because of major version mismatch",
+			)
 			continue
 		}
 



Test patch:
diff --git a/pkg/reconciler/persistentvolumeclaim/storagesource_test.go b/pkg/reconciler/persistentvolumeclaim/storagesource_test.go
index 5044a0c9..172f9788 100644
--- a/pkg/reconciler/persistentvolumeclaim/storagesource_test.go
+++ b/pkg/reconciler/persistentvolumeclaim/storagesource_test.go
@@ -315,7 +315,12 @@ var _ = Describe("candidate backups", func() {
 		}
 		backupList.SortByReverseCreationTime()
 
-		source := getCandidateSourceFromBackupList(ctx, metav1.NewTime(time.Now().Add(-1*time.Hour)), backupList)
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				CreationTimestamp: metav1.NewTime(time.Now().Add(-1 * time.Hour)),
+			},
+		}
+		source := getCandidateSourceFromBackupList(ctx, cluster, backupList)
 		Expect(source).ToNot(BeNil())
 		Expect(source.DataSource.Name).To(Equal("completed-backup"))
 	})
@@ -331,7 +336,111 @@ var _ = Describe("candidate backups", func() {
 		}
 		backupList.SortByReverseCreationTime()
 
-		source := getCandidateSourceFromBackupList(ctx, metav1.NewTime(time.Now().Add(1*time.Hour)), backupList)
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				CreationTimestamp: metav1.NewTime(time.Now().Add(1 * time.Hour)),
+			},
+		}
+		source := getCandidateSourceFromBackupList(ctx, cluster, backupList)
+		Expect(source).To(BeNil())
+	})
+})
+
+var _ = Describe("major version filtering in candidate backup selection", func() {
+	makeCompletedSnapshotBackup := func(name string) apiv1.Backup {
+		return apiv1.Backup{
+			ObjectMeta: metav1.ObjectMeta{
+				CreationTimestamp: metav1.NewTime(time.Now()),
+				Name:              name,
+			},
+			Spec: apiv1.BackupSpec{
+				Method: apiv1.BackupMethodVolumeSnapshot,
+			},
+			Status: apiv1.BackupStatus{
+				Phase: apiv1.BackupPhaseCompleted,
+				BackupSnapshotStatus: apiv1.BackupSnapshotStatus{
+					Elements: []apiv1.BackupSnapshotElementStatus{
+						{
+							Name: "completed-backup",
+							Type: string(utils.PVCRolePgData),
+						},
+					},
+				},
+			},
+		}
+	}
+
+	It("does not filter by major version when PGDataImageInfo is nil", func(ctx context.Context) {
+		backup := makeCompletedSnapshotBackup("no-major-version")
+		// Explicitly leave MajorVersion=0 and no annotation to simulate missing version
+		backupList := apiv1.BackupList{Items: []apiv1.Backup{backup}}
+
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				CreationTimestamp: metav1.NewTime(time.Now().Add(-1 * time.Hour)),
+			},
+			Spec: apiv1.ClusterSpec{
+				Backup: &apiv1.BackupConfiguration{
+					BarmanObjectStore: &apiv1.BarmanObjectStoreConfiguration{
+						DestinationPath: "s3://test",
+					},
+				},
+			},
+			// Status.PGDataImageInfo is nil here on purpose
+		}
+
+		source := getCandidateSourceFromBackupList(ctx, cluster, backupList)
+		Expect(source).ToNot(BeNil())
+		Expect(source.DataSource.Name).To(Equal("completed-backup"))
+	})
+
+	It("skips backup when PGDataImageInfo is set and backup major version is missing", func(ctx context.Context) {
+		backup := makeCompletedSnapshotBackup("missing-major-version")
+		// MajorVersion=0 and no annotation, will be rejected when PGDataImageInfo != nil
+		backupList := apiv1.BackupList{Items: []apiv1.Backup{backup}}
+
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				CreationTimestamp: metav1.NewTime(time.Now().Add(-1 * time.Hour)),
+			},
+			Spec: apiv1.ClusterSpec{
+				Backup: &apiv1.BackupConfiguration{
+					BarmanObjectStore: &apiv1.BarmanObjectStoreConfiguration{
+						DestinationPath: "s3://test",
+					},
+				},
+			},
+			Status: apiv1.ClusterStatus{
+				PGDataImageInfo: &apiv1.ImageInfo{},
+			},
+		}
+
+		source := getCandidateSourceFromBackupList(ctx, cluster, backupList)
+		Expect(source).To(BeNil())
+	})
+
+	It("skips backup when PGDataImageInfo is set and major version mismatches", func(ctx context.Context) {
+		backup := makeCompletedSnapshotBackup("mismatching-major-version")
+		backup.Status.MajorVersion = 99 // intentionally mismatching
+		backupList := apiv1.BackupList{Items: []apiv1.Backup{backup}}
+
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				CreationTimestamp: metav1.NewTime(time.Now().Add(-1 * time.Hour)),
+			},
+			Spec: apiv1.ClusterSpec{
+				Backup: &apiv1.BackupConfiguration{
+					BarmanObjectStore: &apiv1.BarmanObjectStoreConfiguration{
+						DestinationPath: "s3://test",
+					},
+				},
+			},
+			Status: apiv1.ClusterStatus{
+				PGDataImageInfo: &apiv1.ImageInfo{},
+			},
+		}
+
+		source := getCandidateSourceFromBackupList(ctx, cluster, backupList)
 		Expect(source).To(BeNil())
 	})
 })