Repository: cloudnative-pg/cloudnative-pg

Issue:
[Bug]: "exec /manager: argument list too long" in  bootstrap-controller
### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

maxim.i.makarov@gmail.com

### Version

1.24.0

### What version of Kubernetes are you using?

1.28

### What is your Kubernetes environment?

Cloud: Other

### How did you install the operator?

Helm

### What happened?

I have a k8s managed cluster and a namespace with thousands of pods with different VENV. My cnpg worked fine for over a year and suddenly, newly created clusters or rebooted old ones start hanging in pending status due to sidecar container error.
Log:
` exec /manager: argument list too long`
Exactly same cluster manifest worked fine in other namespaces, so the problem relays to bash problem with large amount of VENVs in one namespace.

### Proposed solution:

Add an ability to set Pod parameter: `enableServiceLinks: false` to any created CNPG pod.

Current working solution is to use kyverno: 

```
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disable-service-links-for-database
spec:
  rules:
    - name: set-enable-service-links-false
      match:
        resources:
          kinds:
            - Pod
          namespaces:
            - <some namespace>
      mutate:
        patchesJson6902: |
          - op: replace
            path: /spec/enableServiceLinks
            value: false
```
and with `enableServiceLinks=false` clusters are able to start again 


### Cluster resource

```shell

```

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Hints from issue comments:
@gbartolini: Have you considered reducing the number of services deployed by the operator with `disableDefaultServices`? See https://cloudnative-pg.io/documentation/current/service_management/@gecube: @gbartolini Hi! Thanks for the attention. I want to emphasise that `disableDefaultServices` would not help as the issue is not linked directly to number of services deployed by cnpg, but rather to number of services in the same NS. And these services could be managed by other means. I think the proper solution would be to allow the cluster operator to change some fields from Pod Template Spec in cnpg cluster definition.@mnencia: Now you can fix this issue by using the new `cnpg.io/podPatch` annotation. I.e.,

```
  annotations:
    cnpg.io/podPatch: |
      [
      	{
      		"op": "add",
      		"path": "/spec/enableServiceLinks",
      		"value": false
      	}
      ]
```@simplysoft: Did experience this issue as well. Gave `cnpg.io/podPatch` a try, but it does not seem to fix it in our case. We got the `exec /manager: argument list too long` for the `initdb` jobs


Solution patch:
diff --git a/.wordlist-en-custom.txt b/.wordlist-en-custom.txt
index b80faf7d..6ee17eac 100644
--- a/.wordlist-en-custom.txt
+++ b/.wordlist-en-custom.txt
@@ -661,6 +661,7 @@ configmapkeyselector
 configmaps
 configs
 configurability
+congruency
 conn
 connectionLimit
 connectionParameters
diff --git a/docs/src/labels_annotations.md b/docs/src/labels_annotations.md
index 4ab08190..9acdbb13 100644
--- a/docs/src/labels_annotations.md
+++ b/docs/src/labels_annotations.md
@@ -160,6 +160,19 @@ These predefined annotations are managed by CloudNativePG.
 `cnpg.io/podEnvHash`
 :   Deprecated, as the `cnpg.io/podSpec` annotation now also contains the pod environment.
 
+`cnpg.io/podPatch`
+:   Annotation can be applied on a `Cluster` resource.
+
+    When set to JSON-patch formatted patch, the patch will be applied on the instance Pods.
+
+    **⚠️ WARNING:** This feature may introduce discrepancies between the
+    operator’s expectations and Kubernetes behavior. Use with caution and only as a
+    last resort.
+
+    **IMPORTANT**: adding or changing this annotation won't trigger a rolling deployment
+    of the generated Pods. The latter can be triggered manually by the user with
+    `kubectl cnpg restart`.
+
 `cnpg.io/podSpec`
 :   Snapshot of the `spec` of the pod generated by the operator. This annotation replaces
     the old, deprecated `cnpg.io/podEnvHash` annotation.
diff --git a/internal/controller/cluster_create.go b/internal/controller/cluster_create.go
index 39029dd4..2a88deec 100644
--- a/internal/controller/cluster_create.go
+++ b/internal/controller/cluster_create.go
@@ -1443,7 +1443,7 @@ func findInstancePodToCreate(
 		if err != nil {
 			return nil, err
 		}
-		return specs.PodWithExistingStorage(*cluster, serial), nil
+		return specs.PodWithExistingStorage(*cluster, serial)
 	}
 
 	return nil, nil
diff --git a/internal/webhook/v1/cluster_webhook.go b/internal/webhook/v1/cluster_webhook.go
index fcba1b06..dc7fd675 100644
--- a/internal/webhook/v1/cluster_webhook.go
+++ b/internal/webhook/v1/cluster_webhook.go
@@ -30,6 +30,7 @@ import (
 	"github.com/cloudnative-pg/machinery/pkg/postgres/version"
 	"github.com/cloudnative-pg/machinery/pkg/stringset"
 	"github.com/cloudnative-pg/machinery/pkg/types"
+	jsonpatch "github.com/evanphx/json-patch/v5"
 	storagesnapshotv1 "github.com/kubernetes-csi/external-snapshotter/client/v8/apis/volumesnapshot/v1"
 	corev1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
@@ -46,6 +47,7 @@ import (
 
 	apiv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1"
 	"github.com/cloudnative-pg/cloudnative-pg/pkg/postgres"
+	"github.com/cloudnative-pg/cloudnative-pg/pkg/specs"
 	"github.com/cloudnative-pg/cloudnative-pg/pkg/utils"
 )
 
@@ -206,6 +208,7 @@ func (v *ClusterCustomValidator) validate(r *apiv1.Cluster) (allErrs field.Error
 		v.validateManagedExtensions,
 		v.validateResources,
 		v.validateHibernationAnnotation,
+		v.validatePodPatchAnnotation,
 		v.validatePromotionToken,
 	}
 
@@ -2350,3 +2353,32 @@ func (v *ClusterCustomValidator) validateHibernationAnnotation(r *apiv1.Cluster)
 		),
 	}
 }
+
+func (v *ClusterCustomValidator) validatePodPatchAnnotation(r *apiv1.Cluster) field.ErrorList {
+	jsonPatch, ok := r.Annotations[utils.PodPatchAnnotationName]
+	if !ok {
+		return nil
+	}
+
+	if _, err := jsonpatch.DecodePatch([]byte(jsonPatch)); err != nil {
+		return field.ErrorList{
+			field.Invalid(
+				field.NewPath("metadata", "annotations", utils.PodPatchAnnotationName),
+				jsonPatch,
+				fmt.Sprintf("error decoding JSON patch: %s", err.Error()),
+			),
+		}
+	}
+
+	if _, err := specs.PodWithExistingStorage(*r, 1); err != nil {
+		return field.ErrorList{
+			field.Invalid(
+				field.NewPath("metadata", "annotations", utils.PodPatchAnnotationName),
+				jsonPatch,
+				fmt.Sprintf("jsonpatch doesn't apply cleanly to the pod: %s", err.Error()),
+			),
+		}
+	}
+
+	return nil
+}
diff --git a/pkg/specs/pods.go b/pkg/specs/pods.go
index b20b704f..41482b5d 100644
--- a/pkg/specs/pods.go
+++ b/pkg/specs/pods.go
@@ -27,6 +27,7 @@ import (
 	"slices"
 	"strconv"
 
+	jsonpatch "github.com/evanphx/json-patch/v5"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/intstr"
@@ -433,7 +434,7 @@ func CreatePodSecurityContext(seccompProfile *corev1.SeccompProfile, user, group
 }
 
 // PodWithExistingStorage create a new instance with an existing storage
-func PodWithExistingStorage(cluster apiv1.Cluster, nodeSerial int) *corev1.Pod {
+func PodWithExistingStorage(cluster apiv1.Cluster, nodeSerial int) (*corev1.Pod, error) {
 	podName := GetInstanceName(cluster.Name, nodeSerial)
 	gracePeriod := int64(cluster.GetMaxStopDelay())
 
@@ -474,7 +475,28 @@ func PodWithExistingStorage(cluster apiv1.Cluster, nodeSerial int) *corev1.Pod {
 	if utils.IsAnnotationAppArmorPresent(&pod.Spec, cluster.Annotations) {
 		utils.AnnotateAppArmor(&pod.ObjectMeta, &pod.Spec, cluster.Annotations)
 	}
-	return pod
+
+	if jsonPatch := cluster.Annotations[utils.PodPatchAnnotationName]; jsonPatch != "" {
+		serializedObject, err := json.Marshal(pod)
+		if err != nil {
+			return nil, fmt.Errorf("while serializing pod to JSON: %w", err)
+		}
+		patch, err := jsonpatch.DecodePatch([]byte(jsonPatch))
+		if err != nil {
+			return nil, fmt.Errorf("while decoding JSON patch from annotation: %w", err)
+		}
+
+		serializedObject, err = patch.Apply(serializedObject)
+		if err != nil {
+			return nil, fmt.Errorf("while applying JSON patch from annotation: %w", err)
+		}
+
+		if err = json.Unmarshal(serializedObject, pod); err != nil {
+			return nil, fmt.Errorf("while deserializing pod to JSON: %w", err)
+		}
+	}
+
+	return pod, nil
 }
 
 // GetInstanceName returns a string indicating the instance name
diff --git a/pkg/utils/labels_annotations.go b/pkg/utils/labels_annotations.go
index 18732501..028ab14c 100644
--- a/pkg/utils/labels_annotations.go
+++ b/pkg/utils/labels_annotations.go
@@ -233,6 +233,10 @@ const (
 	// PluginPortAnnotationName is the name of the annotation containing the
 	// port the plugin is listening to
 	PluginPortAnnotationName = MetadataNamespace + "/pluginPort"
+
+	// PodPatchAnnotationName is the name of the annotation containing the
+	// patch to apply to the pod
+	PodPatchAnnotationName = MetadataNamespace + "/podPatch"
 )
 
 type annotationStatus string



Test patch:
diff --git a/internal/controller/cluster_upgrade_test.go b/internal/controller/cluster_upgrade_test.go
index 2037ce12..a1540166 100644
--- a/internal/controller/cluster_upgrade_test.go
+++ b/internal/controller/cluster_upgrade_test.go
@@ -57,7 +57,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("will not require a restart for just created Pods", func(ctx SpecContext) {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 
 		status := postgres.PostgresqlStatus{
 			Pod:            pod,
@@ -71,7 +72,9 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("requires rollout when running a different image name", func(ctx SpecContext) {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
+
 		pod.Spec.Containers[0].Image = "postgres:13.10"
 		status := postgres.PostgresqlStatus{
 			Pod:            pod,
@@ -86,7 +89,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("requires rollout when a restart annotation has been added to the cluster", func(ctx SpecContext) {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 		clusterRestart := cluster
 		clusterRestart.Annotations = make(map[string]string)
 		clusterRestart.Annotations[utils.ClusterRestartAnnotationName] = "now"
@@ -110,7 +114,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("should prioritize full rollout over inplace restarts", func(ctx SpecContext) {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 
 		status := postgres.PostgresqlStatus{
 			Pod:            pod,
@@ -140,7 +145,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("requires rollout when PostgreSQL needs to be restarted", func(ctx SpecContext) {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 
 		status := postgres.PostgresqlStatus{
 			Pod:            pod,
@@ -166,7 +172,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("requires pod rollout if executable does not have a hash", func(ctx SpecContext) {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 		status := postgres.PostgresqlStatus{
 			Pod:            pod,
 			PendingRestart: false,
@@ -181,7 +188,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("checkPodSpecIsOutdated should not return any error", func() {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 		rollout, err := checkPodSpecIsOutdated(pod, &cluster)
 		Expect(rollout.required).To(BeFalse())
 		Expect(rollout.canBeInPlace).To(BeFalse())
@@ -190,7 +198,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 	})
 
 	It("checks when a rollout is needed for any reason", func(ctx SpecContext) {
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 		status := postgres.PostgresqlStatus{
 			Pod:            pod,
 			PendingRestart: true,
@@ -216,7 +225,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 
 	When("the PodSpec annotation is not available", func() {
 		It("should trigger a rollout when the scheduler changes", func(ctx SpecContext) {
-			pod := specs.PodWithExistingStorage(cluster, 1)
+			pod, err := specs.PodWithExistingStorage(cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 			cluster.Spec.SchedulerName = "newScheduler"
 			delete(pod.Annotations, utils.PodSpecAnnotationName)
 
@@ -241,7 +251,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 				ImageName: "postgres:13.11",
 			},
 		}
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 		cluster.Spec.SchedulerName = "newScheduler"
 
 		status := postgres.PostgresqlStatus{
@@ -272,7 +283,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 			},
 		}
 		It("should trigger a rollout when the cluster has a Resource changed", func(ctx SpecContext) {
-			pod := specs.PodWithExistingStorage(clusterWithResources, 1)
+			pod, err := specs.PodWithExistingStorage(clusterWithResources, 1)
+			Expect(err).ToNot(HaveOccurred())
 			clusterWithResources.Spec.Resources.Limits["cpu"] = resource.MustParse("3") // was "2"
 
 			status := postgres.PostgresqlStatus{
@@ -290,7 +302,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 			Expect(rollout.needsChangeOperatorImage).To(BeFalse())
 		})
 		It("should trigger a rollout when the cluster has Resources deleted from spec", func(ctx SpecContext) {
-			pod := specs.PodWithExistingStorage(clusterWithResources, 1)
+			pod, err := specs.PodWithExistingStorage(clusterWithResources, 1)
+			Expect(err).ToNot(HaveOccurred())
 			clusterWithResources.Spec.Resources = corev1.ResourceRequirements{}
 
 			status := postgres.PostgresqlStatus{
@@ -311,7 +324,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 
 	When("the PodSpec annotation is not available", func() {
 		It("detects when a new custom environment variable is set", func(ctx SpecContext) {
-			pod := specs.PodWithExistingStorage(cluster, 1)
+			pod, err := specs.PodWithExistingStorage(cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 			delete(pod.Annotations, utils.PodSpecAnnotationName)
 
 			cluster := cluster.DeepCopy()
@@ -341,7 +355,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 					ImageName: "postgres:13.11",
 				},
 			}
-			pod := specs.PodWithExistingStorage(cluster, 1)
+			pod, err := specs.PodWithExistingStorage(cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 			delete(pod.Annotations, utils.PodSpecAnnotationName)
 
 			status := postgres.PostgresqlStatus{
@@ -365,7 +380,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 					ImageName: "postgres:13.11",
 				},
 			}
-			pod := specs.PodWithExistingStorage(cluster, 1)
+			pod, err := specs.PodWithExistingStorage(cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 			delete(pod.Annotations, utils.PodSpecAnnotationName)
 
 			status := postgres.PostgresqlStatus{
@@ -388,7 +404,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 
 	When("the podSpec annotation is available", func() {
 		It("detects when a new custom environment variable is set", func(ctx SpecContext) {
-			pod := specs.PodWithExistingStorage(cluster, 1)
+			pod, err := specs.PodWithExistingStorage(cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 
 			cluster := cluster.DeepCopy()
 			cluster.Spec.Env = []corev1.EnvVar{
@@ -418,7 +435,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 					ImageName: "postgres:13.11",
 				},
 			}
-			pod := specs.PodWithExistingStorage(cluster, 1)
+			pod, err := specs.PodWithExistingStorage(cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 
 			status := postgres.PostgresqlStatus{
 				Pod:            pod,
@@ -441,7 +459,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 					ImageName: "postgres:13.11",
 				},
 			}
-			pod := specs.PodWithExistingStorage(cluster, 1)
+			pod, err := specs.PodWithExistingStorage(cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 
 			status := postgres.PostgresqlStatus{
 				Pod:            pod,
@@ -467,7 +486,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 				cluster.Spec.ProjectedVolumeTemplate = &corev1.ProjectedVolumeSource{
 					Sources: []corev1.VolumeProjection{},
 				}
-				pod := specs.PodWithExistingStorage(cluster, 1)
+				pod, err := specs.PodWithExistingStorage(cluster, 1)
+				Expect(err).ToNot(HaveOccurred())
 				status := postgres.PostgresqlStatus{
 					Pod:            pod,
 					IsPodReady:     true,
@@ -484,7 +504,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 				cluster.Spec.ProjectedVolumeTemplate = &corev1.ProjectedVolumeSource{
 					Sources: nil,
 				}
-				pod := specs.PodWithExistingStorage(cluster, 1)
+				pod, err := specs.PodWithExistingStorage(cluster, 1)
+				Expect(err).ToNot(HaveOccurred())
 				status := postgres.PostgresqlStatus{
 					Pod:            pod,
 					IsPodReady:     true,
@@ -499,7 +520,8 @@ var _ = Describe("Pod upgrade", Ordered, func() {
 		It("should not require rollout if projected volume  is nil",
 			func(ctx SpecContext) {
 				cluster.Spec.ProjectedVolumeTemplate = nil
-				pod := specs.PodWithExistingStorage(cluster, 1)
+				pod, err := specs.PodWithExistingStorage(cluster, 1)
+				Expect(err).ToNot(HaveOccurred())
 				status := postgres.PostgresqlStatus{
 					Pod:            pod,
 					IsPodReady:     true,
@@ -531,7 +553,9 @@ var _ = Describe("Test pod rollout due to topology", func() {
 				TopologySpreadConstraints: []corev1.TopologySpreadConstraint{topology},
 			},
 		}
-		pod = specs.PodWithExistingStorage(*cluster, 1)
+		var err error
+		pod, err = specs.PodWithExistingStorage(*cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 	})
 
 	When("the original podSpec annotation is available", func() {
@@ -596,7 +620,9 @@ var _ = Describe("Test pod rollout due to topology", func() {
 
 		It("should not require rollout if pod and spec both lack TopologySpreadConstraints", func(ctx SpecContext) {
 			cluster.Spec.TopologySpreadConstraints = nil
-			pod = specs.PodWithExistingStorage(*cluster, 1)
+			var err error
+			pod, err = specs.PodWithExistingStorage(*cluster, 1)
+			Expect(err).ToNot(HaveOccurred())
 			Expect(pod.Spec.TopologySpreadConstraints).To(BeNil())
 
 			status := postgres.PostgresqlStatus{
@@ -755,7 +781,8 @@ var _ = Describe("Cluster upgrade with podSpec reconciliation disabled", func()
 	It("skips the rollout if the annotation that disables PodSpec reconciliation is set", func(ctx SpecContext) {
 		cluster.ObjectMeta.Annotations[utils.ReconcilePodSpecAnnotationName] = "disabled"
 
-		pod := specs.PodWithExistingStorage(cluster, 1)
+		pod, err := specs.PodWithExistingStorage(cluster, 1)
+		Expect(err).ToNot(HaveOccurred())
 		cluster.Spec.SchedulerName = "newScheduler"
 		delete(pod.Annotations, utils.PodSpecAnnotationName)
 
diff --git a/internal/controller/suite_test.go b/internal/controller/suite_test.go
index ae8f8e86..5f1396d8 100644
--- a/internal/controller/suite_test.go
+++ b/internal/controller/suite_test.go
@@ -313,7 +313,7 @@ func generateFakeClusterPods(
 	var pods []corev1.Pod
 	for idx < cluster.Spec.Instances {
 		idx++
-		pod := specs.PodWithExistingStorage(*cluster, idx)
+		pod, _ := specs.PodWithExistingStorage(*cluster, idx)
 		cluster.SetInheritedDataAndOwnership(&pod.ObjectMeta)
 
 		err := c.Create(context.Background(), pod)
diff --git a/internal/webhook/v1/cluster_webhook_test.go b/internal/webhook/v1/cluster_webhook_test.go
index faa1f0f0..a6828768 100644
--- a/internal/webhook/v1/cluster_webhook_test.go
+++ b/internal/webhook/v1/cluster_webhook_test.go
@@ -4884,3 +4884,56 @@ var _ = Describe("ServiceTemplate Validation", func() {
 		})
 	})
 })
+
+var _ = Describe("validatePodPatchAnnotation", func() {
+	var v *ClusterCustomValidator
+
+	It("returns nil if the annotation is not present", func() {
+		cluster := &apiv1.Cluster{ObjectMeta: metav1.ObjectMeta{Annotations: map[string]string{}}}
+		Expect(v.validatePodPatchAnnotation(cluster)).To(BeNil())
+	})
+
+	It("returns an error if decoding the JSON patch fails to decode", func() {
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					utils.PodPatchAnnotationName: "invalid-json-patch",
+				},
+			},
+		}
+
+		errors := v.validatePodPatchAnnotation(cluster)
+		Expect(errors).To(HaveLen(1))
+		Expect(errors[0].Type).To(Equal(field.ErrorTypeInvalid))
+		Expect(errors[0].Field).To(Equal("metadata.annotations." + utils.PodPatchAnnotationName))
+		Expect(errors[0].Detail).To(ContainSubstring("error decoding JSON patch"))
+	})
+
+	It("returns an error if decoding the JSON patch fails to apply", func() {
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					utils.PodPatchAnnotationName: `[{"op": "replace", "path": "/spec/podInvalidSection", "value": "test"}]`,
+				},
+			},
+		}
+
+		errors := v.validatePodPatchAnnotation(cluster)
+		Expect(errors).To(HaveLen(1))
+		Expect(errors[0].Type).To(Equal(field.ErrorTypeInvalid))
+		Expect(errors[0].Field).To(Equal("metadata.annotations." + utils.PodPatchAnnotationName))
+		Expect(errors[0].Detail).To(ContainSubstring("jsonpatch doesn't apply cleanly to the pod"))
+	})
+
+	It("returns nil if the JSON patch is decoded successfully", func() {
+		cluster := &apiv1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					utils.PodPatchAnnotationName: `[{"op": "replace", "path": "/metadata/name", "value": "test"}]`,
+				},
+			},
+		}
+
+		Expect(v.validatePodPatchAnnotation(cluster)).To(BeNil())
+	})
+})
diff --git a/pkg/specs/pg_pods_test.go b/pkg/specs/pg_pods_test.go
index f7a5794e..4fc73269 100644
--- a/pkg/specs/pg_pods_test.go
+++ b/pkg/specs/pg_pods_test.go
@@ -33,7 +33,8 @@ var _ = Describe("Extract the used image name", func() {
 			Namespace: "default",
 		},
 	}
-	pod := PodWithExistingStorage(cluster, 1)
+	pod, err := PodWithExistingStorage(cluster, 1)
+	Expect(err).ToNot(HaveOccurred())
 
 	It("extract the default image name", func() {
 		Expect(GetPostgresImageName(*pod)).To(Equal(configuration.Current.PostgresImageName))
diff --git a/pkg/specs/pods_test.go b/pkg/specs/pods_test.go
index ff4a9c48..de99a9a0 100644
--- a/pkg/specs/pods_test.go
+++ b/pkg/specs/pods_test.go
@@ -26,6 +26,7 @@ import (
 
 	v1 "github.com/cloudnative-pg/cloudnative-pg/api/v1"
 	"github.com/cloudnative-pg/cloudnative-pg/pkg/postgres"
+	"github.com/cloudnative-pg/cloudnative-pg/pkg/utils"
 
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
@@ -926,3 +927,38 @@ var _ = Describe("Compute startup probe failure threshold", func() {
 		Expect(getFailureThreshold(31, LivenessProbePeriod)).To(BeNumerically("==", 4))
 	})
 })
+
+var _ = Describe("PodWithExistingStorage", func() {
+	It("applies JSON patch from annotation", func() {
+		cluster := v1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-cluster",
+				Namespace: "default",
+				Annotations: map[string]string{
+					utils.PodPatchAnnotationName: `[{"op": "replace", "path": "/spec/containers/0/image", "value": "new-image:latest"}]`, // nolint: lll
+				},
+			},
+		}
+
+		pod, err := PodWithExistingStorage(cluster, 1)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(pod).NotTo(BeNil())
+		Expect(pod.Spec.Containers[0].Image).To(Equal("new-image:latest"))
+	})
+
+	It("returns error if JSON patch is invalid", func() {
+		cluster := v1.Cluster{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-cluster",
+				Namespace: "default",
+				Annotations: map[string]string{
+					utils.PodPatchAnnotationName: `invalid-json-patch`,
+				},
+			},
+		}
+
+		_, err := PodWithExistingStorage(cluster, 1)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("while decoding JSON patch from annotation"))
+	})
+})
diff --git a/tests/e2e/pod_patch_test.go b/tests/e2e/pod_patch_test.go
new file mode 100644
index 00000000..3ad5a84e
--- /dev/null
+++ b/tests/e2e/pod_patch_test.go
@@ -0,0 +1,101 @@
+/*
+Copyright The CloudNativePG Contributors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package e2e
+
+import (
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	"github.com/cloudnative-pg/cloudnative-pg/pkg/utils"
+	"github.com/cloudnative-pg/cloudnative-pg/tests"
+	"github.com/cloudnative-pg/cloudnative-pg/tests/utils/clusterutils"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Pod patch", Label(tests.LabelSmoke, tests.LabelBasic), func() {
+	const (
+		sampleFile  = fixturesDir + "/base/cluster-storage-class.yaml.template"
+		clusterName = "postgresql-storage-class"
+		level       = tests.Lowest
+	)
+
+	var namespace string
+
+	BeforeEach(func() {
+		if testLevelEnv.Depth < int(level) {
+			Skip("Test depth is lower than the amount requested for this test")
+		}
+	})
+
+	It("use the podPatch annotation to generate Pods", func(_ SpecContext) {
+		const namespacePrefix = "cluster-patch-e2e"
+		var err error
+
+		namespace, err = env.CreateUniqueTestNamespace(env.Ctx, env.Client, namespacePrefix)
+		Expect(err).ToNot(HaveOccurred())
+
+		AssertCreateCluster(namespace, clusterName, sampleFile, env)
+
+		By("adding the podPatch annotation", func() {
+			cluster, err := clusterutils.Get(env.Ctx, env.Client, namespace, clusterName)
+			Expect(err).ToNot(HaveOccurred())
+
+			patchedCluster := cluster.DeepCopy()
+
+			patchedCluster.SetAnnotations(map[string]string{
+				utils.PodPatchAnnotationName: `
+					[
+						{
+							"op": "add",
+							"path": "/metadata/annotations/e2e.cnpg.io",
+							"value": "this-test"
+						}
+					]
+				`,
+			})
+			err = env.Client.Patch(env.Ctx, patchedCluster, client.MergeFrom(cluster))
+			Expect(err).ToNot(HaveOccurred())
+		})
+
+		By("deleting all the Pods", func() {
+			podList, err := clusterutils.ListPods(env.Ctx, env.Client, namespace, clusterName)
+			Expect(err).ToNot(HaveOccurred())
+
+			for i := range podList.Items {
+				err := env.Client.Delete(env.Ctx, &podList.Items[i])
+				Expect(err).ToNot(HaveOccurred())
+			}
+		})
+
+		By("waiting for the new annotation to be applied to the new Pods", func() {
+			cluster, err := clusterutils.Get(env.Ctx, env.Client, namespace, clusterName)
+			Expect(err).ToNot(HaveOccurred())
+
+			timeout := 120
+			Eventually(func(g Gomega) {
+				podList, err := clusterutils.ListPods(env.Ctx, env.Client, namespace, clusterName)
+				g.Expect(err).ToNot(HaveOccurred())
+				g.Expect(podList.Items).To(HaveLen(cluster.Spec.Instances))
+
+				for _, pod := range podList.Items {
+					g.Expect(pod.Annotations).To(HaveKeyWithValue("e2e.cnpg.io", "this-test"))
+				}
+			}, timeout).Should(Succeed())
+		})
+	})
+})