Repository: cloudnative-pg/cloudnative-pg

Issue:
[Bug]: PG container restarting on an isolated node
### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

leonardo.cecchi@enterprisedb.com

### Version

1.26 (latest patch)

### What version of Kubernetes are you using?

1.33

### What is your Kubernetes environment?

Self-managed: kind (evaluation)

### How did you install the operator?

YAML manifest

### What happened?

The default configuration may trigger automatic shutdown of PostgreSQL instances running on isolated nodes, even if the isolation checker is not active.

This happens because the liveness probe reaches the API server [to detect the current configuration](https://github.com/cloudnative-pg/cloudnative-pg/blob/7f8ddf4e09cb8b88fd460ee2409d94bf160f2591/pkg/management/postgres/webserver/probes/liveness.go#L56), and that request may take more time than the configured probe timeout, whose default is 5 seconds.

When this happens, the liveness probe fails with a timeout, and the container may be restarted.

### Cluster resource

```shell
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:13

  storage:
    size: 1Gi
```

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Discussion:
@dosubot: <!-- Answer -->
This is a known limitation with the default liveness probe in CloudNativePG: it reaches out to the Kubernetes API server, and if the node is isolated or the API server is slow/unreachable, the probe can fail and cause unnecessary container restarts—even if the isolation checker is not active. This is because the probe's default timeout is 5 seconds, which may not be enough in these scenarios. The current implementation is described in the code and confirmed in recent discussions and issues.

Starting with version 1.26, there's an experimental opt-in feature called "Primary Isolation (alpha)" that changes this behavior. When enabled, the liveness probe will only fail (and trigger a restart) if all of these are true: the pod is primary, the API server is unreachable, at least one replica's WAL receiver is not connected, and at least one instance is unreachable via the instance manager's REST API. This helps avoid unnecessary restarts during brief outages or partitions. You can enable this feature by adding the appropriate annotation to your Cluster resource and optionally tuning network settings like `requestTimeout` and `connectionTimeout` to better fit your environment. Details and configuration examples are in the documentation and discussion threads ([source](https://github.com/cloudnative-pg/cloudnative-pg/discussions/7733), [source](https://github.com/cloudnative-pg/cloudnative-pg/issues/7465)).

This feature is experimental in 1.26 and planned to become a definitive API in 1.27. Until then, the recommended workaround is to use synchronous replication with at least three nodes for consistency during network partitions ([source](https://github.com/cloudnative-pg/cloudnative-pg/discussions/7462)). 

If you need more details on enabling the feature or tuning the probe, let me know!


<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*

---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/a82c12af-0022-4ee3-a6cb-94b4ae81b4c6?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/a82c12af-0022-4ee3-a6cb-94b4ae81b4c6?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/a82c12af-0022-4ee3-a6cb-94b4ae81b4c6?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/a82c12af-0022-4ee3-a6cb-94b4ae81b4c6?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/a82c12af-0022-4ee3-a6cb-94b4ae81b4c6?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/a82c12af-0022-4ee3-a6cb-94b4ae81b4c6?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/a82c12af-0022-4ee3-a6cb-94b4ae81b4c6?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/cloudnative-pg/cloudnative-pg/issues/7901)@jmealo: I think I may have hit this recently. I'm on Azure AKS (premium tier) and routinely Kubernetes API operations fail or take several seconds. Support is giving us the run around and they won't take accountability. It's not a given that the Kubernetes API server is: performant, an indication of cluster health (CNPG or K8s). Many cloud providers have their control plane off somewhere else, so... we can't take it for granted. Typical transient network issues apply.

I'll have to try out the: https://cloudnative-pg.io/documentation/current/instance_manager/#primary-isolation-alpha

I hope this helps someone on Azure find this setting ^^^ :) @mmoscher: Yes, I can confirm the behavior. I had already analyzed the behavior yesterday in another issue which was open 3 weeks ago, https://github.com/cloudnative-pg/cloudnative-pg/issues/7732. Turns out, it's exactly the same!

@jmealo did you already tested the primary isolation? 


Solution patch:
diff --git a/pkg/management/postgres/webserver/probes/liveness.go b/pkg/management/postgres/webserver/probes/liveness.go
index 0449b767..bbb07601 100644
--- a/pkg/management/postgres/webserver/probes/liveness.go
+++ b/pkg/management/postgres/webserver/probes/liveness.go
@@ -23,6 +23,7 @@ import (
 	"context"
 	"fmt"
 	"net/http"
+	"time"
 
 	"github.com/cloudnative-pg/machinery/pkg/log"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -49,12 +50,15 @@ func NewLivenessChecker(
 	}
 }
 
-// tryRefreshLatestCluster refreshes the latest cluster definition, returns a bool indicating if the operation was
-// successful
-func (e *livenessExecutor) tryRefreshLatestCluster(ctx context.Context) bool {
+// tryRefreshLatestClusterWithTimeout refreshes the latest cluster definition, returns a bool indicating if the
+// operation was successful
+func (e *livenessExecutor) tryRefreshLatestClusterWithTimeout(ctx context.Context, timeout time.Duration) bool {
+	timeoutContext, cancel := context.WithTimeout(ctx, timeout)
+	defer cancel()
+
 	var cluster apiv1.Cluster
 	err := e.cli.Get(
-		ctx,
+		timeoutContext,
 		client.ObjectKey{Namespace: e.instance.GetNamespaceName(), Name: e.instance.GetClusterName()},
 		&cluster,
 	)
@@ -87,7 +91,10 @@ func (e *livenessExecutor) IsHealthy(
 		return
 	}
 
-	if clusterRefreshed := e.tryRefreshLatestCluster(ctx); clusterRefreshed {
+	// We set a safe context timeout of 500ms to avoid a failed request from taking
+	// more time than the minimum configurable timeout (1s) of the container's livenessProbe,
+	// which otherwise could have triggered a restart of the instance.
+	if clusterRefreshed := e.tryRefreshLatestClusterWithTimeout(ctx, 500*time.Millisecond); clusterRefreshed {
 		// We correctly reached the API server but, as a failsafe measure, we
 		// exercise the reachability checker and leave a log message if something
 		// is not right.



Test patch:
diff --git a/tests/e2e/fixtures/self-fencing/cluster-liveness-pinger-disabled.yaml.template b/tests/e2e/fixtures/self-fencing/cluster-liveness-pinger-disabled.yaml.template
new file mode 100644
index 00000000..daa68613
--- /dev/null
+++ b/tests/e2e/fixtures/self-fencing/cluster-liveness-pinger-disabled.yaml.template
@@ -0,0 +1,27 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: postgresql-liveness-pinger-disabled
+  annotations:
+    alpha.cnpg.io/livenessPinger: '{"enabled": false}'
+spec:
+  instances: 3
+
+  postgresql:
+    parameters:
+      max_connections: "110"
+      log_checkpoints: "on"
+      log_lock_waits: "on"
+      log_min_duration_statement: '1000'
+      log_statement: 'ddl'
+      log_temp_files: '1024'
+      log_autovacuum_min_duration: '1s'
+      log_replication_commands: 'on'
+
+  # Persistent storage configuration
+  storage:
+    storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
+    size: 1Gi
+  walStorage:
+    storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
+    size: 1Gi
diff --git a/tests/e2e/fixtures/self-fencing/cluster-self-fencing.yaml.template b/tests/e2e/fixtures/self-fencing/cluster-liveness-pinger-enabled.yaml.template
similarity index 93%
rename from tests/e2e/fixtures/self-fencing/cluster-self-fencing.yaml.template
rename to tests/e2e/fixtures/self-fencing/cluster-liveness-pinger-enabled.yaml.template
index 5235464f..37654455 100644
--- a/tests/e2e/fixtures/self-fencing/cluster-self-fencing.yaml.template
+++ b/tests/e2e/fixtures/self-fencing/cluster-liveness-pinger-enabled.yaml.template
@@ -1,7 +1,7 @@
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
-  name: postgresql-self-fencing
+  name: postgresql-liveness-pinger-enabled
   annotations:
     alpha.cnpg.io/livenessPinger: '{"enabled": true}'
 spec:
diff --git a/tests/e2e/self_fencing_test.go b/tests/e2e/self_fencing_test.go
index 17d072d9..0e767ed6 100644
--- a/tests/e2e/self_fencing_test.go
+++ b/tests/e2e/self_fencing_test.go
@@ -44,7 +44,6 @@ import (
 var _ = Describe("Self-fencing with liveness probe", Serial, Label(tests.LabelDisruptive), func() {
 	const (
 		level           = tests.Lowest
-		sampleFile      = fixturesDir + "/self-fencing/cluster-self-fencing.yaml.template"
 		namespacePrefix = "self-fencing"
 	)
 
@@ -57,7 +56,38 @@ var _ = Describe("Self-fencing with liveness probe", Serial, Label(tests.LabelDi
 		}
 	})
 
-	It("will terminate an isolated primary", func() {
+	verifyIsolatedPrimary := func(namespace, isolatedPod, isolatedNode string, livenessPingerEnabled bool) {
+		By("verifying the isolatedPod behaviour", func() {
+			defaultCommand := fmt.Sprintf(
+				"docker exec %v crictl ps -a -q "+
+					"--label io.kubernetes.pod.namespace=%s,io.kubernetes.pod.name=%s "+
+					"--name postgres", isolatedNode, namespace, isolatedPod)
+
+			if livenessPingerEnabled {
+				Eventually(func(g Gomega) {
+					out, _, err := run.Unchecked(fmt.Sprintf("%s -s Exited", defaultCommand))
+					g.Expect(err).ToNot(HaveOccurred())
+					g.Expect(out).ToNot(BeEmpty())
+					if out != "" {
+						GinkgoWriter.Printf("Container %s (%s) has been terminated\n",
+							isolatedPod, strings.TrimSpace(out))
+					}
+				}, 120).Should(Succeed())
+			} else {
+				Consistently(func(g Gomega) {
+					out, _, err := run.Unchecked(fmt.Sprintf("%s -s Running", defaultCommand))
+					g.Expect(err).ToNot(HaveOccurred())
+					g.Expect(out).ToNot(BeEmpty())
+					if out != "" {
+						GinkgoWriter.Printf("Container %s (%s) is still running\n",
+							isolatedPod, strings.TrimSpace(out))
+					}
+				}, 20, 5).Should(Succeed())
+			}
+		})
+	}
+
+	assertLivenessPinger := func(clusterManifest string, livenessPingerEnabled bool) {
 		var namespace, clusterName, isolatedNode string
 		var err error
 		var oldPrimaryPod *corev1.Pod
@@ -70,11 +100,11 @@ var _ = Describe("Self-fencing with liveness probe", Serial, Label(tests.LabelDi
 		})
 
 		By("creating a Cluster", func() {
-			clusterName, err = yaml.GetResourceNameFromYAML(env.Scheme, sampleFile)
+			clusterName, err = yaml.GetResourceNameFromYAML(env.Scheme, clusterManifest)
 			Expect(err).ToNot(HaveOccurred())
 			namespace, err = env.CreateUniqueTestNamespace(env.Ctx, env.Client, namespacePrefix)
 			Expect(err).ToNot(HaveOccurred())
-			AssertCreateCluster(namespace, clusterName, sampleFile, env)
+			AssertCreateCluster(namespace, clusterName, clusterManifest, env)
 		})
 
 		By("setting up the environment", func() {
@@ -107,21 +137,7 @@ var _ = Describe("Self-fencing with liveness probe", Serial, Label(tests.LabelDi
 			}, testTimeouts[timeouts.NewPrimaryAfterFailover]).Should(Succeed())
 		})
 
-		By("verifying that oldPrimary will self isolate", func() {
-			// Assert that the oldPrimary is eventually terminated
-			Eventually(func(g Gomega) {
-				out, _, err := run.Unchecked(fmt.Sprintf(
-					"docker exec %v crictl ps -a "+
-						"--label io.kubernetes.pod.namespace=%s,io.kubernetes.pod.name=%s "+
-						"--name postgres -s Exited -q", isolatedNode, namespace, oldPrimaryPod.Name))
-				g.Expect(err).ToNot(HaveOccurred())
-				g.Expect(out).ToNot(BeEmpty())
-				if out != "" {
-					GinkgoWriter.Printf("Container %s (%s) has been terminated\n",
-						oldPrimaryPod.Name, strings.TrimSpace(out))
-				}
-			}, 120).Should(Succeed())
-		})
+		verifyIsolatedPrimary(namespace, oldPrimaryPod.Name, isolatedNode, livenessPingerEnabled)
 
 		By("reconnecting the isolated Node", func() {
 			_, _, err = run.Unchecked(fmt.Sprintf("docker network connect kind %v", isolatedNode))
@@ -143,5 +159,19 @@ var _ = Describe("Self-fencing with liveness probe", Serial, Label(tests.LabelDi
 				g.Expect(nodes.IsNodeReachable(env.Ctx, env.Client, isolatedNode)).To(BeTrue())
 			}, timeout).Should(Succeed())
 		})
+	}
+
+	When("livenessPinger is enabled", func() {
+		const sampleFile = fixturesDir + "/self-fencing/cluster-liveness-pinger-enabled.yaml.template"
+		It("will terminate an isolated primary", func() {
+			assertLivenessPinger(sampleFile, true)
+		})
+	})
+
+	When("livenessPinger is disabled", func() {
+		const sampleFile = fixturesDir + "/self-fencing/cluster-liveness-pinger-disabled.yaml.template"
+		It("will not restart an isolated primary", func() {
+			assertLivenessPinger(sampleFile, false)
+		})
 	})
 })