Repository: cloudnative-pg/cloudnative-pg

Issue:
[Test]: synchronous replication lag-control E2E can exhaust disk space
### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

_No response_

### Version

trunk (main)

### What version of Kubernetes are you using?

1.34

### What is your Kubernetes environment?

Self-managed: kind (evaluation)

### How did you install the operator?

YAML manifest

### What happened?

In this test we generate ~1G of data in the primary node, to generate lag.
We need to provide more disk space because the test is sometimes failing with:
```
      cmd: [psql -U postgres -c create table numbers (i integer); insert into numbers (select generate_series(1,1000000)); insert into numbers (select * from numbers); insert into numbers (select * from numbers); insert into numbers (select * from numbers); ]
      error: command terminated with exit code 2
      stdErr: PANIC:  could not write to file "pg_wal/xlogtemp.1071": No space left on device
```

### Cluster resource

```shell

```

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Hints from issue comments:
@dosubot: <!-- Answer -->
There isn’t an existing issue specifically tracking disk space exhaustion in the synchronous replication lag-control E2E test, so your report is valid as a new bug.

The test generates ~1GB of data on the primary to induce replication lag, which can easily exhaust disk space—especially in environments like kind with limited ephemeral storage. The test itself does not perform any disk cleanup or WAL management, so if storage is undersized or WAL retention is unlimited, you’ll hit errors like `PANIC: could not write to file "pg_wal/xlogtemp.1071": No space left on device` [source](https://github.com/cloudnative-pg/cloudnative-pg/blob/74dc25387b94c87cfca3cab703f2f58a01189839/tests/e2e/syncreplicas_test.go).

CloudNativePG recommends explicitly setting the `max_slot_wal_keep_size` parameter in your cluster manifest to cap WAL retention and avoid unlimited growth [source](https://cloudnative-pg.io/documentation/current/replication/#capping-the-wal-size-retained-for-replication-slots). For example:

```yaml
postgresql:
  parameters:
    max_slot_wal_keep_size: "10GB"
```

If you run into disk exhaustion, the recommended recovery is to increase your PVC size for the WAL volume [source](https://github.com/cloudnative-pg/cloudnative-pg/issues/7150). CloudNativePG 1.24+ also introduced a safety measure: when WAL disk space runs out, the cluster safely stops instead of cascading failovers, making recovery easier [source](https://cloudnative-pg.io/releases/cloudnative-pg-1-24.0-rc1-released/).

For test environments, consider increasing the storage allocation and monitoring WAL usage. If you want to avoid this issue in CI or local runs, you may need to tune the test setup or add cleanup steps after heavy data generation.

Let me know if you need details on configuring storage or WAL retention for your specific test scenario.


<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*

---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/2aa6d953-a943-4323-bfc0-ba30daa44747?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/2aa6d953-a943-4323-bfc0-ba30daa44747?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/2aa6d953-a943-4323-bfc0-ba30daa44747?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/2aa6d953-a943-4323-bfc0-ba30daa44747?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2aa6d953-a943-4323-bfc0-ba30daa44747?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/2aa6d953-a943-4323-bfc0-ba30daa44747?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2aa6d953-a943-4323-bfc0-ba30daa44747?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/89b818f3-8b5e-449b-bfb3-64a9360d9665/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/cloudnative-pg/cloudnative-pg/issues/8969)

Solution patch:
diff --git a/docs/src/failover.md b/docs/src/failover.md
index 1608f2ae..2a4bb228 100644
--- a/docs/src/failover.md
+++ b/docs/src/failover.md
@@ -151,7 +151,7 @@ spec:
       failoverQuorum: true
 
   storage:
-    size: 1G
+    size: 1Gi
 ```
 
 For backward compatibility, the legacy annotation
diff --git a/docs/src/replication.md b/docs/src/replication.md
index c4eb01e4..ce8c915b 100644
--- a/docs/src/replication.md
+++ b/docs/src/replication.md
@@ -187,7 +187,7 @@ spec:
   maxSyncReplicas: 1
 
   storage:
-    size: 1G
+    size: 1Gi
 ```
 
 You can update it to the new format as follows:
@@ -201,7 +201,7 @@ spec:
   instances: 3
 
   storage:
-    size: 1G
+    size: 1Gi
 
   postgresql:
     synchronous:
diff --git a/docs/src/samples/cluster-example-syncreplicas-explicit.yaml b/docs/src/samples/cluster-example-syncreplicas-explicit.yaml
index 14ec8f21..90592cc6 100644
--- a/docs/src/samples/cluster-example-syncreplicas-explicit.yaml
+++ b/docs/src/samples/cluster-example-syncreplicas-explicit.yaml
@@ -11,4 +11,4 @@ spec:
       number: 2
 
   storage:
-    size: 1G
+    size: 1Gi
diff --git a/docs/src/samples/cluster-example-syncreplicas-legacy.yaml b/docs/src/samples/cluster-example-syncreplicas-legacy.yaml
index 450fc8ab..bd54560b 100644
--- a/docs/src/samples/cluster-example-syncreplicas-legacy.yaml
+++ b/docs/src/samples/cluster-example-syncreplicas-legacy.yaml
@@ -9,4 +9,4 @@ spec:
   maxSyncReplicas: 3
 
   storage:
-    size: 1G
+    size: 1Gi
diff --git a/docs/src/samples/cluster-example-syncreplicas-quorum.yaml b/docs/src/samples/cluster-example-syncreplicas-quorum.yaml
index 72dc03b7..ee8887e2 100644
--- a/docs/src/samples/cluster-example-syncreplicas-quorum.yaml
+++ b/docs/src/samples/cluster-example-syncreplicas-quorum.yaml
@@ -12,4 +12,4 @@ spec:
       failoverQuorum: true
 
   storage:
-    size: 1G
+    size: 1Gi



Test patch:
diff --git a/tests/e2e/fixtures/sync_replicas/cluster-pgstatstatements.yaml.template b/tests/e2e/fixtures/sync_replicas/cluster-pgstatstatements.yaml.template
index 20cbcd7b..1d64f9da 100644
--- a/tests/e2e/fixtures/sync_replicas/cluster-pgstatstatements.yaml.template
+++ b/tests/e2e/fixtures/sync_replicas/cluster-pgstatstatements.yaml.template
@@ -25,4 +25,4 @@ spec:
 
   storage:
     storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
-    size: 1G
+    size: 1Gi
diff --git a/tests/e2e/fixtures/sync_replicas/cluster-sync-replica-legacy.yaml.template b/tests/e2e/fixtures/sync_replicas/cluster-sync-replica-legacy.yaml.template
index 973e8ef6..6cbe3d57 100644
--- a/tests/e2e/fixtures/sync_replicas/cluster-sync-replica-legacy.yaml.template
+++ b/tests/e2e/fixtures/sync_replicas/cluster-sync-replica-legacy.yaml.template
@@ -24,4 +24,4 @@ spec:
 
   storage:
     storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
-    size: 1G
+    size: 1Gi
diff --git a/tests/e2e/fixtures/sync_replicas/cluster-sync-replica.yaml.template b/tests/e2e/fixtures/sync_replicas/cluster-sync-replica.yaml.template
index bbdccd10..f6caead5 100644
--- a/tests/e2e/fixtures/sync_replicas/cluster-sync-replica.yaml.template
+++ b/tests/e2e/fixtures/sync_replicas/cluster-sync-replica.yaml.template
@@ -24,4 +24,4 @@ spec:
 
   storage:
     storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
-    size: 1G
+    size: 1Gi
diff --git a/tests/e2e/fixtures/sync_replicas/preferred.yaml.template b/tests/e2e/fixtures/sync_replicas/preferred.yaml.template
index 737ae75d..e273a2df 100644
--- a/tests/e2e/fixtures/sync_replicas/preferred.yaml.template
+++ b/tests/e2e/fixtures/sync_replicas/preferred.yaml.template
@@ -25,4 +25,4 @@ spec:
 
   storage:
     storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
-    size: 1G
+    size: 1Gi
diff --git a/tests/e2e/fixtures/sync_replicas/readiness-probe-lag-control.yaml.template b/tests/e2e/fixtures/sync_replicas/readiness-probe-lag-control.yaml.template
index 33075f2e..c5984192 100644
--- a/tests/e2e/fixtures/sync_replicas/readiness-probe-lag-control.yaml.template
+++ b/tests/e2e/fixtures/sync_replicas/readiness-probe-lag-control.yaml.template
@@ -28,4 +28,4 @@ spec:
 
   storage:
     storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
-    size: 1G
+    size: 3Gi
diff --git a/tests/e2e/fixtures/sync_replicas/startup-probe-lag-control.yaml.template b/tests/e2e/fixtures/sync_replicas/startup-probe-lag-control.yaml.template
index d25f2567..c74ac889 100644
--- a/tests/e2e/fixtures/sync_replicas/startup-probe-lag-control.yaml.template
+++ b/tests/e2e/fixtures/sync_replicas/startup-probe-lag-control.yaml.template
@@ -31,4 +31,4 @@ spec:
 
   storage:
     storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
-    size: 1G
+    size: 3Gi
diff --git a/tests/e2e/pg_wal_volume_test.go b/tests/e2e/pg_wal_volume_test.go
index 164ee134..00fef5ca 100644
--- a/tests/e2e/pg_wal_volume_test.go
+++ b/tests/e2e/pg_wal_volume_test.go
@@ -103,7 +103,7 @@ var _ = Describe("Separate pg_wal volume", Label(tests.LabelStorage), func() {
 			Expect(err).NotTo(HaveOccurred())
 			WalStorageClass := os.Getenv("E2E_DEFAULT_STORAGE_CLASS")
 			cluster.Spec.WalStorage = &apiv1.StorageConfiguration{
-				Size:         "1G",
+				Size:         "1Gi",
 				StorageClass: &WalStorageClass,
 			}
 			return env.Client.Update(env.Ctx, cluster)
diff --git a/tests/e2e/syncreplicas_test.go b/tests/e2e/syncreplicas_test.go
index 912240c5..2c33504a 100644
--- a/tests/e2e/syncreplicas_test.go
+++ b/tests/e2e/syncreplicas_test.go
@@ -139,8 +139,12 @@ var _ = Describe("Synchronous Replicas", Label(tests.LabelReplication), func() {
 			primary, err := clusterutils.GetPrimary(env.Ctx, env.Client, namespace, clusterName)
 			Expect(err).ToNot(HaveOccurred())
 
-			// This will generate 1Gi of data in the primary node and, since the replica we fenced
+			// This will generate ~1GB of data in the primary node and, since the replica we fenced
 			// is not aligned, will generate lag.
+			// The test creates 1M rows, then doubles 3 times (2M, 4M, 8M rows = ~400-500MB data).
+			// However, PostgreSQL generates significant WAL overhead, temporary files, and index data
+			// during these operations. The fixtures using this function should have at least 3Gi
+			// of storage to ensure sufficient space for data, WAL files, and temporary operations.
 			_, _, err = exec.Command(
 				env.Ctx, env.Interface, env.RestClientConfig,
 				*primary, specs.PostgresContainerName, &commandTimeout,