opened
ashokspeelyaal
username_0
**Which chart**:
postgresql-ha-1.4.4

**Describe the bug**
I have an ubuntu worker node where both the pods( 0 and 1) were running. 
The node got restarted and see both the pods status remain CrashLoopBackOff. 
I do not see any issue in pod logs. 

From the logs pod-0: 
repmgr 04:56:39.99 INFO  ==> Preparing repmgr configuration...
repmgr 04:56:40.02 INFO  ==> Initializing Repmgr...
postgresql 04:56:40.03 INFO  ==> Initializing PostgreSQL database...
postgresql 04:56:40.12 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/postgresql.conf detected
postgresql 04:56:40.13 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/pg_hba.conf detected
postgresql 04:56:40.16 INFO  ==> Deploying PostgreSQL with persisted data...
postgresql 04:56:40.19 INFO  ==> Stopping PostgreSQL...
postgresql-repmgr 04:56:40.20 INFO  ==> ** PostgreSQL with Replication Manager setup finished! **

postgresql 04:56:40.35 INFO  ==> Starting PostgreSQL in background...
From the logs pod-1: 
postgresql-repmgr 04:54:00.19 INFO  ==> ** Starting PostgreSQL with Replication Manager setup **
repmgr 04:54:00.25 INFO  ==> Validating settings in REPMGR_* env vars...
postgresql 04:54:00.25 INFO  ==> Validating settings in POSTGRESQL_* env vars..
repmgr 04:54:00.26 INFO  ==> Querying all partner nodes for common upstream node...
repmgr 04:54:00.29 INFO  ==> Preparing PostgreSQL configuration...
postgresql 04:54:00.30 INFO  ==> postgresql.conf file not detected. Generating it...
repmgr 04:54:00.36 INFO  ==> Preparing repmgr configuration...
repmgr 04:54:00.38 INFO  ==> Initializing Repmgr...
repmgr 04:54:00.38 INFO  ==> Waiting for primary node...
postgresql 04:55:10.65 INFO  ==> Stopping PostgreSQL...


**To Reproduce**
Steps to reproduce the behavior:
1. create with default configurations
2. restart one of the worker nodes(with primary running)
3. all other pods are coming back to running stage
4. Except, postgresql pods go to status CrashLoopBackOff

**Expected behavior**
The HA should survive the workder node restarts(for multiple reasons: scale up server, patch, unplanned restart by a cloud instance issue etc., )

**Version of Helm and Kubernetes**:
Helm: 3.1.0
Kubernetes: 1.17

- Output of `helm version`:

```
version.BuildInfo{Version:"v3.1.0", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.8"}
```

- Output of `kubectl version`:

```
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

```

**Additional context**
I am not sure what is the ideal replica count or any other configuration to tackle this issue. 
basically, the HA never comes back if primary fails. Hence in my current setup the HA purpose is not met.
Am I missing something to configure?
HA does not survive worker node restart
issue
created
dani8art
username_1
Hi @username_0  thank you for reporting the issue, could it be related to #2033?
comment
created
ashokspeelyaal
username_0
Hello @username_1 ,  yes I assume so, the primary pod (0) was running on that node.  I can try to reproduce in few days and report back if it is a duplicate issue of #2033
comment
created
dani8art
username_1
Ok, we will appreciate it if you can confirm it. Thank you.
comment
created
ashokspeelyaal
username_0
@username_1 , Yes I can confirm. The primary pod (0) was running on the node which was restarted and this caused the issue.
comment
created
ahilsend
username_2
I am having a similar issue after a failure in the HA pods, one node cannot come back up.

Now both agree that pod 0 is the master, but pod 1 is in a CrashLoopBackOff:


```
repmgr 10:45:01.32 INFO  ==> Auto-detected primary node: 'nextcloud-postgresql-ha-postgresql-0.nextcloud-postgresql-ha-postgresql-headless.nextcloud.svc.cluster.local:5432'
repmgr 10:45:01.33 INFO  ==> Preparing PostgreSQL configuration...
postgresql 10:45:01.40 INFO  ==> postgresql.conf file not detected. Generating it...
repmgr 10:45:01.69 INFO  ==> Preparing repmgr configuration...
repmgr 10:45:01.70 INFO  ==> Initializing Repmgr...
repmgr 10:45:01.71 INFO  ==> Waiting for primary node...
repmgr 10:45:01.74 INFO  ==> Cloning data from primary node...
postgresql 10:45:21.82 INFO  ==> Stopping PostgreSQL...
```
comment
created
dani8art
username_1
Hi @username_2, have you try `postgresqlImage.debug=true` ? as you can see on #2033, we are trying to find a better way of shooting down or graceful exit for the pods.
comment
created
ahilsend
username_2
Ohh, I missed that configuration. Thx! With the debug log I could see the issue was actually with DNS, and not with postgres. Restart of CoreDNS resolved my issue. Now I'll have to check what was up with coredns, thx! :blush:
comment
created
dani8art
username_1
Hi, 
Could you confirm that this issue was solved with the PR above? Thanks!
comment
created
ashokspeelyaal
username_0
Hello @username_1 , I will install with this chart version, verify and confirm as soon as possible.
comment
created
dani8art
username_1
Cool! let us know if you have any problems! thanks!
comment
created
ashokspeelyaal
username_0
Hello @username_1,  I can confirm it works now. It tried creating a new release and tried restarting nodes. When the nodes are up again the all the pods are back to status running after a while. Thanks for this.
comment