Repository: zalando/patroni

Issue:
Postmaster starting process hangs because sys.stderr is locked after fork
In version `v1.6.0` Patroni uses forking `multiprocessing.Process` for creating a postmaster parent.
Unfortunately there are other threads which may hold some locks during `fork()` call.

```
$ docker-compose exec patroni2 ps xaf
  PID TTY      STAT   TIME COMMAND
  382 pts/0    Rs+    0:00 ps xaf
    1 ?        Ss     0:00 python3 /patroni.py postgres0.yml
    8 ?        Sl     0:00 python3 /patroni.py postgres0.yml
   24 ?        Ss     0:00  \_ python3 /patroni.py postgres0.yml
   25 ?        S      0:00      \_ postgres -D /var/lib/postgresql/data/pgdata -
   26 ?        Ss     0:00          \_ postgres: batman: startup process   recov
   27 ?        Ss     0:00          \_ postgres: batman: checkpointer process   
   28 ?        Ss     0:00          \_ postgres: batman: writer process   
   29 ?        Ss     0:00          \_ postgres: batman: stats collector process
   31 ?        Ss     0:00          \_ postgres: batman: postgres postgres 127.0
```
The postmaster parent hangs on lock on stderr:
```
Thread 0x7fbf86ffd700
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/patroni/async_executor.py", line 97, in run
    wakeup = func(*args) if args else func()
  File "/patroni/postgresql/__init__.py", line 728, in follow
    self.start(timeout=timeout, block_callbacks=change_role, role=role)
  File "/patroni/postgresql/__init__.py", line 434, in start
    options)
  File "/patroni/postgresql/postmaster.py", line 175, in start
    proc.start()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 269, in _bootstrap
    sys.stderr.flush()
  File "<string>", line 1, in <module>
  File "<string>", line 7, in <module>
```
While a bootstrap thread waits for child to finish:
```
Thread 0x7fbf86ffd700
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/patroni/async_executor.py", line 97, in run
    wakeup = func(*args) if args else func()
  File "/patroni/postgresql/__init__.py", line 728, in follow
    self.start(timeout=timeout, block_callbacks=change_role, role=role)
  File "/patroni/postgresql/__init__.py", line 434, in start
    options)
  File "/patroni/postgresql/postmaster.py", line 177, in start
    proc.join()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 121, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 51, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 29, in poll
    pid, sts = os.waitpid(self.pid, flag)
```

The hanging patroni process repeatedly logs: `INFO: restarting after failure in progress`.

It is a known issue in Python that threading and multiprocessing do not mix well: [https://bugs.python.org/issue6721](url)
While the issue with logging lock might have been fixed in Python 3.7 the recommended workaround is to use `spawn` of `forkserver` start methods, e.g. by adding `multiprocessing.set_start_method('spawn')` at the beggining of `main()`.

This can be quickly reproduced using following `docker-compose.yml`:
```
version: "2.1"
volumes:
    patroni1_volume:
    patroni2_volume:
services:
    zookeeper:
        image: zookeeper:3.4.10
    patroni1:
        build:
          context: https://github.com/zalando/patroni.git
        depends_on:
            - zookeeper
        environment:
            PATRONI_POSTGRESQL_DATA_DIR: /var/lib/postgresql/data/pgdata
            PATRONI_ZOOKEEPER_HOSTS: "[\"zookeeper:2181\"]"
        volumes:
            - patroni1_volume:/var/lib/postgresql/data
        healthcheck:
            test: "curl --fail localhost:8008/health && (curl --fail localhost:8008/sync || curl --fail localhost:8008/master)"
            interval: 10s
            timeout: 5s
            retries: 30
    patroni2:
        build:
          context: https://github.com/zalando/patroni.git
        depends_on:
            - zookeeper
        environment:
            PATRONI_POSTGRESQL_DATA_DIR: /var/lib/postgresql/data/pgdata
            PATRONI_ZOOKEEPER_HOSTS: "[\"zookeeper:2181\"]"
        volumes:
            - patroni2_volume:/var/lib/postgresql/data
        healthcheck:
            test: "curl --fail localhost:8008/health && (curl --fail localhost:8008/sync || curl --fail localhost:8008/master)"
            interval: 10s
            timeout: 5s
            retries: 30
```
and following shell script:
```
#!/bin/bash
docker-compose up -d
docker-compose exec patroni1 patronictl edit-config -s synchronous_mode=true -q --force batman
CHECK() { ! docker-compose ps  | grep patroni[12] | tee /dev/stderr | grep -v '(healthy)' > /dev/null; }
WAIT() { for ((i=0;i<300;i++)); do ! CHECK || break; sleep 1; done; CHECK; }
while WAIT; do docker-compose restart patroni{1,2}; done
```

Postmaster starting process hangs because sys.stderr is locked after fork
In version `v1.6.0` Patroni uses forking `multiprocessing.Process` for creating a postmaster parent.
Unfortunately there are other threads which may hold some locks during `fork()` call.

```
$ docker-compose exec patroni2 ps xaf
  PID TTY      STAT   TIME COMMAND
  382 pts/0    Rs+    0:00 ps xaf
    1 ?        Ss     0:00 python3 /patroni.py postgres0.yml
    8 ?        Sl     0:00 python3 /patroni.py postgres0.yml
   24 ?        Ss     0:00  \_ python3 /patroni.py postgres0.yml
   25 ?        S      0:00      \_ postgres -D /var/lib/postgresql/data/pgdata -
   26 ?        Ss     0:00          \_ postgres: batman: startup process   recov
   27 ?        Ss     0:00          \_ postgres: batman: checkpointer process   
   28 ?        Ss     0:00          \_ postgres: batman: writer process   
   29 ?        Ss     0:00          \_ postgres: batman: stats collector process
   31 ?        Ss     0:00          \_ postgres: batman: postgres postgres 127.0
```
The postmaster parent hangs on lock on stderr:
```
Thread 0x7fbf86ffd700
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/patroni/async_executor.py", line 97, in run
    wakeup = func(*args) if args else func()
  File "/patroni/postgresql/__init__.py", line 728, in follow
    self.start(timeout=timeout, block_callbacks=change_role, role=role)
  File "/patroni/postgresql/__init__.py", line 434, in start
    options)
  File "/patroni/postgresql/postmaster.py", line 175, in start
    proc.start()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 269, in _bootstrap
    sys.stderr.flush()
  File "<string>", line 1, in <module>
  File "<string>", line 7, in <module>
```
While a bootstrap thread waits for child to finish:
```
Thread 0x7fbf86ffd700
  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/patroni/async_executor.py", line 97, in run
    wakeup = func(*args) if args else func()
  File "/patroni/postgresql/__init__.py", line 728, in follow
    self.start(timeout=timeout, block_callbacks=change_role, role=role)
  File "/patroni/postgresql/__init__.py", line 434, in start
    options)
  File "/patroni/postgresql/postmaster.py", line 177, in start
    proc.join()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 121, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 51, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 29, in poll
    pid, sts = os.waitpid(self.pid, flag)
```

The hanging patroni process repeatedly logs: `INFO: restarting after failure in progress`.

It is a known issue in Python that threading and multiprocessing do not mix well: [https://bugs.python.org/issue6721](url)
While the issue with logging lock might have been fixed in Python 3.7 the recommended workaround is to use `spawn` of `forkserver` start methods, e.g. by adding `multiprocessing.set_start_method('spawn')` at the beggining of `main()`.

This can be quickly reproduced using following `docker-compose.yml`:
```
version: "2.1"
volumes:
    patroni1_volume:
    patroni2_volume:
services:
    zookeeper:
        image: zookeeper:3.4.10
    patroni1:
        build:
          context: https://github.com/zalando/patroni.git
        depends_on:
            - zookeeper
        environment:
            PATRONI_POSTGRESQL_DATA_DIR: /var/lib/postgresql/data/pgdata
            PATRONI_ZOOKEEPER_HOSTS: "[\"zookeeper:2181\"]"
        volumes:
            - patroni1_volume:/var/lib/postgresql/data
        healthcheck:
            test: "curl --fail localhost:8008/health && (curl --fail localhost:8008/sync || curl --fail localhost:8008/master)"
            interval: 10s
            timeout: 5s
            retries: 30
    patroni2:
        build:
          context: https://github.com/zalando/patroni.git
        depends_on:
            - zookeeper
        environment:
            PATRONI_POSTGRESQL_DATA_DIR: /var/lib/postgresql/data/pgdata
            PATRONI_ZOOKEEPER_HOSTS: "[\"zookeeper:2181\"]"
        volumes:
            - patroni2_volume:/var/lib/postgresql/data
        healthcheck:
            test: "curl --fail localhost:8008/health && (curl --fail localhost:8008/sync || curl --fail localhost:8008/master)"
            interval: 10s
            timeout: 5s
            retries: 30
```
and following shell script:
```
#!/bin/bash
docker-compose up -d
docker-compose exec patroni1 patronictl edit-config -s synchronous_mode=true -q --force batman
CHECK() { ! docker-compose ps  | grep patroni[12] | tee /dev/stderr | grep -v '(healthy)' > /dev/null; }
WAIT() { for ((i=0;i<300;i++)); do ! CHECK || break; sleep 1; done; CHECK; }
while WAIT; do docker-compose restart patroni{1,2}; done
```



Discussion:
Interesting, somehow we have never seen it in the wild.
Maybe because we don't run Patroni with PID=1?

@maciejkowalczyk9ld could you please prepare a fix?
> Interesting, somehow we have never seen it in the wild.
> Maybe because we don't run Patroni with PID=1?

It occurs even if you add some proper init process with PID=1 above Patroni:
```diff
--- docker-compose.yml
+++ docker-compose.yml
@@ -1,4 +1,4 @@
-version: "2.1"
+version: "2.2"
 volumes:
     patroni1_volume:
     patroni2_volume:
@@ -6,6 +6,7 @@ services:
     zookeeper:
         image: zookeeper:3.4.10
     patroni1:
+        init: true
         build:
           context: https://github.com/zalando/patroni.git
         depends_on:
@@ -21,6 +22,7 @@ services:
             timeout: 5s
             retries: 30
     patroni2:
+        init: true
         build:
           context: https://github.com/zalando/patroni.git
         depends_on:
```

> @maciejkowalczyk9ld could you please prepare a fix?

Do you still support Python 2.7?
I'm not sure this bug also occurs there and it does not have capability of setting `spawn` start method.
So my fix would set the start method conditionally.
> Do you still support Python 2.7?

Yes, we do. Despite all attempts to make 2.7 die some enterprises continue using it :(

> I'm not sure this bug also occurs there and it does not have capability of setting spawn start method.

Yes, I know that it lacks such capability, and probably it was a reason why I didn't set it myself.
Before going the `multiprocessing` way we tried to use `subprocess.Popen` to execute itself and send the postmaster pid via stdout-stdin. It was sort of working, but closing the stdout of postgres process :(

> So my fix would set the start method conditionally.

I am fine with it.
Interesting, somehow we have never seen it in the wild.
Maybe because we don't run Patroni with PID=1?

@maciejkowalczyk9ld could you please prepare a fix?
> Interesting, somehow we have never seen it in the wild.
> Maybe because we don't run Patroni with PID=1?

It occurs even if you add some proper init process with PID=1 above Patroni:
```diff
--- docker-compose.yml
+++ docker-compose.yml
@@ -1,4 +1,4 @@
-version: "2.1"
+version: "2.2"
 volumes:
     patroni1_volume:
     patroni2_volume:
@@ -6,6 +6,7 @@ services:
     zookeeper:
         image: zookeeper:3.4.10
     patroni1:
+        init: true
         build:
           context: https://github.com/zalando/patroni.git
         depends_on:
@@ -21,6 +22,7 @@ services:
             timeout: 5s
             retries: 30
     patroni2:
+        init: true
         build:
           context: https://github.com/zalando/patroni.git
         depends_on:
```

> @maciejkowalczyk9ld could you please prepare a fix?

Do you still support Python 2.7?
I'm not sure this bug also occurs there and it does not have capability of setting `spawn` start method.
So my fix would set the start method conditionally.
> Do you still support Python 2.7?

Yes, we do. Despite all attempts to make 2.7 die some enterprises continue using it :(

> I'm not sure this bug also occurs there and it does not have capability of setting spawn start method.

Yes, I know that it lacks such capability, and probably it was a reason why I didn't set it myself.
Before going the `multiprocessing` way we tried to use `subprocess.Popen` to execute itself and send the postmaster pid via stdout-stdin. It was sort of working, but closing the stdout of postgres process :(

> So my fix would set the start method conditionally.

I am fine with it.

Resolving patch:
diff --git a/patroni/__init__.py b/patroni/__init__.py
--- a/patroni/__init__.py
+++ b/patroni/__init__.py
@@ -176,6 +176,13 @@ def fatal(string, *args):
     sys.exit(1)
 
 
+def use_spawn_start_method():
+    if sys.version_info >= (3, 4):
+        # The default, forking, method is not a good idea in a multithreaded process: https://bugs.python.org/issue6721
+        import multiprocessing
+        multiprocessing.set_start_method('spawn')
+
+
 def check_psycopg2():
     min_psycopg2 = (2, 5, 4)
     min_psycopg2_str = '.'.join(map(str, min_psycopg2))
@@ -198,6 +205,7 @@ def parse_version(version):
 
 
 def main():
+    use_spawn_start_method()
     check_psycopg2()
     if os.getpid() != 1:
         return patroni_main()


Test patch:
diff --git a/tests/test_patroni.py b/tests/test_patroni.py
--- a/tests/test_patroni.py
+++ b/tests/test_patroni.py
@@ -83,6 +83,7 @@ def test_patroni_patroni_main(self, mock_machines):
 
     @patch('os.getpid')
     @patch('multiprocessing.Process')
+    @patch('patroni.use_spawn_start_method', Mock())
     @patch('patroni.patroni_main', Mock())
     def test_patroni_main(self, mock_process, mock_getpid):
         mock_getpid.return_value = 2