Repository: benoitc/gunicorn

Issue:
"Booting worker" is looping infinitely despite no exit signals
I'm trying to get gunicorn set up on Docker. It works well locally, and the production image is exactly the same as the local image, but I'm getting this strange behaviour on the production Docker engine:

```
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [INFO] Starting gunicorn 19.7.1
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [DEBUG] Arbiter booted
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [INFO] Using worker: sync
ml-server_1     | [2017-12-11 13:18:50 +0000] [8] [INFO] Booting worker with pid: 8
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [DEBUG] 1 workers
ml-server_1     | Using TensorFlow backend.
ml-server_1     | [2017-12-11 13:18:54 +0000] [11] [INFO] Booting worker with pid: 11
ml-server_1     | Using TensorFlow backend.
ml-server_1     | [2017-12-11 13:18:58 +0000] [14] [INFO] Booting worker with pid: 14
ml-server_1     | Using TensorFlow backend.
ml-server_1     | [2017-12-11 13:19:02 +0000] [17] [INFO] Booting worker with pid: 17
ml-server_1     | Using TensorFlow backend.
```

It looks like gunicorn is booting workers every 4-5 seconds, despite no apparent error messages or exit signals. This behaviour continues indefinitely until terminated.

Is it possible that a worker can exit without logging anything to stderr/stdout, or for the arbiter to spawn workers infinitely?

Since they are the same docker image, they are running exactly the same code, on exactly the same architecture, so I'm really confused what this could be (bug?). Any help greatly appreciated!


Discussion:
`ssh`-ing into the Docker container led me to find this error:

`Illegal instruction (core dumped)`

Perhaps gunicorn should surface errors like this instead of swallowing them, or handle them differently? Not sure, just thought I'd raise this since it might help someone else!
Thanks for reporting the issue!

If you can figure out where this happens, that would be very helpful.

Perhaps we can add logging when workers exit. Usually, the worker itself logs, but if it's killed very abruptly it won't.
No worries!

There seems to be a problem with [Spacy](https://spacy.io/usage/) which I've just added to this thread: https://github.com/explosion/spaCy/issues/1589

Anyway, it's causing a `SIGILL` as `strace` confirms:

```
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x7ff48bbe6cea} ---
+++ killed by SIGILL (core dumped) +++
Illegal instruction (core dumped)
```

I suppose it would be nice if gunicorn could identify this and log an error rather than phantomly restarting the worker, but tbh I know very little about how exit codes work!
Some exit codes definitely have special meanings and we could probably log those.
http://tldp.org/LDP/abs/html/exitcodes.html
Sounds good! Additionally if the exit code isn't a reserved exit code (such as this case), it would be cool if that could be logged (without an explanation) so it's apparent that the worker is indeed terminating 🙂 
I have kind of similar issue, gunicorn is booting new worker always when I make http request. I do not get any response back, it just reboots new worker always. Strace log from two http requests:

```
select(4, [3], [], [], {1, 0})          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0})          = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=510, si_uid=0, si_status=SIGSEGV, si_utime=160, si_stime=32} ---
getpid()                                = 495
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV && WCOREDUMP(s)}], WNOHANG, NULL) = 510
lseek(8, 0, SEEK_CUR)                   = 0
close(8)                                = 0
wait4(-1, 0x7ffd455ad844, WNOHANG, NULL) = 0
write(4, ".", 1)                        = 1
select(4, [3], [], [], {0, 840340})     = 1 (in [3], left {0, 840338})
read(3, ".", 1)                         = 1
read(3, 0x7f2682025fa0, 1)              = -1 EAGAIN (Resource temporarily unavailable)
fstat(6, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG, st_size=0, ...}) = 0
umask(0)                                = 022
getpid()                                = 495
open("/tmp/wgunicorn-q4aa72u7", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = 8
fcntl(8, F_SETFD, FD_CLOEXEC)           = 0
chown("/tmp/wgunicorn-q4aa72u7", 0, 0)  = 0
umask(022)                              = 0
unlink("/tmp/wgunicorn-q4aa72u7")       = 0
fstat(8, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
ioctl(8, TIOCGWINSZ, 0x7ffd455b8e50)    = -1 ENOTTY (Not a tty)
lseek(8, 0, SEEK_CUR)                   = 0
lseek(8, 0, SEEK_CUR)                   = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
fork()                                  = 558
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
select(0, NULL, NULL, NULL, {0, 37381}[2017-12-28 17:50:23 +0000] [558] [INFO] Booting worker with pid: 558
) = 0 (Timeout)
select(4, [3], [], [], {1, 0}loading test-eu-ovh settings
)          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0}
)          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0})          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0})          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0})          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0})          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0})          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG, st_size=0, ...}) = 0
select(4, [3], [], [], {1, 0})          = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=499, si_uid=0, si_status=SIGSEGV, si_utime=160, si_stime=31} ---
getpid()                                = 495
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV && WCOREDUMP(s)}], WNOHANG, NULL) = 499
lseek(7, 0, SEEK_CUR)                   = 0
close(7)                                = 0
wait4(-1, 0x7ffd455ad844, WNOHANG, NULL) = 0
write(4, ".", 1)                        = 1
select(4, [3], [], [], {0, 450691})     = 1 (in [3], left {0, 450689})
read(3, ".", 1)                         = 1
read(3, 0x7f2682067de8, 1)              = -1 EAGAIN (Resource temporarily unavailable)
fstat(6, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG, st_size=0, ...}) = 0
umask(0)                                = 022
getpid()                                = 495
open("/tmp/wgunicorn-5x9a40ca", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = 7
fcntl(7, F_SETFD, FD_CLOEXEC)           = 0
chown("/tmp/wgunicorn-5x9a40ca", 0, 0)  = 0
umask(022)                              = 0
unlink("/tmp/wgunicorn-5x9a40ca")       = 0
fstat(7, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
ioctl(7, TIOCGWINSZ, 0x7ffd455b8e50)    = -1 ENOTTY (Not a tty)
lseek(7, 0, SEEK_CUR)                   = 0
lseek(7, 0, SEEK_CUR)                   = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
fork()                                  = 579
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
select(0, NULL, NULL, NULL, {0, 8144}[2017-12-28 17:50:30 +0000] [579] [INFO] Booting worker with pid: 579
)  = 0 (Timeout)
select(4, [3], [], [], {1, 0})          = 0 (Timeout)
fstat(6, {st_mode=S_IFREG, st_size=0, ...}) = 0
fstat(7, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
fstat(9, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
fstat(8, {st_mode=S_IFREG|01, st_size=0, ...}) = 0
```
I am facing the same issue, `gunicorn` is booting repeated within seconds for `sync` worker type. Setting worker timeout to `900` is not helping.

In my load before action, I am downloading data from AWS S3. It takes approx 1min 10 sec to download the various files. 
@sara-02 what is your command line to launch gunicorn ? 
@benoitc `gunicorn --pythonpath /src -b 0.0.0.0:$SERVICE_PORT --workers=1 -k sync -t $SERVICE_TIMEOUT flask_endpoint:app`
Present [here](https://github.com/fabric8-analytics/f8a-hpf-insights/blob/master/entrypoint.sh)
@sara-02 Thanks.  

Are old workers really exiting or they are kept online and new workers are spawned? What the debug log shows also?
The logs are mixed with botocore logs, but it is something like this
```
[INFO] Booting worker with pid:  a
[INFO] Booting worker with pid:  b
[INFO] Booting worker with pid:  c
```
but are the worker killed ? what return the command `ps ax|grep gunicorn` ?
@benoitc 
![screenshot from 2018-07-05 19-14-00](https://user-images.githubusercontent.com/7047079/42326804-d9359726-8087-11e8-86a9-f88bd8f0c9ac.png)

One question though, why do we see 2 `gunicorn` processes, when worker limit is set to 1? Is one master, and one worker?
There is 1 arbiter process (master) and N workers processes ye :)

So  you run the command each time a worker booted right ?  If so it seems that the older worker is killed a new one is spawned. I will investigate.
@sara-02 one last thing, this also happening in docker?
@benoitc on `docker-compose` it is working as expected, but when I am putting the same code on `Openshift`, I see this error. Increasing the memory requirement did fix, but when I run the application via `docker-compose` it is using less than `limited` memory. 
Just an update, the issue for me was actually a memory error and was fixed when the memory issue was fixed.
@benoitc 
I am facing the same Issue while trying to spawn 5 gunicorn worker in docker.
@sara-02 
How did you identify the cause to be a memory error?

![image](https://user-images.githubusercontent.com/7349925/46630580-3e76b280-cb62-11e8-82bc-ea70deaa4e41.png)


@gulshan-gaurav 2 things helped me:
I increased the memory assigned to my Pod and stopped crashing. Second, we checked our Openshift Zabbix logs.
@sara-02 
Even on my staging Pod the files + models that I am loading in memory amounts to 50Mb so 2GB of memory should be sufficient for 5 workers.
@gulshan-gaurav which issue are you facing? Having 5 process there looks good....
I had the same issue. I didn't locate the exact problem, but it was solved once I upgraded from python 3.5 to 3.6.  
I am facing the same issue in a Docker container. Gunicorn keeps botting a new worker every time I call an endpoint that causes the failure but not exception or error is outputted into the Gunicorn's log files. Things that I choose to print are logged and then suddenly the log file just says "Booting worker with pid..."

One step that helped was to add env variable PYTHONUNBUFFERED. Before that, even the print statements would disappear and would not be saved in Gunicorn's logs.

2 other endpoints of the app work correctly.

I run Gunicorn with: gunicorn run:app -b localhost:5000 --enable-stdio-inheritance --error-logfile /var/log/gunicorn/error.log --access-logfile /var/log/gunicorn/access.log --capture-output --log-level debug

Already running Python 3.6 and checked with top that the memory doesn't seem to be an issue.

EDIT: It looks like it was a Python issue and not a Gunicorn's fault. Some version discrepancies were causing Python to just die without any trace while performing a certain operation.
I am facing similar issue where worker node keep coming up  with 
`Booting worker with pid: 17636`. i don't know if it is killing the previous worker node or previous worker node still exists. But the number of workers mentioned in gunicorn command line arguments is only 3 - `-workers=3 `.   Also I am using python version 3.7


I had a mismatch of scikit-learn dependency, but even after resolving that, i am getting still same infinite workers coming up. what kind of python version discrepancies should i look for and how to identify them?
I'm facing the same issue inside OpenShift.

![image](https://user-images.githubusercontent.com/13910652/52298786-f1b5b380-296a-11e9-9d40-46f1f637794a.png)

As you can see in the image, I'm using 6 workers (I was trying with 3).
I increased the pod's memory and it does not worked.

BuildConfig:

![image](https://user-images.githubusercontent.com/13910652/52298938-5ffa7600-296b-11e9-90f4-f424389b1e49.png)

Any idea?

Thanks
are you running this in aws behind elb? I solved that issue by putting nginx ingress between elb and gunicorn
Have the same issue.
```
flask_1  | [2019-02-23 09:08:17 +0000] [1] [INFO] Starting gunicorn 19.9.0
flask_1  | [2019-02-23 09:08:17 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
flask_1  | [2019-02-23 09:08:17 +0000] [1] [INFO] Using worker: sync
flask_1  | [2019-02-23 09:08:17 +0000] [8] [INFO] Booting worker with pid: 8
flask_1  | [2019-02-23 09:08:19 +0000] [12] [INFO] Booting worker with pid: 12
flask_1  | [2019-02-23 09:08:19 +0000] [16] [INFO] Booting worker with pid: 16
flask_1  | [2019-02-23 09:08:20 +0000] [20] [INFO] Booting worker with pid: 20
flask_1  | [2019-02-23 09:08:21 +0000] [24] [INFO] Booting worker with pid: 24
flask_1  | [2019-02-23 09:08:22 +0000] [28] [INFO] Booting worker with pid: 28
flask_1  | [2019-02-23 09:08:23 +0000] [32] [INFO] Booting worker with pid: 32
flask_1  | [2019-02-23 09:08:25 +0000] [36] [INFO] Booting worker with pid: 36
flask_1  | [2019-02-23 09:08:26 +0000] [40] [INFO] Booting worker with pid: 40
flask_1  | [2019-02-23 09:08:27 +0000] [44] [INFO] Booting worker with pid: 44
flask_1  | [2019-02-23 09:08:29 +0000] [48] [INFO] Booting worker with pid: 48
flask_1  | [2019-02-23 09:08:30 +0000] [52] [INFO] Booting worker with pid: 52
flask_1  | [2019-02-23 09:08:31 +0000] [56] [INFO] Booting worker with pid: 56
flask_1  | [2019-02-23 09:08:33 +0000] [60] [INFO] Booting worker with pid: 60
flask_1  | [2019-02-23 09:08:34 +0000] [64] [INFO] Booting worker with pid: 64
flask_1  | [2019-02-23 09:08:35 +0000] [68] [INFO] Booting worker with pid: 68
flask_1  | [2019-02-23 09:08:36 +0000] [72] [INFO] Booting worker with pid: 72
flask_1  | [2019-02-23 09:08:37 +0000] [76] [INFO] Booting worker with pid: 76
flask_1  | [2019-02-23 09:08:38 +0000] [80] [INFO] Booting worker with pid: 80
flask_1  | [2019-02-23 09:08:40 +0000] [84] [INFO] Booting worker with pid: 84
flask_1  | [2019-02-23 09:08:41 +0000] [88] [INFO] Booting worker with pid: 88
flask_1  | [2019-02-23 09:08:42 +0000] [92] [INFO] Booting worker with pid: 92
flask_1  | [2019-02-23 09:08:44 +0000] [96] [INFO] Booting worker with pid: 96
flask_1  | [2019-02-23 09:08:45 +0000] [100] [INFO] Booting worker with pid: 100
flask_1  | [2019-02-23 09:08:45 +0000] [104] [INFO] Booting worker with pid: 104
flask_1  | [2019-02-23 09:08:46 +0000] [108] [INFO] Booting worker with pid: 108
flask_1  | [2019-02-23 09:08:47 +0000] [112] [INFO] Booting worker with pid: 112
flask_1  | [2019-02-23 09:08:48 +0000] [116] [INFO] Booting worker with pid: 116
flask_1  | [2019-02-23 09:08:49 +0000] [120] [INFO] Booting worker with pid: 120
flask_1  | [2019-02-23 09:08:50 +0000] [124] [INFO] Booting worker with pid: 124
flask_1  | [2019-02-23 09:08:52 +0000] [128] [INFO] Booting worker with pid: 128
```

Here is `docker-compose.yml`:
```
version: '3'
services:
  flask:
    build: .
    command: gunicorn -b 0.0.0.0:5000 hello:app --reload
    environment:
      - FLASK_APP=hello.py
      - FLASK_DEBUG=1
      - PYTHONUNBUFFERED=True
    ports:
      - "5000:5000"
    volumes:
      - ./:/root
```
what docker image does it use?
@benoitc 
```
[ec2-user@ip-172-31-85-181 web-services-course]$ docker --version
Docker version 18.06.1-ce, build e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
[ec2-user@ip-172-31-85-181 web-services-course]$ docker-compose --version
docker-compose version 1.23.2, build 1110ad01
```
Here is the links for:
- `Dockerfile` https://github.com/glebmikha/web-services-course/blob/master/Dockerfile
- `docker-compose.yml` https://github.com/glebmikha/web-services-course/blob/master/docker-compose.yml
figured out it might be caused by memory lack. the app needs more memory than is available.
but it's just assumption
Just as information: I had observed exactly this  behavior when I have a gunicorn conf to 3 workers, but I deployed the code in a virtual machine with a single core CPU. Then, I change the environment to use 2 cores, and obviously the problem disappeared
Why is 'Worker exiting' only at level INFO -- why would a worker exit except as the result of an error? It took me a long time to work out that my worker threads were being killed by the system OOM killer, with nothing in the logs except, as reported by some others above, 'Booting worker with pid' from time to time.
@HughWarrington because a worker exiting is not necessarily an error. Workers can be terminated by signals or by options like `--max-requests`.
@HughWarrington we could probably add logging in the arbiter for when a worker exits with an abnormal exit code.
You can open a ticket for that, or contribute a PR that adds this code to the `reap_workers` method.
I had the same issue, and the solution was to increase the memory size of the pod.
Had the same issue running Gunicorn on Docker with a large spaCy model, it kept restarting workers without any error messages. The solution is to increase the memory for the Docker container. 
Just encountered this issue today on latest (19.9.0) gunicorn with gevent (1.4.0) workers running on Kubernetes. The app is a [Falcon](https://falconframework.org/) app and the Docker image is the [official Python image](https://hub.docker.com/_/python) with tag `3.7.3`.

```
[2019-07-05 00:07:42 +0000] [8] [INFO] Starting gunicorn 19.9.0
[2019-07-05 00:07:42 +0000] [8] [INFO] Listening at: http://0.0.0.0:5000 (8)
[2019-07-05 00:07:42 +0000] [8] [INFO] Using worker: gevent
[2019-07-05 00:07:43 +0000] [35] [INFO] Booting worker with pid: 35
[2019-07-05 00:07:43 +0000] [36] [INFO] Booting worker with pid: 36
[2019-07-05 00:07:43 +0000] [37] [INFO] Booting worker with pid: 37
[2019-07-05 00:07:43 +0000] [38] [INFO] Booting worker with pid: 38
[2019-07-05 00:07:43 +0000] [41] [INFO] Booting worker with pid: 41
[2019-07-05 00:07:43 +0000] [43] [INFO] Booting worker with pid: 43
[2019-07-05 00:07:43 +0000] [45] [INFO] Booting worker with pid: 45
[2019-07-05 00:07:43 +0000] [49] [INFO] Booting worker with pid: 49
[2019-07-05 00:07:43 +0000] [47] [INFO] Booting worker with pid: 47
[2019-07-05 00:07:49 +0000] [53] [INFO] Booting worker with pid: 53
[2019-07-05 00:07:50 +0000] [54] [INFO] Booting worker with pid: 54
[2019-07-05 00:07:53 +0000] [57] [INFO] Booting worker with pid: 57
[...]
```

The pod had the following resources settings:
```
resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi
```

Doubling everything fixed the issue.
One interesting thing we noticed was that when looking at `dmesg` on the host machine we can see that it is `segfault`-ing at `libcrypto` when accessing the server using SSL
Memory seems not to be an issue for me because I'm not loading any big model in memory. Workers just keep crashing and I can't see any error messages. Is there a fix for this? 
![image](https://user-images.githubusercontent.com/4526759/64958367-87ca9880-d8c1-11e9-9d19-bd1950d28e76.png)

same issue for me, any idea to fix ? python 3.6.3 with gunicorn 19.9.0
@MrKiven whta does your app? are you using stuff like request?
can someone provides a way to reproduce the issue?
It's a manager of several components that are executed in a pipeline. Some of them might start HTTP requests to other components on the same machine or on remote machines. Some of the modules of the pipeline can be executed in parallel but they're executed using a ThreadPoolExecutor. They don't use any shared objects but they only generate data structures that are later on aggregated in a single resulting one. 

Unfortunately I'm not sure if I can put together a minimal example without exposing the system we have.
requests does a lot of unsafe things with threads which sometimes fork a new process. I would advise to use another client. can you paste at least the lines you'reusing to do a request? Are you using its timeout feature?
One of them could be:

```
try:
     resp = requests.post(self._endpoint, json=request_data)

     if resp.status_code != 200:
          logger.critical("[Error]: status code is {}".format(resp.status_code))
          return None

     response = resp.json()
     return {"intent": response["intent"], "intent_ranking": response["intent_ranking"]}
except ConnectionError as exc:
     logger.critical("[Exception] {}".format(str(exc)))
     return None
```
Thanks. I will try to create a simple from it.

It would be cool anyway if someone can send us a pr that reproduce the behaviour either as an example or a unit test so we make sure we are actually fixing the right thing.
Not sure if it can help someone but I had the same issue while running a dockerized flask webapp  and solved it updating the base image of my dockerfile to `python:3.6.9-alpine`

Dmesg on the host showed a segfault on lilibpython3.6m.so.1.0:

`[626278.653010] gunicorn[19965]: segfault at 70 ip 00007f6423e7faee sp 00007ffc4e9a2a38 error 4 in libpython3.6m.so.1.0[7f6423d8a000+194000]`

My docker image was based on `python:3.6-alpine` and doing an `apk update` which was updating python to to 3.6.8.

As said changing the base image to `python:3.6.9-alpine` solved it for me

I faced the same challenge running a Flask + Docker + Kubernetes. Increasing the CPU and Memory limits solved it for me.
The same thing happened to us. Increasing resource limits fixed the problem.
This suddenly happened to me on macOS Catalina (not containerized).

What helped me is:

1. Installing openssl:
```bash
brew install openssl
```

2. Running and adding this to my `~/.zshrc`:
```bash
export DYLD_LIBRARY_PATH=/usr/local/opt/openssl/lib:$DYLD_LIBRARY_PATH
```

Source: https://stackoverflow.com/a/58445755/5811984

Am having similar challenge and would be grateful if someone can help me out.
This is what I had;

"root@ubuntu-s-1vcpu-1gb-nyc1-01:~# sudo systemctl status gunicorn.service ● gunicorn.service - gunicorn daemon    Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)    Active: active (running) since Mon 2020-02-24 07:48:04 UTC; 44min ago  Main PID: 4846 (gunicorn)     Tasks: 4 (limit: 1151)    CGroup: /system.slice/gunicorn.service            ├─4846 /home/bright/djangoprojectdir/djangoprojectenv/bin/python /home/bright/djangoprojectdir/djangoprojectenv/bin/gunicorn -            ├─4866 /home/bright/djangoprojectdir/djangoprojectenv/bin/python /home/bright/djangoprojectdir/djangoprojectenv/bin/gunicorn -            ├─4868 /home/bright/djangoprojectdir/djangoprojectenv/bin/python /home/bright/djangoprojectdir/djangoprojectenv/bin/gunicorn -            └─4869 /home/bright/djangoprojectdir/djangoprojectenv/bin/python /home/bright/djangoprojectdir/djangoprojectenv/bin/gunicorn -  Feb 24 07:48:04 ubuntu-s-1vcpu-1gb-nyc1-01 systemd[1]: Stopped gunicorn daemon. Feb 24 07:48:04 ubuntu-s-1vcpu-1gb-nyc1-01 systemd[1]: Started gunicorn daemon. Feb 24 07:48:05 ubuntu-s-1vcpu-1gb-nyc1-01 gunicorn[4846]: [2020-02-24 07:48:05 +0000] [4846] [INFO] Starting gunicorn 20.0.4 Feb 24 07:48:05 ubuntu-s-1vcpu-1gb-nyc1-01 gunicorn[4846]: [2020-02-24 07:48:05 +0000] [4846] [INFO] Listening at: unix:/run/gunicorn.soc Feb 24 07:48:05 ubuntu-s-1vcpu-1gb-nyc1-01 gunicorn[4846]: [2020-02-24 07:48:05 +0000] [4846] [INFO] Using worker: sync Feb 24 07:48:05 ubuntu-s-1vcpu-1gb-nyc1-01 gunicorn[4846]: [2020-02-24 07:48:05 +0000] [4866] [INFO] Booting worker with pid: 4866 Feb 24 07:48:05 ubuntu-s-1vcpu-1gb-nyc1-01 gunicorn[4846]: [2020-02-24 07:48:05 +0000] [4868] [INFO] Booting worker with pid: 4868 Feb 24 07:48:05 ubuntu-s-1vcpu-1gb-nyc1-01 gunicorn[4846]: [2020-02-24 07:48:05 +0000] [4869] [INFO] Booting worker with pid: 4869 Feb 24 08:03:41 ubuntu-s-1vcpu-1gb-nyc1-01 gunicorn[4846]:  - - [24/Feb/2020:08:03:41 +0000] "GET / HTTP/1.0" 400 26 "-" "Mozilla/5.0 (Wi lines 1-20/20 (END)" Can anyone please help me fix that? 
@BrightNana can you try to give a `dmesg` and see if you have any gunicorn errors?
`dmesg | grep gunicorn` could help filter out the other errors
Hello,
i have the same bug in debian 9 when i want to provide gunicorn as a systemd service. If I start it from the CLI, gunicorn runs without errors.

Extract from `dmesg | grep gunicorn`:
`
[ 6379.465022] gunicorn[28100]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]                                                                                                                                               [ 6379.686421] gunicorn[28103]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]                                                                                                                                               [ 6379.885803] gunicorn[28106]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]                                                                                                                                               [ 6380.079670] gunicorn[28109]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]                                                                                                                                               [ 6380.302551] gunicorn[28112]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]                                                                                                                                               [ 6380.525240] gunicorn[28115]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]                                                                                                                                               [ 6380.730884] gunicorn[28118]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]                                                                                                                                               [ 6380.898665] gunicorn[28121]: segfault at b0 ip 00007f06656e4c7c sp 00007ffcc060b030 error 4 in ld-2.24.so[7f06656d9000+23000]
`

Excerpt from `journalctl`:
`
Mär 12 07:01:06 build-server gunicorn[828]: [2020-03-12 07:01:06 +0100] [1054] [INFO] Booting worker with pid: 1054                                                                                                                                                       Mär 12 07:01:06 build-server gunicorn[828]: [2020-03-12 07:01:06 +0100] [1057] [INFO] Booting worker with pid: 1057                                                                                                                                                       Mär 12 07:01:06 build-server gunicorn[828]: [2020-03-12 07:01:06 +0100] [1060] [INFO] Booting worker with pid: 1060                                                                                                                                                       Mär 12 07:01:07 build-server gunicorn[828]: [2020-03-12 07:01:07 +0100] [1064] [INFO] Booting worker with pid: 1064                                                                                                                                                       Mär 12 07:01:07 build-server gunicorn[828]: [2020-03-12 07:01:07 +0100] [1067] [INFO] Booting worker with pid: 1067                                                                                                                                                       Mär 12 07:01:07 build-server gunicorn[828]: [2020-03-12 07:01:07 +0100] [1070] [INFO] Booting worker with pid: 1070                                                                                                                                                       Mär 12 07:01:07 build-server gunicorn[828]: [2020-03-12 07:01:07 +0100] [1073] [INFO] Booting worker with pid: 1073                                                                                                                                                       Mär 12 07:01:07 build-server gunicorn[828]: [2020-03-12 07:01:07 +0100] [1076] [INFO] Booting worker with pid: 1076                                                                                                                                                       Mär 12 07:01:08 build-server gunicorn[828]: [2020-03-12 07:01:08 +0100] [1079] [INFO] Booting worker with pid: 1079                                                                                                                                                       Mär 12 07:01:08 build-server gunicorn[828]: [2020-03-12 07:01:08 +0100] [1082] [INFO] Booting worker with pid: 1082                                                                                                                                                       Mär 12 07:01:08 build-server gunicorn[828]: [2020-03-12 07:01:08 +0100] [1085] [INFO] Booting worker with pid: 1085                                                                                                                                                       Mär 12 07:01:08 build-server gunicorn[828]: [2020-03-12 07:01:08 +0100] [1088] [INFO] Booting worker with pid: 1088                                                                                                                                                       Mär 12 07:01:08 build-server gunicorn[828]: [2020-03-12 07:01:08 +0100] [1091] [INFO] Booting worker with pid: 1091                                                                                                                                                       Mär 12 07:01:09 build-server gunicorn[828]: [2020-03-12 07:01:09 +0100] [1094] [INFO] Booting worker with pid: 1094 
`
Extract from `systemctl status`:
`
● api.service - API Server for BuildingChallenge served with Gunicorn                                                                                                                                                                                                             Loaded: loaded (/etc/systemd/system/api.service; disabled; vendor preset: enabled)                                                                                                                                                                                             Active: active (running) since Thu 2020-03-12 08:26:01 CET; 22min ago                                                                                                                                                                                                        Main PID: 8150 (gunicorn)                                                                                                                                                                                                                                                         Tasks: 3 (limit: 4915)                                                                                                                                                                                                                                                        Memory: 37.7M (high: 100.0M max: 500.0M)                                                                                                                                                                                                                                       CGroup: /system.slice/api.service                                                                                                                                                                                                                                                      ├─ 8150 /opt/api/venv/bin/python /opt/api/venv/bin/gunicorn --bind unix:api.sock wsgi:app                                                                                                                                                                                      ├─28936 /opt/api/venv/bin/python /opt/api/venv/bin/gunicorn --bind unix:api.sock wsgi:app                                                                                                                                                                                      └─28938 /usr/bin/python3 -Es /usr/bin/lsb_release -a                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Mär 12 08:48:01 build-server gunicorn[8150]: [2020-03-12 08:48:01 +0100] [28909] [INFO] Booting worker with pid: 28909                                                                                                                                                    Mär 12 08:48:01 build-server gunicorn[8150]: [2020-03-12 08:48:01 +0100] [28912] [INFO] Booting worker with pid: 28912                                                                                                                                                    Mär 12 08:48:01 build-server gunicorn[8150]: [2020-03-12 08:48:01 +0100] [28915] [INFO] Booting worker with pid: 28915                                                                                                                                                    Mär 12 08:48:01 build-server gunicorn[8150]: [2020-03-12 08:48:01 +0100] [28918] [INFO] Booting worker with pid: 28918                                                                                                                                                    Mär 12 08:48:01 build-server gunicorn[8150]: [2020-03-12 08:48:01 +0100] [28921] [INFO] Booting worker with pid: 28921                                                                                                                                                    Mär 12 08:48:01 build-server gunicorn[8150]: [2020-03-12 08:48:01 +0100] [28924] [INFO] Booting worker with pid: 28924                                                                                                                                                    Mär 12 08:48:02 build-server gunicorn[8150]: [2020-03-12 08:48:02 +0100] [28927] [INFO] Booting worker with pid: 28927                                                                                                                                                    Mär 12 08:48:02 build-server gunicorn[8150]: [2020-03-12 08:48:02 +0100] [28930] [INFO] Booting worker with pid: 28930                                                                                                                                                    Mär 12 08:48:02 build-server gunicorn[8150]: [2020-03-12 08:48:02 +0100] [28933] [INFO] Booting worker with pid: 28933                                                                                                                                                    Mär 12 08:48:02 build-server gunicorn[8150]: [2020-03-12 08:48:02 +0100] [28936] [INFO] Booting worker with pid: 28936 
`

Thanks for your help.
I put up a PR that might help debug these kinds of situations. Can anyone take a look?
https://github.com/benoitc/gunicorn/pull/2315
I had the same issue with a Flask application running inside Docker. The workers were restarting infinitely with an increasing process ID. 
![image](https://user-images.githubusercontent.com/9307041/84937512-277cb780-b0dc-11ea-8380-60df71fa9f17.png)

The issue was memory related for me, when I increased the memory allowed for Docker, the workers spawned up effectively. 
<img width="1108" alt="image" src="https://user-images.githubusercontent.com/9307041/84937460-116ef700-b0dc-11ea-8149-77882064e811.png">


Resolving patch:
diff --git a/gunicorn/arbiter.py b/gunicorn/arbiter.py
--- a/gunicorn/arbiter.py
+++ b/gunicorn/arbiter.py
@@ -527,6 +527,27 @@ def reap_workers(self):
                         reason = "App failed to load."
                         raise HaltServer(reason, self.APP_LOAD_ERROR)
 
+                    if exitcode > 0:
+                        # If the exit code of the worker is greater than 0,
+                        # let the user know.
+                        self.log.error("Worker (pid:%s) exited with code %s.",
+                                       wpid, exitcode)
+                    elif status > 0:
+                        # If the exit code of the worker is 0 and the status
+                        # is greater than 0, then it was most likely killed
+                        # via a signal.
+                        try:
+                            sig_name = signal.Signals(status).name
+                        except ValueError:
+                            sig_name = "code {}".format(status)
+                        msg = "Worker (pid:{}) was sent {}!".format(
+                            wpid, sig_name)
+
+                        # Additional hint for SIGKILL
+                        if status == signal.SIGKILL:
+                            msg += " Perhaps out of memory?"
+                        self.log.error(msg)
+
                     worker = self.WORKERS.pop(wpid, None)
                     if not worker:
                         continue


Test patch: