Repository: encode/uvicorn

Issue:
gunicorn keeps booting workers when exceptions raised at startup events
Hi, I have the same issues  when deploy FastAPI using gunicorn + uvicorn，it keeps booting workers when exceptions raised at startup events.

**main.py**

```python
from fastapi import FastAPI

app = FastAPI()


@app.on_event("startup")
async def startup():
    raise Exception("error")


@app.get("/")
async def hello():
    return {"msg": "hello world"}
```

**gunicorn.conf.py**

```python
import json
import os

worker_class = "uvicorn.workers.UvicornWorker"

host = os.getenv("HOST", "0.0.0.0")
port = os.getenv("PORT", "8000")
bind_env = os.getenv("BIND", None)
use_loglevel = os.getenv("LOG_LEVEL", "info")
if bind_env:
    use_bind = bind_env
else:
    use_bind = f"{host}:{port}"


accesslog_var = os.getenv("ACCESS_LOG", "-")
use_accesslog = accesslog_var or None
errorlog_var = os.getenv("ERROR_LOG", "-")
use_errorlog = errorlog_var or None
graceful_timeout_str = os.getenv("GRACEFUL_TIMEOUT", "120")
timeout_str = os.getenv("TIMEOUT", "120")
keepalive_str = os.getenv("KEEP_ALIVE", "5")

# Gunicorn config variables
loglevel = use_loglevel
bind = use_bind
errorlog = use_errorlog
accesslog = use_accesslog
graceful_timeout = int(graceful_timeout_str)
timeout = int(timeout_str)
keepalive = int(keepalive_str)

# For debugging and testing
log_data = {
    "loglevel": loglevel,
    "bind": bind,
    "graceful_timeout": graceful_timeout,
    "timeout": timeout,
    "keepalive": keepalive,
    "errorlog": errorlog,
    "accesslog": accesslog,
}
print(json.dumps(log_data))
```

**gunicorn log**

```text
{"loglevel": "info", "bind": "0.0.0.0:8000", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "host": "0.0.0.0", "port": "8000"}
[2021-06-01 21:13:05 +0800] [40946] [INFO] Starting gunicorn 20.0.4
[2021-06-01 21:13:05 +0800] [40946] [INFO] Listening at: http://0.0.0.0:8000 (40946)
[2021-06-01 21:13:05 +0800] [40946] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-06-01 21:13:05 +0800] [40949] [INFO] Booting worker with pid: 40949
[2021-06-01 21:13:05 +0800] [40949] [INFO] Started server process [40949]
[2021-06-01 21:13:05 +0800] [40949] [INFO] Waiting for application startup.
[2021-06-01 21:13:05 +0800] [40949] [ERROR] Traceback (most recent call last):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "./app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-01 21:13:05 +0800] [40949] [ERROR] Application startup failed. Exiting.
[2021-06-01 21:13:05 +0800] [40949] [INFO] Worker exiting (pid: 40949)
[2021-06-01 21:13:05 +0800] [40950] [INFO] Booting worker with pid: 40950
[2021-06-01 21:13:06 +0800] [40950] [INFO] Started server process [40950]
[2021-06-01 21:13:06 +0800] [40950] [INFO] Waiting for application startup.
[2021-06-01 21:13:06 +0800] [40950] [ERROR] Traceback (most recent call last):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "./app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-01 21:13:06 +0800] [40950] [ERROR] Application startup failed. Exiting.
```

I hope the gunicorn would exit when exceptions raised, but it keeps booting workers. Can someone helps me, thanks?

_Originally posted by @fanchunke1991 in https://github.com/encode/uvicorn/issues/1031#issuecomment-852121760_


Discussion:
this one is tricky to solve, I dont know gunicorn enough to have a clear view on what it needs to not restart failed workers here.
it should be interesting to see if solving this issue https://github.com/encode/uvicorn/issues/780 solves this one too
today i debug this problem，below are some findings.

**gunicorn/arbiter.py**

```python
...

def run(self):
    "Main master loop."
    self.start()
    util._setproctitle("master [%s]" % self.proc_name)

    try:
        self.manage_workers()

        while True:
            self.maybe_promote_master()

            sig = self.SIG_QUEUE.pop(0) if self.SIG_QUEUE else None
            if sig is None:
                self.sleep()
                self.murder_workers()
                self.manage_workers()
                continue

...
```

above is the main function to start workers by gunicorn. I think the main reason that gunicorn keeps booting workers is that `self.manage_workers()` returns with no exceptions even if workers boot failed, and then guicorn runs into the `While` loop.

**guicorn/arbiter.py**

```python
...

def spawn_worker(self):
    ...

    # Process Child
    worker.pid = os.getpid()
    try:
        util._setproctitle("worker [%s]" % self.proc_name)
        self.log.info("Booting worker with pid: %s", worker.pid)
        self.cfg.post_fork(self, worker)
        worker.init_process()
        sys.exit(0)
    except SystemExit:
        raise
    except AppImportError as e:
        self.log.debug("Exception while loading the application",
                       exc_info=True)
        print("%s" % e, file=sys.stderr)
        sys.stderr.flush()
        sys.exit(self.APP_LOAD_ERROR)
    except Exception:
        self.log.exception("Exception in worker process")
        if not worker.booted:
            sys.exit(self.WORKER_BOOT_ERROR)
        sys.exit(-1)
    finally:
        self.log.info("Worker exiting (pid: %s)", worker.pid)
        try:
            worker.tmp.close()
            self.cfg.worker_exit(self, worker)
        except Exception:
            self.log.warning("Exception during worker exit:\n%s",
                             traceback.format_exc())
``` 

**uvicorn/workers.py**

```python
def run(self) -> None:
      self.config.app = self.wsgi
      server = Server(config=self.config)
      loop = asyncio.get_event_loop()
      loop.run_until_complete(server.serve(sockets=self.sockets))
```

**uvicorn/server.py**

```python
async def serve(self, sockets=None):
    ......

    await self.startup(sockets=sockets)
    if self.should_exit:
        return
    await self.main_loop()
    await self.shutdown(sockets=sockets)
```

above is the main logic . gunicorn's `worker.init_process()` start wokers, uvicorn's `serve()` serve requests. But if uvicorn's startup events failed, it returns `None` instead of an exception, so gunicorn's child processes exit with `sys.exit(0)`

----

I did some changes, it seems solved the problem. The code is a bit ugly, please forgive me, maybe you guys have other better solutions.

**uvicorn/workers.py**

```python
def run(self) -> None:
      self.config.app = self.wsgi
      server = Server(config=self.config)
      loop = asyncio.get_event_loop()
      try:
          loop.run_until_complete(server.serve(sockets=self.sockets))
      except:
          self.booted = False
          raise
```

**uvicorn/server.py**

```python
async def serve(self, sockets=None):
    ......

    await self.startup(sockets=sockets)
    if self.should_exit:
        raise Exception("Booted failed")
        # return
    await self.main_loop()
    await self.shutdown(sockets=sockets)
```
ok thanks for the layout here, very helpful
would it be possible if you have time to check if this PR https://github.com/encode/uvicorn/pull/835 that exits the worker on a failed startup solves the infinite rety here ?
I have checked #835 PR, and it's for `uvicorn` to start the server. For `gunicorn`, I changed `UvicornWorker` a little like the PR.

**uvicorn/workers.py**

```python
import sys

from gunicorn.workers.base import Worker


class UvicornWorker(Worker):

    ......

    def run(self) -> None:
        self.config.app = self.wsgi
        server = Server(config=self.config)
        loop = asyncio.get_event_loop()
        loop.run_until_complete(server.serve(sockets=self.sockets))
        if not server.started:
            sys.exit(3)
```

If `uvicorn` workers don't start correctly, then workers will exit using `sys.exit(3)` and gunicorn stop booting. The exit code `3` defined in gunicorn which means `WORKER_BOOT_ERROR` has effect, but `1` doesn't. 

```text
[2021-06-08 23:44:11 +0800] [44375] [INFO] Starting gunicorn 20.1.0
[2021-06-08 23:44:11 +0800] [44375] [INFO] Listening at: http://0.0.0.0:8000 (44375)
[2021-06-08 23:44:11 +0800] [44375] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-06-08 23:44:11 +0800] [44378] [INFO] Booting worker with pid: 44378
[2021-06-08 23:44:11 +0800] [44378] [INFO] Started server process [44378]
[2021-06-08 23:44:11 +0800] [44378] [INFO] Waiting for application startup.
[2021-06-08 23:44:11 +0800] [44378] [ERROR] Traceback (most recent call last):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "./app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-08 23:44:11 +0800] [44378] [ERROR] Application startup failed. Exiting.
[2021-06-08 23:44:11 +0800] [44378] [INFO] Worker exiting (pid: 44378)
[2021-06-08 23:44:11 +0800] [44375] [INFO] Shutting down: Master
[2021-06-08 23:44:11 +0800] [44375] [INFO] Reason: Worker failed to boot.
```
thanks @fanchunke1991 , fancy sending a PR please ?
> thanks @fanchunke1991 , fancy sending a PR please ?

I'm glad to send a PR😁. blow are some updates about this issue.

I found `gunicorn` master process handles `SIGCHLD` signal when subprocess exits. If the exit_code equals `WORKER_BOOT_ERROR`, master process will then exit as well.

**gunicorn/arbiter.py**

```python
...
class Arbiter(object):
...

    def handle_chld(self, sig, frame):
        "SIGCHLD handling"
        self.reap_workers()
        self.wakeup()
...

    def reap_workers(self):
        """\
        Reap workers to avoid zombie processes
        """
        try:
            while True:
                wpid, status = os.waitpid(-1, os.WNOHANG)
                print(f"wpid: {wpid}, status: {status}, reexec_pid: {self.reexec_pid}")
                if not wpid:
                    break
                if self.reexec_pid == wpid:
                    self.reexec_pid = 0
                else:
                    # A worker was terminated. If the termination reason was
                    # that it could not boot, we'll shut it down to avoid
                    # infinite start/stop cycles.
                    exitcode = status >> 8
                    if exitcode == self.WORKER_BOOT_ERROR:
                        reason = "Worker failed to boot."
                        raise HaltServer(reason, self.WORKER_BOOT_ERROR)
                    if exitcode == self.APP_LOAD_ERROR:
                        reason = "App failed to load."
                        raise HaltServer(reason, self.APP_LOAD_ERROR)
...
```

I use `uvicorn==0.14.0` for testing, here are some logs:

```text
[2021-06-09 13:53:03 +0800] [49141] [INFO] Starting gunicorn 20.1.0
[2021-06-09 13:53:03 +0800] [49141] [INFO] Listening at: http://0.0.0.0:8000 (49141)
[2021-06-09 13:53:03 +0800] [49141] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-06-09 13:53:03 +0800] [49144] [INFO] Booting worker with pid: 49144
[2021-06-09 13:53:04 +0800] [49144] [INFO] Started server process [49144]
[2021-06-09 13:53:04 +0800] [49144] [INFO] Waiting for application startup.
[2021-06-09 13:53:04 +0800] [49144] [ERROR] Traceback (most recent call last):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "./app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-09 13:53:04 +0800] [49144] [ERROR] Application startup failed. Exiting.
[2021-06-09 13:53:04 +0800] [49144] [INFO] Worker exiting (pid: 49144)

wpid: 49144, status: 0, reexec_pid: 0

[2021-06-09 13:53:04 +0800] [49145] [INFO] Booting worker with pid: 49145
[2021-06-09 13:53:04 +0800] [49145] [INFO] Started server process [49145]
[2021-06-09 13:53:04 +0800] [49145] [INFO] Waiting for application startup.
[2021-06-09 13:53:04 +0800] [49145] [ERROR] Traceback (most recent call last):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "./app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-09 13:53:04 +0800] [49145] [ERROR] Application startup failed. Exiting.
[2021-06-09 13:53:04 +0800] [49145] [INFO] Worker exiting (pid: 49145)

wpid: 49145, status: 0, reexec_pid: 0

[2021-06-09 13:53:04 +0800] [49146] [INFO] Booting worker with pid: 49146
[2021-06-09 13:53:04 +0800] [49146] [INFO] Started server process [49146]
......
```

above logs show that `uvicorn` worker exit with exit_code `0` even if some exception raised at startup events.

If I do some changes on `UvicornWorker`

```python
import sys

from gunicorn.workers.base import Worker


class UvicornWorker(Worker):

    ......

    def run(self) -> None:
        self.config.app = self.wsgi
        server = Server(config=self.config)
        loop = asyncio.get_event_loop()
        loop.run_until_complete(server.serve(sockets=self.sockets))
        if not server.started:
            sys.exit(3)
```

Then the problem gone.

```text
[2021-06-09 14:05:11 +0800] [49308] [INFO] Starting gunicorn 20.1.0
[2021-06-09 14:05:11 +0800] [49308] [INFO] Listening at: http://0.0.0.0:8000 (49308)
[2021-06-09 14:05:11 +0800] [49308] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-06-09 14:05:11 +0800] [49311] [INFO] Booting worker with pid: 49311
[2021-06-09 14:05:11 +0800] [49311] [INFO] Started server process [49311]
[2021-06-09 14:05:11 +0800] [49311] [INFO] Waiting for application startup.
[2021-06-09 14:05:11 +0800] [49311] [ERROR] Traceback (most recent call last):
  File "/Users/fanchunke/.pyenv/versions/3.7.4/envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "/Users/fanchunke/.pyenv/versions/3.7.4/envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "/Users/fanchunke/.pyenv/versions/3.7.4/envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "/Users/fanchunke/Desktop/tmp/app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-09 14:05:11 +0800] [49311] [ERROR] Application startup failed. Exiting.
[2021-06-09 14:05:11 +0800] [49311] [INFO] Worker exiting (pid: 49311)

wpid: 49311, status: 768, reexec_pid: 0

[2021-06-09 14:05:11 +0800] [49308] [INFO] Shutting down: Master
[2021-06-09 14:05:11 +0800] [49308] [INFO] Reason: Worker failed to boot.
```
> > thanks @fanchunke1991 , fancy sending a PR please ?
> 
> I'm glad to send a PR. blow are some updates about this issue.

cool, you realistically have a good overview of the situation, a PR would be awesome to discuss the details !

Resolving patch:
diff --git a/uvicorn/workers.py b/uvicorn/workers.py
--- a/uvicorn/workers.py
+++ b/uvicorn/workers.py
@@ -1,8 +1,10 @@
 import asyncio
 import logging
 import signal
+import sys
 from typing import Any
 
+from gunicorn.arbiter import Arbiter
 from gunicorn.workers.base import Worker
 
 from uvicorn.config import Config
@@ -75,6 +77,11 @@ def run(self) -> None:
         server = Server(config=self.config)
         loop = asyncio.get_event_loop()
         loop.run_until_complete(server.serve(sockets=self.sockets))
+        # Exit with status 3 when worker starts failed, so Gunicorn
+        # can shut it down to avoid infinite start/stop cycles.
+        # See: https://github.com/encode/uvicorn/issues/1066
+        if not server.started:
+            sys.exit(Arbiter.WORKER_BOOT_ERROR)
 
     async def callback_notify(self) -> None:
         self.notify()


Test patch: