Repository: encode/uvicorn

Issue:
Shutdown process is broken in 0.15
### Checklist

<!-- Please make sure you check all these items before submitting your bug report. -->

- [x] The bug is reproducible against the latest release and/or `master`.
- [x] There are no similar issues or pull requests to fix it yet.

### Describe the bug

My FastAPI ASGI server cannot shutdown properly with uvicorn==0.15 while it can with 0.14

### To reproduce

Setup minimal FastAPI app and add some functions with logs(prints) to shutdown event

### Expected behavior

You see all logs(prints) from functions on shutdown

### Actual behavior

Get `ASGI 'lifespan' protocol appears unsupported.` without --lifespan on
Get error trace with --lifespan on

### Debugging material

uvicorn scheduler.main:app --host=0.0.0.0 --port ${WEB_PORT:-8000} --reload --lifespan on
INFO:     Will watch for changes in these directories: ['/home/dmytro/storage/chimplie/projects/raok-main/raok-scheduler']
INFO:     Uvicorn running on http://0.0.0.0:8004 (Press CTRL+C to quit)
INFO:     Started reloader process [177653] using statreload
INFO:     Started server process [177655]
INFO:     Waiting for application startup.
INFO:     Tortoise-ORM started, {'default': <tortoise.backends.asyncpg.client.AsyncpgDBClient object at 0x7f63d4a10e50>}, {'models': {'Task': <class 'scheduler.models.task.Task'>, 'Aerich': <class 'aerich.models.Aerich'>}}
INFO:     Application startup complete.
^CINFO:     Shutting down
INFO:     Finished server process [177655]
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/uvicorn/lifespan/on.py", line 84, in main
    await app(scope, self.receive, self.send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/starlette/middleware/errors.py", line 146, in __call__
    await self.app(scope, receive, send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/starlette/middleware/cors.py", line 70, in __call__
    await self.app(scope, receive, send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/starlette/exceptions.py", line 58, in __call__
    await self.app(scope, receive, send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/starlette/routing.py", line 569, in __call__
    await self.lifespan(scope, receive, send)
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/starlette/routing.py", line 544, in lifespan
    await receive()
  File "/home/dmytro/.local/share/virtualenvs/raok-scheduler-hpGGYNLi/lib/python3.8/site-packages/uvicorn/lifespan/on.py", line 135, in receive
    return await self.receive_queue.get()
  File "/usr/lib64/python3.8/asyncio/queues.py", line 163, in get
    await getter
asyncio.exceptions.CancelledError
INFO:     Stopping reloader process [177653]

![image](https://user-images.githubusercontent.com/29193926/129881063-c363dcd2-ed99-45cd-9cda-271d78c05ce8.png)

### Environment

- Fedora 34 / Python 3.8 / Uvicorn version: 0.15 - bug, 0.14 - ok
- command to run: `uvicorn main:app --host=0.0.0.0 --port 8000 --reload`

Revert "When receiving a SIGTERM supervisors should terminate their processes before joining them"
Reverts encode/uvicorn#1069

Closes #1160

I've taken my time to understand the situation. I'm going to explain it and then suggest a solution to the previous issue.

Right now, the issue on #1160 (`CancelledError`) is caused because the `CTRL + C` sends a `SIGINT (2)` to both parent and child processes, and given that we merged #1069, we have that the parent is also sending a `SIGTERM (15)` to the child. In other words, the child is receiving two signals, and the way `uvicorn` deals with multiple signals (two) is to forcefully exit the process.

To be more precise, when we press `CTRL + C` we send a `kill` signal to the process group, not to a single process.

Ok. Now, let's go back to the original issue: if we send `SIGINT` to the parent process, it doesn't terminate the children. And that's expected, because `kill -2 <uvicorn_pid>` will only send a signal to the `<uvicorn_pid>`.

The solution here would be to use `kill -2 -<uvicorn_pid>`, which sends a signal to the process group instead of only the parent. That being said, this solves the issue that #1069 (`process.terminate()`) solved, but we also avoid #1160 (`CancelledError`). 

Reference: https://stackoverflow.com/a/392155


Discussion:
When I press `CTRL + C` it sends two signals to `handle_exit()`, first a `SIGNIT` followed by a `SIGTERM`.

I'm debugging it.
Ok... It was quite fast to find it :sweat_smile: 

The problem is that on reload mode, we terminate the process on the `shutdown()`: https://github.com/encode/uvicorn/commit/de53c2366f1df9f55ac88d8bde67cfd314d2d98e
> Ok... It was quite fast to find it 
> 
> The problem is that on reload mode, we terminate the process on the `shutdown()`: [de53c23](https://github.com/encode/uvicorn/commit/de53c2366f1df9f55ac88d8bde67cfd314d2d98e)

seems more like an issue in op lifespan event, cant reproduce here
the actual app would help
> Ok... It was quite fast to find it 
> 
> The problem is that on reload mode, we terminate the process on the `shutdown()`: [de53c23](https://github.com/encode/uvicorn/commit/de53c2366f1df9f55ac88d8bde67cfd314d2d98e)

I see. I am using a script for same purpose (send SIGTERM to process group) in docker exactly to avoid dead reloader while server still working. It runs into the same problem with `CTRL+C` as it`s sending signal to whole group and as a result child processes receive two SIGTERMs instead of one.
In my case I decide to use a boolean that indicate if my script is propagating signals to process group or not

this simple app reloads fine and go through lifespan events fine too:

```
routes = [
    Route("/index", index),
]

async def lifespan(app):
    print(f'startup lifespan')
    yield
    print('shutdown lifespan')

app = Starlette(routes=routes, lifespan=lifespan)

```
```
/home/lotso/PycharmProjects/uvicorn/venv/bin/python -m uvicorn dev.app:app --reload --lifespan=on --reload-dir=dev --log-config=dev/logging_dev.yaml
INFO:     Will watch for changes in these directories: ['/home/lotso/PycharmProjects/uvicorn/dev']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [407820] using watchgod
WARNING   asyncio - 44 - [SFJAPI-dev]- Executing <Task pending name='Task-1' coro=<Server.serve() running at /home/lotso/PycharmProjects/uvicorn/./uvicorn/server.py:86> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fa9758adca0>()] created at /home/lotso/.asdf/installs/python/3.9.5/lib/python3.9/asyncio/locks.py:223> cb=[run_until_complete.<locals>.done_cb()] created at /home/lotso/.asdf/installs/python/3.9.5/lib/python3.9/asyncio/runners.py:44> took 0.167 seconds
startup lifespan
INFO:     Started server process [407827]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:38310 - 2021-08-19 09:28:30,450 -'GET / HTTP/1.1' 404 Not Found
INFO:     127.0.0.1:38310 - 2021-08-19 09:28:30,703 -'GET /favicon.ico HTTP/1.1' 404 Not Found
INFO:     127.0.0.1:38310 - 2021-08-19 09:28:35,012 -'GET /index HTTP/1.1' 200 OK
WARNING:  WatchGodReload detected file change in '['/home/lotso/PycharmProjects/uvicorn/dev/app.py']'. Reloading...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [407827]
shutdown lifespan
INFO:     Started server process [409414]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
WARNING   asyncio - 44 - [SFJAPI-dev]- Executing <Task pending name='Task-1' coro=<Server.serve() running at /home/lotso/PycharmProjects/uvicorn/./uvicorn/server.py:86> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f0b05d7ac10>()] created at /home/lotso/.asdf/installs/python/3.9.5/lib/python3.9/asyncio/locks.py:223> cb=[run_until_complete.<locals>.done_cb()] created at /home/lotso/.asdf/installs/python/3.9.5/lib/python3.9/asyncio/runners.py:44> took 0.167 seconds
startup lifespan
INFO:     127.0.0.1:38764 - 2021-08-19 09:29:01,921 -'GET /index HTTP/1.1' 200 OK
```
> > Ok... It was quite fast to find it
> > The problem is that on reload mode, we terminate the process on the `shutdown()`: [de53c23](https://github.com/encode/uvicorn/commit/de53c2366f1df9f55ac88d8bde67cfd314d2d98e)
> 
> seems more like an issue in op lifespan event, cant reproduce here
> the actual app would help

I`m using just fastapi app itself
![image](https://user-images.githubusercontent.com/29193926/130026925-8d162744-baa6-41dc-a2fe-c674a82c4d5b.png)

And it shutdown properly without `--reload` flag, and failed with it
@euri10 I'm reproducing your code, and it does raise `CancelledError` here (it doesn't handle the lifespan shutdown event). 

EDIT: On reload is fine. It's just when you `CTRL + C` there.
ok can get  the error on ctrl+c, thought it was the same as kill -2 which works fine, as well as kill -15
I wouldn't worry too much

That's because `kill` sends signal to single process, but `CTRL+C` sends signal to process group. That's why with reloaded up(which send additional signal to worker), web worker would receive two signals instead of one.
I get the same error with Starlette 0.16.0 when I send a `SIGINT`:

```
[2021-08-19 11:01:19,614][25024] INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
[2021-08-19 11:01:23,189][25024] ERROR:    Traceback (most recent call last):
  File "c:\Users\...\.venv\lib\site-packages\starlette\routing.py", line 624, in lifespan
    await receive()
  File "c:\Users\...\.venv\lib\site-packages\uvicorn\lifespan\on.py", line 135, in receive
    return await self.receive_queue.get()
  File "C:\Users\...\python\current\lib\asyncio\queues.py", line 166, in get
    await getter
asyncio.exceptions.CancelledError
```

I never had to create a lifespan function in previous versions, but I found that the error disappears if I bootstrap Starlette like so:

```python
async def _lifespan(app):
  import asyncio
  try:
    yield
  except asyncio.exceptions.CancelledError:
    pass
server_app = Starlette(routes=routes, lifespan=_lifespan)
```
I'm having the exactly same issue!! Here's the code https://github.com/igormcsouza/kitty-api/blob/master/api/__init__.py

I downgrade to 0.14 just to avoid error
Downgrade is not recommended.

Reload works normally, it's just when you CTRL+C, it will force exit, but network resources are closed anyway.
Even though the problem persist! I tried once more yesterday! That error at the end causes issues on my routine! I will keep on 0.14 until it is fixed!
I don't think we can avoid reverting that PR, but for reference, we can also do something like this:

https://github.com/benoitc/gunicorn/blob/1299ea9e967a61ae2edebe191082fd169b864c64/gunicorn/workers/gthread.py#L176-L181

In other words, check if the parent is alive, and then terminate the worker if it's not.
hypercorn just ignores the SIGINT before creating the processes: https://gitlab.com/pgjones/hypercorn/-/blob/master/src/hypercorn/run.py#L49-52. Then it adds a handler after: https://gitlab.com/pgjones/hypercorn/-/blob/master/src/hypercorn/run.py#L70-72.
@euri10 Can you check this, please?
> The solution here would be to use `kill -2 -<uvicorn_pid>`, which sends a signal to the process group instead of only the parent. 

I was originally interested in this issue (i.e., the previous one) because it was annoying to wait for my docker container to get killed on a timeout.  Keeping in mind that I am a stupid stupid idiot who can barely read my own name, does this mean that my docker environment is going to become rubbish again, or is there some way to add a hyphen to me running `docker-compose stop` or `^C`ing `docker-compose up`?
@Asday if you want the container to be killed immediately without waiting for a graceful shutdown I think you could do: 
```
docker-compose kill
```
That's not at all what I want, I want the SIGTERM to gracefully shut down uvicorn, including its workers.
Hm... if there is really no way to signal to docker to terminate the process group instead of only the parent process (which I don't know if that's the case or not because I'm not an expert on the ins and outs of docker (but if it is, that would seem a bit ridiculous in light of [standard CTRL+C behavior](https://unix.stackexchange.com/a/149756/502695))), one other option would be, instead of reverting the old PR, to never force shutdown on SIGTERMs... only on the second SIGINT, specifically. Which is basically equivalent to a monkeypatch I did to get around this:

```python
def _handle_exit(self, sig, frame):
    if self.should_exit and sig == signal.Signals.SIGINT:
        self.force_exit = True
    else:
        self.should_exit = True
 
uvicorn.server.Server.handle_exit = _handle_exit
```

Oooh, another option I just thought of: rather than starting the server process as a foreground process, the reloader process could start the server as a _background_ process. In that case, CTRL+C would _not_ send a SIGINT to the server process since it specifically only sends SIGINTs to "the **foreground** process group" (See [spec](https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap11.html#tag_11_01_09)), which would align behavior between CTRL + C and docker containers (I think). But I really have only spent about a half-hour researching this so I could be totally off-base. Any thoughts @Kludex @euri10 ?

Resolving patch:
diff --git a/uvicorn/server.py b/uvicorn/server.py
--- a/uvicorn/server.py
+++ b/uvicorn/server.py
@@ -310,7 +310,7 @@ def install_signal_handlers(self) -> None:
 
     def handle_exit(self, sig: signal.Signals, frame: FrameType) -> None:
 
-        if self.should_exit:
+        if self.should_exit and sig == signal.SIGINT:
             self.force_exit = True
         else:
             self.should_exit = True


Test patch: