Repository: aio-libs/aiohttp

Issue:
Don't cancel web handler on disconnection
Raise an error on reading/writing instead.
CancelledError
### Describe the bug

Using the aiohttp web server, I struggle with `asyncio.CancelledError()` when a client disconnect for some time.
I shielded what needed to be but it has its limits.
Today I found this old PR:
https://github.com/aio-libs/aiohttp/pull/4080

It looks like my issue.
If I understand correctly  `asyncio.CancelledError()` was fixed at the beginning of v3.7.0 (1th October 2019).
But I still have frequent `asyncio.CancelledError()` in my server.

I have investigated a bit more and I find another PR, at the end of v3.7.0 (just before v3.7.0b0 I think) that for me re-introduce the issue:
https://github.com/aio-libs/aiohttp/pull/4415

For me the wrong code is the following:
```py
        if self._task_handler is not None:
            self._task_handler.cancel()
```    

It was removed by https://github.com/aio-libs/aiohttp/pull/4080 (and with all the documentation regarding this issue) but re-introduced silently by https://github.com/aio-libs/aiohttp/pull/4415.

I'm clearly not an expert so I'm not sure of what I'm saying, but does it make sense for you?

### To Reproduce

NA

### Expected behavior

NA

### Logs/tracebacks

```python-traceback
NA
```


### Python Version

```console
3.10.4
```


### aiohttp Version

```console
3.8.1
```


### multidict Version

```console
5.1.0
```


### yarl Version

```console
1.6.3
```


### OS

Fedora 35

### Related component

Server

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct


Discussion:
# [Codecov](https://codecov.io/gh/aio-libs/aiohttp/pull/4080?src=pr&el=h1) Report
> Merging [#4080](https://codecov.io/gh/aio-libs/aiohttp/pull/4080?src=pr&el=desc) into [master](https://codecov.io/gh/aio-libs/aiohttp/commit/3bc5bd6db16ebaf31deb351043abbfb29b461656?src=pr&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/aio-libs/aiohttp/pull/4080/graphs/tree.svg?width=650&token=LJn22G3rd7&height=150&src=pr)](https://codecov.io/gh/aio-libs/aiohttp/pull/4080?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4080      +/-   ##
==========================================
+ Coverage   97.81%   97.85%   +0.03%     
==========================================
  Files          43       43              
  Lines        8807     8820      +13     
  Branches     1378     1381       +3     
==========================================
+ Hits         8615     8631      +16     
+ Misses         78       76       -2     
+ Partials      114      113       -1
```


| [Impacted Files](https://codecov.io/gh/aio-libs/aiohttp/pull/4080?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [aiohttp/http\_websocket.py](https://codecov.io/gh/aio-libs/aiohttp/pull/4080/diff?src=pr&el=tree#diff-YWlvaHR0cC9odHRwX3dlYnNvY2tldC5weQ==) | `98.64% <100%> (+0.01%)` | :arrow_up: |
| [aiohttp/web\_request.py](https://codecov.io/gh/aio-libs/aiohttp/pull/4080/diff?src=pr&el=tree#diff-YWlvaHR0cC93ZWJfcmVxdWVzdC5weQ==) | `97.43% <100%> (+0.01%)` | :arrow_up: |
| [aiohttp/web\_protocol.py](https://codecov.io/gh/aio-libs/aiohttp/pull/4080/diff?src=pr&el=tree#diff-YWlvaHR0cC93ZWJfcHJvdG9jb2wucHk=) | `92.66% <100%> (+1.07%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/aio-libs/aiohttp/pull/4080?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/aio-libs/aiohttp/pull/4080?src=pr&el=footer). Last update [3bc5bd6...49584c3](https://codecov.io/gh/aio-libs/aiohttp/pull/4080?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

The PR is ready for review
I think the first one caused a regression (as mentioned at https://github.com/aio-libs/aiohttp/issues/4408#issuecomment-572033137), so the second one has reverted that.

Not sure if the documentation should be reverted or if the plan is to improve this later. Will need some feedback from @asvetlov. Personally, I like the fact that the handler can be aborted as soon as the connection has dropped.
Hey @Dreamsorcerer thanks a lot for the quick answer!

In the following comment that proposed a fix to the regression it was only about `self._waiter.cancel()` not `self._task_handler.cancel()`:
https://github.com/aio-libs/aiohttp/pull/4415#issuecomment-689356233

The final fix of the regression does both `self._waiter.cancel()` and  `self._task_handler.cancel()`, maybe `self._task_handler.cancel()` was a mistake?

> Not sure if the documentation should be reverted or if the plan is to improve this later. 

OK!

> Personally, I like the fact that the handler can be aborted as soon as the connection has dropped.

I find a lot of all issues on the subject, most closed because the subject was thought to be close.
Personally, the solution I prefer would be one discussed in the below issue, with an `allow_cancel` parameter per route:
https://github.com/aio-libs/aiohttp/issues/2492#issue-272673197

The solution of https://github.com/aio-libs/aiohttp/pull/4080 is not ok for me, I need some part of my code to keep executing even when the client disconnect.
But at least the situation would be clearer, and easier to hack in a future proof way, if the code and doc were aligned.


> Personally, the solution I prefer would be one discussed in the below issue, with an `allow_cancel` parameter per route: [#2492 (comment)](https://github.com/aio-libs/aiohttp/issues/2492#issue-272673197)

Well spotted. Appears there's already a solution in master which closes the issue, even if it's not allowing cancellation throughout the app.


I think you're probably right, and that line can be reverted. Please make a PR.
Have you already tested the change fixes it for you? Ideally, with `-Wall -X dev`, so you can check those warnings don't return.

> The solution of #4080 is not ok for me, I need some part of my code to keep executing even when the client disconnect.

Well, that's what `asyncio.shield()` is for, to avoid cancellation. So, you can still use that for now, until the revert is released.
Hey @Dreamsorcerer !

> Have you already tested the change fixes it for you? Ideally, with -Wall -X dev, so you can check those warnings don't return.

I'm sorry, I don't understand the ` -Wall -X dev` you are talking about, I think I don't know the project well enough.

I still tried a few things.

With the simplest server example from here https://github.com/aio-libs/aiohttp/tree/0215cdd4275ea0150f169545b25b66e1f5dc69f0#server, I reverted https://github.com/aio-libs/aiohttp/pull/4415 with the following diff:
```diff
--- a/aiohttp/web_protocol.py
+++ b/aiohttp/web_protocol.py
@@ -331,10 +331,10 @@ class RequestHandler(BaseProtocol):
                 exc = ConnectionResetError("Connection lost")
             self._current_request._cancel(exc)
 
-        if self._task_handler is not None:
-            self._task_handler.cancel()
-        if self._waiter is not None:
-            self._waiter.cancel()
 
         self._task_handler = None
 ```

However, I'm not able to reproduce the issue described in https://github.com/aio-libs/aiohttp/issues/4408, so I can't test if I will re-create the regression or not.
I tried both on master and on v3.8.0.

I can't try on v3.7.0 because my server doesn't start:
```py
$ python test.py                                                                               16:52:00
Traceback (most recent call last):
  File "/home/hoel/Tolteck/test.py", line 2, in <module>
    from aiohttp import web
  File "/home/hoel/.local/lib/python3.10/site-packages/aiohttp/__init__.py", line 6, in <module>
    from .client import BaseConnector as BaseConnector
  File "/home/hoel/.local/lib/python3.10/site-packages/aiohttp/client.py", line 35, in <module>
    from . import hdrs, http, payload
  File "/home/hoel/.local/lib/python3.10/site-packages/aiohttp/http.py", line 7, in <module>
    from .http_parser import HeadersParser as HeadersParser
  File "/home/hoel/.local/lib/python3.10/site-packages/aiohttp/http_parser.py", line 15, in <module>
    from .helpers import NO_EXTENSIONS, BaseTimerContext
  File "/home/hoel/.local/lib/python3.10/site-packages/aiohttp/helpers.py", line 612, in <module>
    class CeilTimeout(async_timeout.timeout):
TypeError: function() argument 'code' must be code, not str
```

I did another test with the following smaller diff:
```diff
--- a/aiohttp/web_protocol.py
+++ b/aiohttp/web_protocol.py
@@ -331,8 +331,8 @@ class RequestHandler(BaseProtocol):
                 exc = ConnectionResetError("Connection lost")
             self._current_request._cancel(exc)
 
-        if self._task_handler is not None:
-            self._task_handler.cancel()
         if self._waiter is not None:
             self._waiter.cancel()
 ```
With the above diff, on client disconnection, I don't have `CancelledError` exception anymore, and in fact I don't have any exception, no `OSError`.
I think the doc say we should have `OSError` https://docs.aiohttp.org/en/stable/web_advanced.html?highlight=disconnection#peer-disconnection so my fix doesn't do what the documentation expects.

For info I'm using Python 3.10.4

What do you think?





> I'm sorry, I don't understand the ` -Wall -X dev` you are talking about, I think I don't know the project well enough.

`python -Wall -X dev`
It enables all warnings and developer mode features in python.

> I can't try on v3.7.0 because my server doesn't start:

That error is caused by an incompatible version of async-timeout. You would need to reinstall the correct dependencies when switching between versions.

> With the above diff, on client disconnection, I don't have `CancelledError` exception anymore, and in fact I don't have any exception, no `OSError`.

It says you'll get OSError when reading/writing. It also says that aiohttp will handle it, so I'd expect no exception to appear in your logs. If you want to test it, you'd probably have to add something to the end of your handler (i.e. after the disconnection happens) like:
```
try:
    await request.text()
except OSError:
    print("OSError occurred")
```
@Dreamsorcerer Thanks for your help! PR opened https://github.com/aio-libs/aiohttp/pull/6727
Any chance you managed to reproduce the warnings from #4408 with `python -Wall -X dev`?
Yes I did! You can see in the PR I talk about it in the end of the commit&PR message.
Ah, perfect, haven't got to the PR just yet. Thanks.

Resolving patch:
diff --git a/aiohttp/web_protocol.py b/aiohttp/web_protocol.py
--- a/aiohttp/web_protocol.py
+++ b/aiohttp/web_protocol.py
@@ -311,8 +311,6 @@ def connection_lost(self, exc: Optional[BaseException]) -> None:
                 exc = ConnectionResetError("Connection lost")
             self._current_request._cancel(exc)
 
-        if self._task_handler is not None:
-            self._task_handler.cancel()
         if self._waiter is not None:
             self._waiter.cancel()
 


Test patch: