Repository: aio-libs/aiohttp (Python)

Issue:
Server closes connection if wrong type returned by handler
🐞 **Describe the bug**
<!-- A clear and concise description of what the bug is, on the next line. -->
Server closes session if handler returns anything else that `Response` instance.

💡 **To Reproduce**
<!-- How to reproduce the behavior?

For example:
1. Have certain environment
2. Run given code snippet in a certain way
3. See some behavior described

Add these steps below this comment: -->
```python
from aiohttp import web


async def handler(req):
    return 1

app = web.Application()
app.add_routes([web.get("/", handler)])
web.run_app(app)
```

```bash
~ curl localhost:8080
curl: (52) Empty reply from server
```

💡 **Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->
Error message in console and returned status 500.

📋 **Your version of the Python**
<!-- Attach your version of the Python. -->
```console
$ python --version
Python 3.8.1
```

📋 **Your version of the aiohttp/yarl/multidict distributions**
<!-- Attach your version of the distributions in the code blocks below. -->
Tested with aiohttp `3.6.2` and `master` branch fresh installations.



Discussion:
@Fogapod thanks for the report! It'd be cool if you could submit a failing test marked with xfail in a PR as per https://pganssle-talks.github.io/xfail-lightning.
@webknjaz alright, I'll try to make a test for this issue.

I found that returning `HTTPException` instance in aiohttp 3.6.2 resulted in returning corresponding status code by server, in master branch it results in closed connection.
Is returning exceptions deprecated? I didn't find it in documentation, there is no traceback in console.
I think it is, or should be. Exceptions must be raised.
`RuntimeError` is raised here https://github.com/aio-libs/aiohttp/blob/f80029b553ec041370babf75d07e2abb9b2bdca3/aiohttp/web_protocol.py#L577-L579
after calling https://github.com/aio-libs/aiohttp/blob/f80029b553ec041370babf75d07e2abb9b2bdca3/aiohttp/web_protocol.py#L480
and catched here https://github.com/aio-libs/aiohttp/blob/f80029b553ec041370babf75d07e2abb9b2bdca3/aiohttp/web_protocol.py#L523-L526
For some reason these are only logged in asyncio debug mode.

I'm not sure how this could be fixed. Returning status 500 for all Runtime errors?
> Returning status 500 for all Runtime errors?

I think so
> Is returning exceptions deprecated? I didn't find it in documentation, there is no traceback in console.

v3.6 deprecates returning exceptions but it still works (see the red warning box): https://docs.aiohttp.org/en/stable/web_quickstart.html#exceptions

master/v4.0 doesn't support this anymore and doesn't mention returning exceptions: https://docs.aiohttp.org/en/latest/web_exceptions.html

I don't understand this condition: https://github.com/aio-libs/aiohttp/blob/f80029b553ec041370babf75d07e2abb9b2bdca3/aiohttp/web_protocol.py#L524-L526
What kind of exceptions can happen there and why they are silenced by default?
If there are exceptions that should be silenced and cause server to close connection without response, what could a workaround be? Raising different kind of exception from handler if wrong type returned?

Edit: according to this comment https://github.com/aio-libs/aiohttp/issues/1790#issuecomment-292745428, runtime exceptions should not be handled, but in this case it can be, so exception type should be changed?



Patch:
diff --git a/CHANGES/8845.bugfix.rst b/CHANGES/8845.bugfix.rst
new file mode 100644
index 00000000000..ff0016ac14b
--- /dev/null
+++ b/CHANGES/8845.bugfix.rst
@@ -0,0 +1,1 @@
+Changed behaviour when returning an invalid response to send a 500 response -- by :user:`Dreamsorcerer`.
diff --git a/aiohttp/web_protocol.py b/aiohttp/web_protocol.py
index 5e12d611ae2..e0c174e0649 100644
--- a/aiohttp/web_protocol.py
+++ b/aiohttp/web_protocol.py
@@ -39,7 +39,7 @@
 from .log import access_logger, server_logger
 from .streams import EMPTY_PAYLOAD, StreamReader
 from .tcp_helpers import tcp_keepalive
-from .web_exceptions import HTTPException
+from .web_exceptions import HTTPException, HTTPInternalServerError
 from .web_log import AccessLogger
 from .web_request import BaseRequest
 from .web_response import Response, StreamResponse
@@ -490,18 +490,18 @@ async def _handle_request(
                 status=exc.status, reason=exc.reason, text=exc.text, headers=exc.headers
             )
             resp._cookies = exc._cookies
-            reset = await self.finish_response(request, resp, start_time)
+            resp, reset = await self.finish_response(request, resp, start_time)
         except asyncio.CancelledError:
             raise
         except asyncio.TimeoutError as exc:
             self.log_debug("Request handler timed out.", exc_info=exc)
             resp = self.handle_error(request, 504)
-            reset = await self.finish_response(request, resp, start_time)
+            resp, reset = await self.finish_response(request, resp, start_time)
         except Exception as exc:
             resp = self.handle_error(request, 500, exc)
-            reset = await self.finish_response(request, resp, start_time)
+            resp, reset = await self.finish_response(request, resp, start_time)
         else:
-            reset = await self.finish_response(request, resp, start_time)
+            resp, reset = await self.finish_response(request, resp, start_time)
         finally:
             self._handler_waiter.set_result(None)
 
@@ -603,10 +603,6 @@ async def start(self) -> None:
             except asyncio.CancelledError:
                 self.log_debug("Ignored premature client disconnection ")
                 break
-            except RuntimeError as exc:
-                if self._loop.get_debug():
-                    self.log_exception("Unhandled runtime exception", exc_info=exc)
-                self.force_close()
             except Exception as exc:
                 self.log_exception("Unhandled exception", exc_info=exc)
                 self.force_close()
@@ -635,7 +631,7 @@ async def start(self) -> None:
 
     async def finish_response(
         self, request: BaseRequest, resp: StreamResponse, start_time: float
-    ) -> bool:
+    ) -> Tuple[StreamResponse, bool]:
         """Prepare the response and write_eof, then log access.
 
         This has to
@@ -654,22 +650,26 @@ async def finish_response(
             prepare_meth = resp.prepare
         except AttributeError:
             if resp is None:
-                raise RuntimeError("Missing return " "statement on request handler")
+                self.log_exception("Missing return statement on request handler")
             else:
-                raise RuntimeError(
-                    "Web-handler should return "
-                    "a response instance, "
+                self.log_exception(
+                    "Web-handler should return a response instance, "
                     "got {!r}".format(resp)
                 )
+            exc = HTTPInternalServerError()
+            resp = Response(
+                status=exc.status, reason=exc.reason, text=exc.text, headers=exc.headers
+            )
+            prepare_meth = resp.prepare
         try:
             await prepare_meth(request)
             await resp.write_eof()
         except ConnectionError:
             await self.log_access(request, resp, start_time)
-            return True
-        else:
-            await self.log_access(request, resp, start_time)
-            return False
+            return resp, True
+
+        await self.log_access(request, resp, start_time)
+        return resp, False
 
     def handle_error(
         self,


Test patch:
diff --git a/tests/test_web_functional.py b/tests/test_web_functional.py
index d82fee43623..17eec2bb368 100644
--- a/tests/test_web_functional.py
+++ b/tests/test_web_functional.py
@@ -103,12 +103,8 @@ async def handler(request):
     server = await aiohttp_server(app, logger=logger)
     client = await aiohttp_client(server)
 
-    with pytest.raises(aiohttp.ServerDisconnectedError):
-        await client.get("/")
-
-    logger.exception.assert_called_with(
-        "Unhandled runtime exception", exc_info=mock.ANY
-    )
+    async with client.get("/") as resp:
+        assert resp.status == 500
 
 
 async def test_handler_returns_none(aiohttp_server: Any, aiohttp_client: Any) -> None:
@@ -123,13 +119,8 @@ async def handler(request):
     server = await aiohttp_server(app, logger=logger)
     client = await aiohttp_client(server)
 
-    with pytest.raises(aiohttp.ServerDisconnectedError):
-        await client.get("/")
-
-    # Actual error text is placed in exc_info
-    logger.exception.assert_called_with(
-        "Unhandled runtime exception", exc_info=mock.ANY
-    )
+    async with client.get("/") as resp:
+        assert resp.status == 500
 
 
 async def test_head_returns_empty_body(aiohttp_client: Any) -> None:
diff --git a/tests/test_web_request.py b/tests/test_web_request.py
index 58394886534..b78c4811fa6 100644
--- a/tests/test_web_request.py
+++ b/tests/test_web_request.py
@@ -12,7 +12,6 @@
 from yarl import URL
 
 from aiohttp import HttpVersion, web
-from aiohttp.client_exceptions import ServerDisconnectedError
 from aiohttp.http_parser import RawRequestMessage
 from aiohttp.streams import StreamReader
 from aiohttp.test_utils import make_mocked_request
@@ -820,22 +819,6 @@ def test_weakref_creation() -> None:
     weakref.ref(req)
 
 
-@pytest.mark.xfail(
-    raises=ServerDisconnectedError,
-    reason="see https://github.com/aio-libs/aiohttp/issues/4572",
-)
-async def test_handler_return_type(aiohttp_client: Any) -> None:
-    async def invalid_handler_1(request):
-        return 1
-
-    app = web.Application()
-    app.router.add_get("/1", invalid_handler_1)
-    client = await aiohttp_client(app)
-
-    async with client.get("/1") as resp:
-        assert 500 == resp.status
-
-
 @pytest.mark.parametrize(
     ["header", "header_attr"],
     [