Repository: sanic-org/sanic

Issue:
Possible bug in response.py when using Sanic + ASGI (chunked stream)
**Describe the bug**
In response.py write method, if chunked==true the code will push the data into the stream specifying the len:
```python
if self.chunked:
            await self.protocol.push_data(b"%x\r\n%b\r\n" % (len(data), data))
```
It looks like that uvicorn and daphne already execute this in httptools._impl.py (uvicorn>protocol>http) resulting in changing what the client will consider the body of the response.
If sanic is not using ASGI obviously we need that operation but if we use ASGI the content is prepared by the corresponding gateway.

**How to reproduce**
It is enough to stream a generic file and observe the response received by the client. With text files (like in the example) it's not big issue, but if you deal with media files, the decoder on client-side will just fail (I figured this out trying to stream mp4).

main.py
```python
import os
from sanic import Sanic
from sanic.handlers import ContentRangeHandler
from sanic.exceptions import NotFound, HeaderNotFound, InvalidUsage, SanicException
from sanic import Blueprint, response
from aiofiles import os as async_os
from sanic.response import file_stream
import uvicorn

app = Sanic(__name__)

@app.route("/test.txt")
async def handler_file_stream(request):
    return await response.file_stream(
        "./test.txt",
        chunk_size=1024
    )

if __name__ == "__main__":
    #app.run(host="0.0.0.0", port=8000, debug=False)
    uvicorn.run(app, host="0.0.0.0", port=8000, workers=1, log_level="debug")
```
test.txt
```txt
"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut 
labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris 
nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit 
esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt
 in culpa qui officia deserunt mollit anim id est laborum."
```

*Note*: setting chunked=True in the response.file_stream() will make the response working but it is not the default value.

**Environment:**
 - OS: Tested with MacOS and Linux Ubuntu 20
 - Version 20.9.1, uvicorn 0.12.2, daphne 3.0.0
- Python 3.8

Thanks!


Discussion:
Hi @logtheta 
Looks like this is very similar to, but not the same as https://github.com/huge-success/sanic/issues/1730
https://github.com/huge-success/sanic/issues/1730 was fixed by https://github.com/huge-success/sanic/pull/1957 and released in v20.9.1, but looks like it doesn't fix _this_ issue.
I'll look into whether it might be a quick fix.
Here in uvicorn, for reference: https://github.com/encode/uvicorn/blob/6468b70c85e10ae2d405165ca69f2b6a5bd55878/uvicorn/protocols/http/httptools_impl.py#L512
@ashleysommer Thanks for looking into 🙏 . Yes it looks similar to #1730, I think it is worth to fix it since the chunked feature is very useful, I personally use it for video pseduo streaming.

Thanks again!
Looks like the way we're doing chunked-encoding is completely incompatible with the ASGI-spec here:
https://asgi.readthedocs.io/en/latest/specs/www.html#response-start-send-event

Thats why our current solution works if httpx is the asgi-response-transport, but not if uvicorn or daphne are the transport.
Thats why the tests pass, but we see breakage in real-world applications.

@huge-success/sanic-core-devs See the section in the spec regarding 
>You may send a Transfer-Encoding header in this message, but the server must ignore it. Servers handle Transfer-Encoding themselves, and may opt to use Transfer-Encoding: chunked if the application presents a response that has no Content-Length set.

So in asgi-mode we need to _not_ set the `Transfer-Encoding: chunked` header, and to signal to the asgi-transport we're doing chunked mode, we need to _not_ specify a "content-length" header, then the asgi-transport will do its own chunking, based on our subsequent http.response.body messages.
@ashleysommer  I agree, it should be ASGI's responsibility to "prepare" the chunking (when from Sanic we just set the flag). At the beginning I thought it was uvicorn' issue but then I realized that other gateways were doing the same so...I decided to open the issue. 
If anything, the bug is in the `httpx` asgi-response-transport mechanism, because thats what we test against. It doesn't do _any_ chunking at the response level, so that lead us to assume we still do it at the Sanic level.
But response chunking is optional for the ASGI transport, so its not really a bug that it doesn't do it. The httpx transport just gathers up all of the async body parts and sends it when its done, rather than doing chunked transport, which is a perfectly valid way of doing it if the ASGI-transport doesn't support chunked transport.
I've got a fix made, just need to package it up into a nice PR for master, and for the 20.9.x series, and also for the 19.12.x LTS series

Resolving patch:
diff --git a/sanic/app.py b/sanic/app.py
--- a/sanic/app.py
+++ b/sanic/app.py
@@ -1454,6 +1454,8 @@ async def __call__(self, scope, receive, send):
         asgi_app = await ASGIApp.create(self, scope, receive, send)
         await asgi_app()
 
+    _asgi_single_callable = True  # We conform to ASGI 3.0 single-callable
+
     # -------------------------------------------------------------------- #
     # Configuration
     # -------------------------------------------------------------------- #
diff --git a/sanic/asgi.py b/sanic/asgi.py
--- a/sanic/asgi.py
+++ b/sanic/asgi.py
@@ -312,13 +312,19 @@ async def __call__(self) -> None:
         callback = None if self.ws else self.stream_callback
         await handler(self.request, None, callback)
 
-    async def stream_callback(self, response: HTTPResponse) -> None:
+    _asgi_single_callable = True  # We conform to ASGI 3.0 single-callable
+
+    async def stream_callback(
+        self, response: Union[HTTPResponse, StreamingHTTPResponse]
+    ) -> None:
         """
         Write the response.
         """
         headers: List[Tuple[bytes, bytes]] = []
         cookies: Dict[str, str] = {}
+        content_length: List[str] = []
         try:
+            content_length = response.headers.popall("content-length", [])
             cookies = {
                 v.key: v
                 for _, v in list(
@@ -351,12 +357,22 @@ async def stream_callback(self, response: HTTPResponse) -> None:
             ]
 
         response.asgi = True
-
-        if "content-length" not in response.headers and not isinstance(
-            response, StreamingHTTPResponse
-        ):
+        is_streaming = isinstance(response, StreamingHTTPResponse)
+        if is_streaming and getattr(response, "chunked", False):
+            # disable sanic chunking, this is done at the ASGI-server level
+            setattr(response, "chunked", False)
+            # content-length header is removed to signal to the ASGI-server
+            # to use automatic-chunking if it supports it
+        elif len(content_length) > 0:
             headers += [
-                (b"content-length", str(len(response.body)).encode("latin-1"))
+                (b"content-length", str(content_length[0]).encode("latin-1"))
+            ]
+        elif not is_streaming:
+            headers += [
+                (
+                    b"content-length",
+                    str(len(getattr(response, "body", b""))).encode("latin-1"),
+                )
             ]
 
         if "content-type" not in response.headers:
diff --git a/sanic/response.py b/sanic/response.py
--- a/sanic/response.py
+++ b/sanic/response.py
@@ -100,6 +100,8 @@ async def write(self, data):
         """
         data = self._encode_body(data)
 
+        # `chunked` will always be False in ASGI-mode, even if the underlying
+        # ASGI Transport implements Chunked transport. That does it itself.
         if self.chunked:
             await self.protocol.push_data(b"%x\r\n%b\r\n" % (len(data), data))
         else:


Test patch:
diff --git a/tests/test_response.py b/tests/test_response.py
--- a/tests/test_response.py
+++ b/tests/test_response.py
@@ -238,7 +238,7 @@ def test_chunked_streaming_returns_correct_content(streaming_app):
 @pytest.mark.asyncio
 async def test_chunked_streaming_returns_correct_content_asgi(streaming_app):
     request, response = await streaming_app.asgi_client.get("/")
-    assert response.text == "4\r\nfoo,\r\n3\r\nbar\r\n0\r\n\r\n"
+    assert response.text == "foo,bar"
 
 
 def test_non_chunked_streaming_adds_correct_headers(non_chunked_streaming_app):