Repository: huge-success/sanic
Pull request #1800 (2020-03-04 10:23:14)

Streaming migration for 20.3 release
This PR, meant for 20.3 release has changes to prepare for the streaming branch. Adds API compatibility and deprecations to ease transition to the streaming branch if merged in 20.6.

- Deprecation docstrings on request `body_init/push/finish`
- `request.receive_body` function
- Added tests for `receive_body` and `async for data in request.stream`


codecov[bot]: # [Codecov](https://codecov.io/gh/huge-success/sanic/pull/1800?src=pr&el=h1) Report
> Merging [#1800](https://codecov.io/gh/huge-success/sanic/pull/1800?src=pr&el=desc) into [master](https://codecov.io/gh/huge-success/sanic/commit/ce71514d71e1425ff4ed410c8eb741c700182538?src=pr&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `80%`.

[![Impacted file tree graph](https://codecov.io/gh/huge-success/sanic/pull/1800/graphs/tree.svg?width=650&token=NqLbJHUqoR&height=150&src=pr)](https://codecov.io/gh/huge-success/sanic/pull/1800?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1800      +/-   ##
==========================================
- Coverage   91.97%   91.71%   -0.27%     
==========================================
  Files          23       23              
  Lines        2319     2329      +10     
  Branches      428      431       +3     
==========================================
+ Hits         2133     2136       +3     
- Misses        143      149       +6     
- Partials       43       44       +1
```


| [Impacted Files](https://codecov.io/gh/huge-success/sanic/pull/1800?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sanic/request.py](https://codecov.io/gh/huge-success/sanic/pull/1800/diff?src=pr&el=tree#diff-c2FuaWMvcmVxdWVzdC5weQ==) | `98.23% <80%> (-1.41%)` | :arrow_down: |
| [sanic/response.py](https://codecov.io/gh/huge-success/sanic/pull/1800/diff?src=pr&el=tree#diff-c2FuaWMvcmVzcG9uc2UucHk=) | `96.73% <0%> (-1.97%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huge-success/sanic/pull/1800?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huge-success/sanic/pull/1800?src=pr&el=footer). Last update [ce71514...3a3e567](https://codecov.io/gh/huge-success/sanic/pull/1800?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/docs/sanic/config.rst b/docs/sanic/config.rst
index 0e34bcc7..366d22d1 100644
--- a/docs/sanic/config.rst
+++ b/docs/sanic/config.rst
@@ -115,15 +115,25 @@ Out of the box there are just a few predefined values which can be overwritten w
 +---------------------------+-------------------+-----------------------------------------------------------------------------+
 | KEEP_ALIVE_TIMEOUT        | 5                 | How long to hold a TCP connection open (sec)                                |
 +---------------------------+-------------------+-----------------------------------------------------------------------------+
+| WEBSOCKET_MAX_SIZE        | 2^20              | Maximum size for incoming messages (bytes)                                  |
++---------------------------+-------------------+-----------------------------------------------------------------------------+
+| WEBSOCKET_MAX_QUEUE       | 32                | Maximum length of the queue that holds incoming messages                    |
++---------------------------+-------------------+-----------------------------------------------------------------------------+
+| WEBSOCKET_READ_LIMIT      | 2^16              | High-water limit of the buffer for incoming bytes                           |
++---------------------------+-------------------+-----------------------------------------------------------------------------+
+| WEBSOCKET_WRITE_LIMIT     | 2^16              | High-water limit of the buffer for outgoing bytes                           |
++---------------------------+-------------------+-----------------------------------------------------------------------------+
 | GRACEFUL_SHUTDOWN_TIMEOUT | 15.0              | How long to wait to force close non-idle connection (sec)                   |
 +---------------------------+-------------------+-----------------------------------------------------------------------------+
 | ACCESS_LOG                | True              | Disable or enable access log                                                |
 +---------------------------+-------------------+-----------------------------------------------------------------------------+
-| PROXIES_COUNT             | -1                | The number of proxy servers in front of the app (e.g. nginx; see below)     |
+| FORWARDED_SECRET          | None              | Used to securely identify a specific proxy server (see below)               |
++---------------------------+-------------------+-----------------------------------------------------------------------------+
+| PROXIES_COUNT             | None              | The number of proxy servers in front of the app (e.g. nginx; see below)     |
 +---------------------------+-------------------+-----------------------------------------------------------------------------+
 | FORWARDED_FOR_HEADER      | "X-Forwarded-For" | The name of "X-Forwarded-For" HTTP header that contains client and proxy ip |
 +---------------------------+-------------------+-----------------------------------------------------------------------------+
-| REAL_IP_HEADER            | "X-Real-IP"       | The name of "X-Real-IP" HTTP header that contains real client ip            |
+| REAL_IP_HEADER            |  None             | The name of "X-Real-IP" HTTP header that contains real client ip            |
 +---------------------------+-------------------+-----------------------------------------------------------------------------+
 
 The different Timeout variables:
diff --git a/sanic/config.py b/sanic/config.py
index 539ea45c..58de2110 100644
--- a/sanic/config.py
+++ b/sanic/config.py
@@ -20,7 +20,7 @@ DEFAULT_CONFIG = {
     "RESPONSE_TIMEOUT": 60,  # 60 seconds
     "KEEP_ALIVE": True,
     "KEEP_ALIVE_TIMEOUT": 5,  # 5 seconds
-    "WEBSOCKET_MAX_SIZE": 2 ** 20,  # 1 megabytes
+    "WEBSOCKET_MAX_SIZE": 2 ** 20,  # 1 megabyte
     "WEBSOCKET_MAX_QUEUE": 32,
     "WEBSOCKET_READ_LIMIT": 2 ** 16,
     "WEBSOCKET_WRITE_LIMIT": 2 ** 16,
diff --git a/sanic/request.py b/sanic/request.py
index 246eb351..dca15901 100644
--- a/sanic/request.py
+++ b/sanic/request.py
@@ -56,6 +56,14 @@ class StreamBuffer:
         self._queue.task_done()
         return payload
 
+    async def __aiter__(self):
+        """Support `async for data in request.stream`"""
+        while True:
+            data = await self.read()
+            if not data:
+                break
+            yield data
+
     async def put(self, payload):
         await self._queue.put(payload)
 
@@ -127,40 +135,34 @@ class Request:
             self.__class__.__name__, self.method, self.path
         )
 
-    def get(self, key, default=None):
-        """.. deprecated:: 19.9
-           Custom context is now stored in `request.custom_context.yourkey`"""
-        return self.ctx.__dict__.get(key, default)
-
-    def __contains__(self, key):
-        """.. deprecated:: 19.9
-           Custom context is now stored in `request.custom_context.yourkey`"""
-        return key in self.ctx.__dict__
-
-    def __getitem__(self, key):
-        """.. deprecated:: 19.9
-           Custom context is now stored in `request.custom_context.yourkey`"""
-        return self.ctx.__dict__[key]
-
-    def __delitem__(self, key):
-        """.. deprecated:: 19.9
-           Custom context is now stored in `request.custom_context.yourkey`"""
-        del self.ctx.__dict__[key]
-
-    def __setitem__(self, key, value):
-        """.. deprecated:: 19.9
-           Custom context is now stored in `request.custom_context.yourkey`"""
-        setattr(self.ctx, key, value)
-
     def body_init(self):
+        """.. deprecated:: 20.3"""
         self.body = []
 
     def body_push(self, data):
+        """.. deprecated:: 20.3"""
         self.body.append(data)
 
     def body_finish(self):
+        """.. deprecated:: 20.3"""
         self.body = b"".join(self.body)
 
+    async def receive_body(self):
+        """Receive request.body, if not already received.
+
+        Streaming handlers may call this to receive the full body.
+
+        This is added as a compatibility shim in Sanic 20.3 because future
+        versions of Sanic will make all requests streaming and will use this
+        function instead of the non-async body_init/push/finish functions.
+
+        Please make an issue if your code depends on the old functionality and
+        cannot be upgraded to the new API.
+        """
+        if not self.stream:
+            return
+        self.body = b"".join([data async for data in self.stream])
+
     @property
     def json(self):
         if self.parsed_json is None:


Test patch:
diff --git a/tests/test_request_data.py b/tests/test_request_data.py
index 684561de..8136eeb3 100644
--- a/tests/test_request_data.py
+++ b/tests/test_request_data.py
@@ -44,44 +44,6 @@ def test_custom_context(app):
     }
 
 
-# Remove this once the deprecated API is abolished.
-def test_custom_context_old(app):
-    @app.middleware("request")
-    def store(request):
-        try:
-            request["foo"]
-        except KeyError:
-            pass
-        request["user"] = "sanic"
-        sidekick = request.get("sidekick", "tails")  # Item missing -> default
-        request["sidekick"] = sidekick
-        request["bar"] = request["sidekick"]
-        del request["sidekick"]
-
-    @app.route("/")
-    def handler(request):
-        return json(
-            {
-                "user": request.get("user"),
-                "sidekick": request.get("sidekick"),
-                "has_bar": "bar" in request,
-                "has_sidekick": "sidekick" in request,
-            }
-        )
-
-    request, response = app.test_client.get("/")
-
-    assert response.json == {
-        "user": "sanic",
-        "sidekick": None,
-        "has_bar": True,
-        "has_sidekick": False,
-    }
-    response_json = loads(response.text)
-    assert response_json["user"] == "sanic"
-    assert response_json.get("sidekick") is None
-
-
 def test_app_injection(app):
     expected = random.choice(range(0, 100))
 
diff --git a/tests/test_request_stream.py b/tests/test_request_stream.py
index b76e7248..17b0ac82 100644
--- a/tests/test_request_stream.py
+++ b/tests/test_request_stream.py
@@ -3,7 +3,7 @@ import pytest
 from sanic.blueprints import Blueprint
 from sanic.exceptions import HeaderExpectationFailed
 from sanic.request import StreamBuffer
-from sanic.response import stream, text
+from sanic.response import json, stream, text
 from sanic.views import CompositionView, HTTPMethodView
 from sanic.views import stream as stream_decorator
 
@@ -613,3 +613,43 @@ def test_request_stream(app):
     request, response = app.test_client.post("/bp_stream", data=data)
     assert response.status == 200
     assert response.text == data
+
+def test_streaming_new_api(app):
+    @app.post("/non-stream")
+    async def handler(request):
+        assert request.body == b"x"
+        await request.receive_body()  # This should do nothing
+        assert request.body == b"x"
+        return text("OK")
+
+    @app.post("/1", stream=True)
+    async def handler(request):
+        assert request.stream
+        assert not request.body
+        await request.receive_body()
+        return text(request.body.decode().upper())
+
+    @app.post("/2", stream=True)
+    async def handler(request):
+        ret = []
+        async for data in request.stream:
+            # We should have no b"" or None, just proper chunks
+            assert data
+            assert isinstance(data, bytes)
+            ret.append(data.decode("ASCII"))
+        return json(ret)
+
+    request, response = app.test_client.post("/non-stream", data="x")
+    assert response.status == 200
+
+    request, response = app.test_client.post("/1", data="TEST data")
+    assert request.body == b"TEST data"
+    assert response.status == 200
+    assert response.text == "TEST DATA"
+
+    request, response = app.test_client.post("/2", data=data)
+    assert response.status == 200
+    res = response.json
+    assert isinstance(res, list)
+    assert len(res) > 1
+    assert "".join(res) == data
diff --git a/tests/test_response.py b/tests/test_response.py
index aa72c3db..acb715db 100644
--- a/tests/test_response.py
+++ b/tests/test_response.py
@@ -15,6 +15,7 @@ from aiofiles import os as async_os
 from sanic.response import (
     HTTPResponse,
     StreamingHTTPResponse,
+    empty,
     file,
     file_stream,
     json,
@@ -22,7 +23,6 @@ from sanic.response import (
     stream,
     text,
 )
-from sanic.response import empty
 from sanic.server import HttpProtocol
 from sanic.testing import HOST, PORT