Repository: encode/httpx

Issue:
Garbage collected responses should warn if still open.
See https://github.com/encode/httpx/issues/393#issuecomment-538370554
Reviewing the streaming API
Right now we issue streaming requests using...

```python
response = await client.get(url, stream=True)
```

And have the following stream-specific methods...

* `.read()`
* `.raw()`
* `.stream()`
* `.stream_text()`
* `.stream_lines()`
* `.close()`

For the 1.0 release I think we should consider? droping `stream=True`, as it makes it *exceptionally* easy to leave unclosed connections hanging around.

I'm interested in if we might want to use the following instead...

```python
async with client.stream('GET', url) as response:  # Can only be used as a context manager.
    ...
```

That way users never have to think about "did I call close on the response", or leave open connections hanging around accidentally.

This is related to #393, but it's a stricter API, that doesn't rely on "users *might* use a response as a context manger". And also makes it far clearer *where and when* the streaming methods should be used. Ie. they only make sense inside a `.stream()` block.

I also think we could rationalize the streaming methods, so...

* `.read()`
* `.stream_raw()`
* `.stream_bytes()`
* `.stream_text()`
* `.stream_lines()`
* `.close()`

Discussion:

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e6d5a84..a589d30 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,11 +4,34 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## Master
+
+### Added
+
+- Added concurrency backend autodetection. (Pull #585)
+- Added `Client(backend='trio')` and `Client(backend='asyncio')` API. (Pull #585)
+- Added `response.stream_lines()` API. (Pull #575)
+- Added `response.is_error` API. (Pull #574)
+- Added support for `timeout=Timeout(5.0, connect_timeout=60.0)` styles. (Pull #593)
+
+### Fixed
+
+- Requests or Clients with `timeout=None` now correctly always disable timeouts. (Pull #592)
+- Request 'Authorization' headers now have priority over `.netrc` authentication info. (Commit 095b691)
+- Files without a filename no longer set a Content-Type in multipart data. (Commit ed94950)
+
+### Changed
+
+- HTTP/2 support is switched to "off by default", but can be enabled explicitly. (Pull #584)
+- Switched to `Client(http2=True)` API from `Client(http_versions=["HTTP/1.1", "HTTP/2"])`. (Pull #586)
+- Pool timeouts are now on the timeout configuration, not the pool limits configuration. (Pull #563)
+- The timeout configuration is now named `httpx.Timeout(...)`, not `httpx.TimeoutConfig(...)`. The old version currently remains as a synonym for backwards compatability.  (Pull #591)
+
 ## 0.8.0 (November 27, 2019)
 
 ### Removed
 
-- The syncronous API has been removed, in order to allow us to fundamentally change how we approach supporting both sync and async variants. (See #588 for more details.)
+- The synchronous API has been removed, in order to allow us to fundamentally change how we approach supporting both sync and async variants. (See #588 for more details.)
 
 ## 0.7.8 (November 17, 2019)
 
diff --git a/docs/compatibility.md b/docs/compatibility.md
index d652a0a..f77512f 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -10,11 +10,10 @@ Pretty much any API mentioned in the `requests` QuickStart should be identical
 to the API in our own documentation. The following exceptions apply:
 
 * `Response.url` - Returns a `URL` instance, rather than a string. Use `str(response.url)` if you need a string instance.
-* `httpx.codes` - In our documentation we prefer the uppercased versions, such as `codes.NOT_FOUND`,
-but also provide lower-cased versions for API compatibility with `requests`.
-* `stream=True`. - Streaming responses provide the `.stream()` and `.raw()` byte iterator interfaces, rather than the `.iter_content()` method and the `.raw` socket interface.
+* `httpx.codes` - In our documentation we prefer the uppercased versions, such as `codes.NOT_FOUND`, but also provide lower-cased versions for API compatibility with `requests`.
+* `stream()`. - HTTPX provides a `.stream()` interface rather than using `stream=True`. This ensures that streaming responses are always properly closed outside of the stream block, and makes it visually clearer at which points streaming I/O APIs may be used with a response. Streaming request data is made avialable with `.stream_bytes()`, `.stream_text()`, `.stream_lines()`, and `.stream_raw()`.
 * `.get`, `.delete`, `.head`, `.options` -  These methods do not support `files`, `data`, or `json` arguments. Use `.request` if you need to need to send data using these http methods.
-* We don't support `response.is_ok` since the naming is ambiguous there, and might incorrectly imply an equivalence to `response.status_code == codes.OK`. Instead we provide an `.is_error` property.
+* We don't support `response.is_ok` since the naming is ambiguous there, and might incorrectly imply an equivalence to `response.status_code == codes.OK`. Instead we provide the `response.is_error` property. Use `if not response.is_error:` instead of `if response.is_ok:`.
 
 ## Advanced Usage
 
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 636dfe6..01d2215 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -292,11 +292,57 @@ The `Headers` data type is case-insensitive, so you can use any capitalization.
 'application/json'
 ```
 
-Multiple values for a single response header are represented as a single comma-separated
-value, as per [RFC 7230](https://tools.ietf.org/html/rfc7230#section-3.2):
+Multiple values for a single response header are represented as a single comma-separated value, as per [RFC 7230](https://tools.ietf.org/html/rfc7230#section-3.2):
 
 > A recipient MAY combine multiple header fields with the same field name into one “field-name: field-value” pair, without changing the semantics of the message, by appending each subsequent field-value to the combined field value in order, separated by a comma.
 
+## Streaming Responses
+
+For large downloads you may want to use streaming responses that do not load the entire response body into memory at once.
+
+You can stream the binary content of the response...
+
+```
+>>> async with httpx.stream("GET", "https://www.example.com") as r:
+...     async for data in r.stream_bytes():
+...         print(data)
+```
+
+Or the text of the response...
+
+```
+>>> async with httpx.stream("GET", "https://www.example.com") as r:
+...     async for text in r.stream_text():
+...         print(text)
+```
+
+Or stream the text, on a line-by-line basis...
+
+```
+>>> async with httpx.stream("GET", "https://www.example.com") as r:
+...     async for line in r.stream_lines():
+...         print(line)
+```
+
+HTTPX will use universal line endings, normalising all cases to `\n`.
+
+In some cases you might want to access the raw bytes on the response without applying any HTTP content decoding. In this case any content encoding that the web server has applied such as `gzip`, `deflate`, or `brotli` will not be automatically decoded.
+
+```
+>>> async with httpx.stream("GET", "https://www.example.com") as r:
+...     async for chunk in r.stream_raw():
+...         print(chunk)
+```
+
+If you're using streaming responses in any of these ways then the `response.content` and `response.text` attributes will not be available, and will raise errors if accessed. However you can also use the response streaming functionality to conditionally load the response body:
+
+```
+>>> async with httpx.stream("GET", "https://www.example.com") as r:
+...     if r.headers['Content-Length'] < TOO_LONG:
+...         await r.read()
+...         print(r.text)
+```
+
 ## Cookies
 
 Any cookies that are set on the response can be easily accessed:
diff --git a/httpx/__init__.py b/httpx/__init__.py
index febe2c7..686359a 100644
--- a/httpx/__init__.py
+++ b/httpx/__init__.py
@@ -1,5 +1,5 @@
 from .__version__ import __description__, __title__, __version__
-from .api import delete, get, head, options, patch, post, put, request
+from .api import delete, get, head, options, patch, post, put, request, stream
 from .auth import BasicAuth, DigestAuth
 from .client import Client
 from .concurrency.asyncio import AsyncioBackend
@@ -75,6 +75,7 @@ __all__ = [
     "patch",
     "put",
     "request",
+    "stream",
     "BasicAuth",
     "Client",
     "DigestAuth",
diff --git a/httpx/api.py b/httpx/api.py
index 893c3e7..d2f1d6d 100644
--- a/httpx/api.py
+++ b/httpx/api.py
@@ -1,12 +1,13 @@
 import typing
 
-from .client import Client
+from .client import Client, StreamContextManager
 from .config import DEFAULT_TIMEOUT_CONFIG, CertTypes, TimeoutTypes, VerifyTypes
 from .models import (
     AuthTypes,
     CookieTypes,
     HeaderTypes,
     QueryParamTypes,
+    Request,
     RequestData,
     RequestFiles,
     Response,
@@ -98,6 +99,44 @@ async def request(
         )
 
 
+def stream(
+    method: str,
+    url: URLTypes,
+    *,
+    params: QueryParamTypes = None,
+    data: RequestData = None,
+    files: RequestFiles = None,
+    json: typing.Any = None,
+    headers: HeaderTypes = None,
+    cookies: CookieTypes = None,
+    auth: AuthTypes = None,
+    timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG,
+    allow_redirects: bool = True,
+    verify: VerifyTypes = True,
+    cert: CertTypes = None,
+    trust_env: bool = True,
+) -> StreamContextManager:
+    client = Client(cert=cert, verify=verify, trust_env=trust_env)
+    request = Request(
+        method=method,
+        url=url,
+        params=params,
+        data=data,
+        files=files,
+        json=json,
+        headers=headers,
+        cookies=cookies,
+    )
+    return StreamContextManager(
+        client=client,
+        request=request,
+        auth=auth,
+        timeout=timeout,
+        allow_redirects=allow_redirects,
+        close_client=True,
+    )
+
+
 async def get(
     url: URLTypes,
     *,
diff --git a/httpx/client.py b/httpx/client.py
index 24b6a45..a4fd8a3 100644
--- a/httpx/client.py
+++ b/httpx/client.py
@@ -1,5 +1,6 @@
 import functools
 import typing
+import warnings
 from types import TracebackType
 
 import hstspreload
@@ -222,6 +223,12 @@ class Client:
         timeout: typing.Union[TimeoutTypes, UnsetType] = UNSET,
         trust_env: bool = None,
     ) -> Response:
+        if stream:
+            warnings.warn(
+                "The 'stream=True' argument is due to be deprecated. "
+                "Use 'async with client.stream(method, url, ...) as response' instead."
+            )
+
         request = self.build_request(
             method=method,
             url=url,
@@ -244,6 +251,39 @@ class Client:
         )
         return response
 
+    def stream(
+        self,
+        method: str,
+        url: URLTypes,
+        *,
+        data: RequestData = None,
+        files: RequestFiles = None,
+        json: typing.Any = None,
+        params: QueryParamTypes = None,
+        headers: HeaderTypes = None,
+        cookies: CookieTypes = None,
+        auth: AuthTypes = None,
+        allow_redirects: bool = True,
+        timeout: typing.Union[TimeoutTypes, UnsetType] = UNSET,
+    ) -> "StreamContextManager":
+        request = self.build_request(
+            method=method,
+            url=url,
+            data=data,
+            files=files,
+            json=json,
+            params=params,
+            headers=headers,
+            cookies=cookies,
+        )
+        return StreamContextManager(
+            client=self,
+            request=request,
+            auth=auth,
+            allow_redirects=allow_redirects,
+            timeout=timeout,
+        )
+
     def build_request(
         self,
         method: str,
@@ -864,3 +904,42 @@ def _proxies_to_dispatchers(
             else:
                 new_proxies[str(key)] = dispatcher_or_url
         return new_proxies
+
+
+class StreamContextManager:
+    def __init__(
+        self,
+        client: Client,
+        request: Request,
+        *,
+        auth: AuthTypes = None,
+        allow_redirects: bool = True,
+        timeout: typing.Union[TimeoutTypes, UnsetType] = UNSET,
+        close_client: bool = False,
+    ) -> None:
+        self.client = client
+        self.request = request
+        self.auth = auth
+        self.allow_redirects = allow_redirects
+        self.timeout = timeout
+        self.close_client = close_client
+
+    async def __aenter__(self) -> "Response":
+        self.response = await self.client.send(
+            request=self.request,
+            auth=self.auth,
+            allow_redirects=self.allow_redirects,
+            timeout=self.timeout,
+            stream=True,
+        )
+        return self.response
+
+    async def __aexit__(
+        self,
+        exc_type: typing.Type[BaseException] = None,
+        exc_value: BaseException = None,
+        traceback: TracebackType = None,
+    ) -> None:
+        await self.response.close()
+        if self.close_client:
+            await self.client.close()


Test patch:
diff --git a/tests/client/test_async_client.py b/tests/client/test_async_client.py
index 69563cb..cc433e5 100644
--- a/tests/client/test_async_client.py
+++ b/tests/client/test_async_client.py
@@ -62,16 +62,19 @@ async def test_post_json(server, backend):
 
 async def test_stream_response(server, backend):
     async with httpx.Client(backend=backend) as client:
-        response = await client.request("GET", server.url, stream=True)
+        async with client.stream("GET", server.url) as response:
+            body = await response.read()
+
     assert response.status_code == 200
-    body = await response.read()
     assert body == b"Hello, world!"
     assert response.content == b"Hello, world!"
 
 
 async def test_access_content_stream_response(server, backend):
     async with httpx.Client(backend=backend) as client:
-        response = await client.request("GET", server.url, stream=True)
+        async with client.stream("GET", server.url) as response:
+            pass
+
     assert response.status_code == 200
     with pytest.raises(httpx.ResponseNotRead):
         response.content
diff --git a/tests/client/test_client.py b/tests/client/test_client.py
index 668ae30..e2ded54 100644
--- a/tests/client/test_client.py
+++ b/tests/client/test_client.py
@@ -59,33 +59,36 @@ async def test_post_json(server):
 @pytest.mark.asyncio
 async def test_stream_response(server):
     async with httpx.Client() as client:
-        response = await client.get(server.url, stream=True)
+        async with client.stream("GET", server.url) as response:
+            content = await response.read()
     assert response.status_code == 200
-    content = await response.read()
     assert content == b"Hello, world!"
 
 
 @pytest.mark.asyncio
 async def test_stream_iterator(server):
+    body = b""
+
     async with httpx.Client() as client:
-        response = await client.get(server.url, stream=True)
+        async with client.stream("GET", server.url) as response:
+            async for chunk in response.stream_bytes():
+                body += chunk
+
     assert response.status_code == 200
-    body = b""
-    async for chunk in response.stream_bytes():
-        body += chunk
     assert body == b"Hello, world!"
 
 
 @pytest.mark.asyncio
 async def test_raw_iterator(server):
+    body = b""
+
     async with httpx.Client() as client:
-        response = await client.get(server.url, stream=True)
+        async with client.stream("GET", server.url) as response:
+            async for chunk in response.stream_raw():
+                body += chunk
+
     assert response.status_code == 200
-    body = b""
-    async for chunk in response.stream_raw():
-        body += chunk
     assert body == b"Hello, world!"
-    await response.close()
 
 
 @pytest.mark.asyncio
@@ -174,17 +177,18 @@ def test_merge_url():
 
 @pytest.mark.asyncio
 async def test_elapsed_delay(server):
+    url = server.url.copy_with(path="/slow_response/100")
     async with httpx.Client() as client:
-        response = await client.get(server.url.copy_with(path="/slow_response/100"))
+        response = await client.get(url)
     assert response.elapsed.total_seconds() == pytest.approx(0.1, rel=0.2)
 
 
 @pytest.mark.asyncio
 async def test_elapsed_delay_ignores_read_time(server):
+    url = server.url.copy_with(path="/slow_response/100")
     async with httpx.Client() as client:
-        response = await client.get(
-            server.url.copy_with(path="/slow_response/100"), stream=True
-        )
-    sleep(0.2)
-    await response.read()
+        async with client.stream("GET", url) as response:
+            sleep(0.2)
+            await response.read()
+
     assert response.elapsed.total_seconds() == pytest.approx(0.1, rel=0.2)
diff --git a/tests/test_api.py b/tests/test_api.py
index a3fb23d..9624842 100644
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -66,6 +66,17 @@ async def test_delete(server):
     assert response.reason_phrase == "OK"
 
 
+@pytest.mark.asyncio
+async def test_stream(server):
+    async with httpx.stream("GET", server.url) as response:
+        await response.read()
+
+    assert response.status_code == 200
+    assert response.reason_phrase == "OK"
+    assert response.text == "Hello, world!"
+    assert response.http_version == "HTTP/1.1"
+
+
 @pytest.mark.asyncio
 async def test_get_invalid_url(server):
     with pytest.raises(httpx.InvalidURL):
diff --git a/tests/test_timeouts.py b/tests/test_timeouts.py
index 7171092..e64c4b9 100644
--- a/tests/test_timeouts.py
+++ b/tests/test_timeouts.py
@@ -44,9 +44,6 @@ async def test_pool_timeout(server, backend):
     async with Client(
         pool_limits=pool_limits, timeout=timeout, backend=backend
     ) as client:
-        response = await client.get(server.url, stream=True)
-
-        with pytest.raises(PoolTimeout):
-            await client.get("http://localhost:8000/")
-
-        await response.read()
+        async with client.stream("GET", server.url):
+            with pytest.raises(PoolTimeout):
+                await client.get("http://localhost:8000/")