Repository: encode/httpx

Issue:
User-provided Content-Length header is missing from streaming requests
### Checklist

<!-- Please make sure you check all these items before submitting your bug report. -->

- [x] The bug is reproducible against the latest release and/or `master`.
- [x] There are no similar issues or pull requests to fix it yet.

### Describe the bug

<!-- A clear and concise description of what the bug is. -->

User-provided Content-Length header is missing from streaming requests

### To reproduce

<!-- Provide a *minimal* example with steps to reproduce the bug locally.

NOTE: try to keep any external dependencies *at an absolute minimum* (servers, proxies, certificates...).
In other words, remove anything that doesn't make the bug go away.

If you need a local server to replicate against, you can build one using eg. Starlette [0] and Uvicorn [1], or any tool you feel comfortable with. Check out other issues for examples and remember to share setup instructions here. :-)

[0]: https://www.starlette.io
[1]: https://www.uvicorn.org
-->

```python
>>> import httpx
>>> client = httpx.AsyncClient()
>>> async def f():
...     yield b'abcd'
... 
>>> r = await client.post('https://httpbin.org/post', data=f(), headers={'content-length': '4'})
>>> r.json()
{'args': {}, 'data': 'abcd', 'files': {}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'Transfer-Encoding': 'chunked', 'User-Agent': 'python-httpx/0.14.1', 'X-Amzn-Trace-Id': 'Root=1-5f3283ee-d5e67484ee0db0346804ce72'}, 'json': None, 'origin': 'x.x.x.x', 'url': 'https://httpbin.org/post'}
>>> 'Content-Length' in r.json()['headers']
False
```

### Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

User-provided Content-Length header is not missing from streaming requests

### Actual behavior

<!-- A clear and concise description of what actually happens. -->

User-provided Content-Length header is missing from streaming requests

### Debugging material

<!-- Any tracebacks, screenshots, etc. that can help understanding the problem.

NOTE:
- Please list tracebacks in full (don't truncate them).
- If relevant, consider turning on DEBUG or TRACE logs for additional details (see https://www.python-httpx.org/environment_variables/#httpx_log_level).
- Consider using `<details>` to make tracebacks/logs collapsible if they're very large (see https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d).
-->

### Environment

- OS: Linux/Windows/macOS.
- Python version: 3.8.5
- HTTPX version: 0.14.1
- Async environment: asyncio
- HTTP proxy: no
- Custom certificates: no

### Additional context

<!-- Any additional information that can help understanding the problem.

Eg. linked issues, or a description of what you were trying to achieve. -->



Discussion:
Thanks for the report.

You can see what's happening a bit more clearly with this...

```python
>>> c = httpx.Client()
>>> c.build_request("POST", "https://www.example.org", headers={'Content-Length': '5'}, data=(i for i in b'abcde')).headers
Headers({'host': 'www.example.org', 'user-agent': 'python-httpx/0.14.0', 'accept': '*/*', 'accept-encoding': 'gzip, deflate, br', 'connection': 'keep-alive', 'content-length': '5', 'transfer-encoding': 'chunked'})
```

Our `.build_request` is including both the `Transfer-Encoding: chunked` that it expects from streaming data... https://github.com/encode/httpx/blob/db4e417a137bf3bb04c1a3ffc492bf80d222959c/httpx/_content_streams.py#L86

While also including the `content-length` that you've set explicitly.

Presumably the `h11` library is then discounting the `Content-Length`. (Since it's invalid to set both `Content-Length` and `Transfer-Encoding`.)

I think what we probably want is an explicit ignore case for setting `Transfer-Encoding` from the stream, if the `Content-Length` has been set explicitly. Probably something like changing this piece of code: https://github.com/encode/httpx/blob/db4e417a137bf3bb04c1a3ffc492bf80d222959c/httpx/_models.py#L615

From...

```python
for key, value in self.stream.get_headers().items():
    self.headers.setdefault(key, value)
```

To this...

```python
for key, value in self.stream.get_headers().items():
    if key.lower() == "transfer-encoding" and "content-length" in self.headers:
        continue
    self.headers.setdefault(key, value)
```

Along with a test case to verify that a `.build_request()` operation like this... `c.build_request("POST", "https://www.example.org", headers={'Content-Length': '5'}, data=(i for i in b'abcde'))` results in the correct headers. (Ie. *just* the Content-Length one.)
Labelling this as a good first issue, since there's hopefully enough into in the comment for someone to be able to resolve this.

Run your scripts and running Wireshark, `httpx` actually contains the `content-length` header:
![image](https://user-images.githubusercontent.com/10510431/89895807-fec1fb80-dc0e-11ea-98a8-feaee7a8c38c.png)

@j178 Okay. It's possible that an intermediary could be stripping it before it gets to the server.
Either way around the issue is valid, since we shouldn't be setting both headers.
Yes, I'll open a PR for this.
To clarify, I noticed this while trying to make requests to an API that checks the presence of the Content-Length header (B2 API, to be precise), so it's definitely not limited to httpbin. For example, if `data` is a `bytes` object, httpbin registers the header correctly.

```python
>>> r = await client.post('https://httpbin.org/post', data=b''.join([x async for x in f()]), headers={'content-length': '4'})
>>> 'Content-Length' in r.json()['headers']
True

Resolving patch:
diff --git a/httpx/_models.py b/httpx/_models.py
--- a/httpx/_models.py
+++ b/httpx/_models.py
@@ -613,6 +613,9 @@ def __init__(
 
     def prepare(self) -> None:
         for key, value in self.stream.get_headers().items():
+            # Ignore Transfer-Encoding if the Content-Length has been set explicitly.
+            if key.lower() == "transfer-encoding" and "content-length" in self.headers:
+                continue
             self.headers.setdefault(key, value)
 
         auto_headers: typing.List[typing.Tuple[bytes, bytes]] = []


Test patch:
diff --git a/tests/models/test_requests.py b/tests/models/test_requests.py
--- a/tests/models/test_requests.py
+++ b/tests/models/test_requests.py
@@ -73,6 +73,23 @@ async def streaming_body(data):
     assert request.headers["Transfer-Encoding"] == "chunked"
 
 
+def test_ignore_transfer_encoding_header_if_content_length_exists():
+    """
+    `Transfer-Encoding` should be ignored if `Content-Length` has been set explicitly.
+    See https://github.com/encode/httpx/issues/1168
+    """
+
+    def streaming_body(data):
+        yield data  # pragma: nocover
+
+    data = streaming_body(b"abcd")
+
+    headers = {"Content-Length": "4"}
+    request = httpx.Request("POST", "http://example.org", data=data, headers=headers)
+    assert "Transfer-Encoding" not in request.headers
+    assert request.headers["Content-Length"] == "4"
+
+
 def test_override_host_header():
     headers = {"host": "1.2.3.4:80"}