Repository: urllib3/urllib3

Issue:
urllib3 2.2 explicitly casts all headers to HTTPHeaderDict
### Subject

Originally reported in https://github.com/boto/botocore/issues/3111, we noticed users started experiencing TypeErrors being raised from urllib3 around the time 2.2.0 was released. After a deeper dive, it appears a change was made in https://github.com/urllib3/urllib3/commit/4ece59b2d438f605dcb9a2448e1155075cd2fe07 that started indiscriminately casting headers passed to `urlopen` as an `HTTPHeaderDict`. This has specific implications for header values that are defined as bytes, while this doesn't match the current typing information, urllib3 (and Requests) both supported this prior to the release of urllib3 2.0.

Talking to @pquentin, it sounds like this is still an intended use case, but there isn't a formal contract at this time. Below I've included a minimal example to demonstrate the error. We would like to get confirmation if this is an undocumented breakage in 2.x or if this is unintended behavior. From the perspective of `HTTPHeaderDict` in isolation, it seems odd to allow instantiation when it fails immediately on use.

I'm happy to look at supplying a patch once we determine the intended behavior.

### Environment

Reproducible on all platforms with urllib3 2.2.0.

### Steps to Reproduce

```python
from urllib3._collections import HTTPHeaderDict

hdict = HTTPHeaderDict({'x-my-header': b'my-binary-value'})
# Any access that invokes `.get()` will fail (print, get, getall, etc.)
print(hdict)
```

### Expected Behavior

`HTTPHeaderDict` either fails at initialization or supports bytes values. 

### Actual Behavior

```bash
>>> print(hdict)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/.venv/lib/python3.11/site-packages/urllib3/_collections.py", line 423, in __repr__
    return f"{type(self).__name__}({dict(self.itermerged())})"
                                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv/lib/python3.11/site-packages/urllib3/_collections.py", line 446, in itermerged
    yield val[0], ", ".join(val[1:])
                  ^^^^^^^^^^^^^^^^^^
```                  



Discussion:
Adding some more context after further digging. The reason this isn't more widely impacting is this will only affect code subclassing HTTPConnection in urllib3. In urllib3<2.2, headers were always left in their provided representation and a copy was created.

botocore has a ~10 year old usage of [`HTTPConnection`](https://github.com/urllib3/urllib3/blob/e22f651079ae65d06efbb28222c27000256ce7a5/src/urllib3/connection.py#L83) where `request` is overwritten to handle HTTP 100-continue responses. This was the earlier sticking point for upgrading botocore to urllib3 2.0.

With the recent change in https://github.com/urllib3/urllib3/commit/4ece59b2d438f605dcb9a2448e1155075cd2fe07, the type being passed to `request` is now different, leading to code accessing headers failing with the above error. This is avoided in the common case because urllib3's `request` implementation uses [`items()` when iterating headers](https://github.com/urllib3/urllib3/blob/2.0.x/src/urllib3/connection.py#L393-L394) and [forwards any bytes values to httplib's `putheader` for handling](https://github.com/urllib3/urllib3/blob/2.0.x/src/urllib3/connection.py#L307-L308). `items()` appears to be one of the only interfaces on `HTTPHeaderDict` that behaves when bytes are provided.

While HTTPConnection isn't explicitly part of the public interface in 2.x, it appears to have been when botocore's code was written. Shazow created the separate connection.py module for this use case in #254 at the request of a similar use case in #253.

I don't know what the current maintenance team wants to do in this case or what the current expectations are around this behavior. I think the options I'm seeing currently are:

1. Revert the type change back to `.copy()` to preserve the previous functionality.
2. Correct the behavior in `HTTPHeaderDict` when bytes values are provided. I think it can be argued the current inconsistencies aren't expected and might be fixed/improved regardless.
3. Attempt to update the workflow in botocore to remove the usage of bytes. It's not immediately clear what the scope of impact would be there since there are other widely used libraries that have some reliance on the current behavior.
There is already an existing ticket + PR to accept bytes values in the HTTPHeaderDict:

https://github.com/urllib3/urllib3/issues/3072
https://github.com/urllib3/urllib3/pull/3279

Would be interesting to see if that PR on its own would fix this issue.
Thanks for pointing those out, @bblommers! Yeah, I agree those seem to be inline with this issue. Anecdotally from Requests, I'll say guessing with the encoding is usually the wrong decision (especially around latin-1). If we get bytes, it should probably keep going over the wire as it was received. Otherwise you end up with data corruption between latin-1 and utf-8.

It looks like Quentin already covered some of that in #3279. I agree that having mixed values for the same key would be an exception case (although perhaps it could be explicitly handled rather than failing on the join):
> So my current thinking is that we should indeed join byte values using b", " and forbid mixed encoding and bytes (to be able to join). And not try to decode bytes to latin-1.

**Edit:** I can confirm #3279 does appear to resolve this issue when rebased onto main.
Thanks for the investigation!

We need to review and merge #3279, but we should also consider if we really want to cast headers to HTTPHeaderDict.

Resolving patch:
diff --git a/src/urllib3/connectionpool.py b/src/urllib3/connectionpool.py
--- a/src/urllib3/connectionpool.py
+++ b/src/urllib3/connectionpool.py
@@ -751,8 +751,8 @@ def urlopen(  # type: ignore[override]
         # have to copy the headers dict so we can safely change it without those
         # changes being reflected in anyone else's copy.
         if not http_tunnel_required:
-            headers = HTTPHeaderDict(headers)
-            headers.update(self.proxy_headers)
+            headers = headers.copy()  # type: ignore[attr-defined]
+            headers.update(self.proxy_headers)  # type: ignore[union-attr]
 
         # Must keep the exception bound to a separate variable or else Python 3
         # complains about UnboundLocalError.


Test patch: