Repository: urllib3/urllib3

Issue:
Allow using HTTPHeaderDict and multiple header keys in requests
### Context

What are you trying to do?

I am trying to pass repeated headers such as the `Impersonate-Group` header used by the kubernetes API. (https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation)

How do you expect to do it?

I would like to be able to call the urllib3 request method and pass the parameter `headers` as an HTTPHeaderDict.

Is it something you currently cannot do?

It is nearly supported. It is broken for requests with request body because of the implementation of `request_encode_body`:

https://github.com/urllib3/urllib3/blob/main/src/urllib3/request.py#L174
```
        extra_kw: Dict[str, Any] = {"headers": {}}
```

https://github.com/urllib3/urllib3/blob/main/src/urllib3/request.py#L196
```
        extra_kw["headers"].update(headers)
```

These lines force headers to be a plain python dictionary even when an HTTPHeaderDict is passed. Hacking on the installed library I was able to get in fully functional by changing the logic to preserve the type.

Is this related to an existing issue/problem?

Not that I am aware of.

### Alternatives

Can you achieve the same result doing it in an alternative way?

Not that I know of

Is the alternative considerable?

### Duplicate

Has the feature been requested before?
If so, please provide a link to the issue.

Not that I know of.

### Contribution

Would you be willing to submit a PR?
_(Help can be provided if you need assistance submitting a PR)_

Yes

### Minimum requirements

:moneybag: **You can get paid to complete this issue! [Please read the docs for more information](https://urllib3.readthedocs.io/en/latest/contributing.html#getting-paid-for-your-contributions).**

- [x] For all methods (`GET`, `POST`, etc) and all body values (`None`, `b"..."`, `io.BytesIO(...)`) the behavior of headers is consistent.
- [x] Add a new parameter `combine` with a default value of `False` to `HTTPHeaderDict.add`
  - [ ] If the parameter is `False` then headers names and values are added as separate items in the request. (ie `A: 1\r\nA: 2\r\nA: 3`)
  - [ ] If parameter is `True` then the value should be combined with a comma+space  (ie `A: 1, 2, 3`).
- [x] `HTTPHeaderDict.itermerged` should iterate with all values merged.
- [x] `HTTPHeaderDict.items` should iterate with values as they were added


Discussion:
+1, I believe this used to be a feature we supported at one point! Specifically, you could pass in a list of tuples which `dict.update(...)` accepted.

If the only thing blocking this is the new type annotations, makes sense to fix that. Let's wait for Seth/Quentin to give guidance on the best way to move forward though. :)
So `HTTPHeaderDict` for requests is an interesting one, because it makes total sense to use but it really wasn't meant to be a public API in v1.x. Users are still able to import it from `urllib3.connection` and `urllib3.response` and use it without knowing it isn't a public API though. But now when you try to pass that `HTTPHeaderDict` as request headers in v1.x it... almost works? And that's an unfortunate thing for us cuz now our users are using something that isn't tested anywhere!

So starting in v2.0 we're making `HTTPHeaderDict` a first-class citizen and actually the recommended way of managing headers for both requests and responses.

Basically all of this is to say we don't support duplicate headers with any released version, sorry! My recommendation is to try using commas to combine the header values. You can also use an `HTTPHeaderDict` to do this for you like so:

```python
header_dict = HTTPHeaderDict()
header_dict.add("Header", "1")
header_dict.add("Header", "2")
headers = dict(header_dict.itermerged())
{"Header": "1, 2"}
```

(Btw the `main` branch that you linked to is for v2 development and hasn't been released yet, we have a `1.26.x` branch for `v1.x` releases)
(Are we sure this works correctly in the main branch? We didn't add tests when documenting it for requests)
@pquentin Doesn't work in the main branch, just signaled the intent but should probably also create an issue to track it too :)
Going to use this issue as it contains all the discussion, I'll rename and add to the v2 milestone.
Thank you for your attention on this issue.
@sethmlarson I tried using commas in my values, but unfortunately the kubernetes API (and many others it seems) do not follow RFC 7230 https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.2

"A sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list [i.e., #(values)] or the header field is a well-known exception (as noted below)."

So it seems to me the current behavior of urllib3 is technically correct, but unfortunately prevents communicating with some apis.
I've discussed the issue of allowing multiple headers with a few maintainers. There's no question that this is something we want to support, given the real-world use cases that need it.

We currently have two proposals, both handling overwrite, sending multiple headers, and folding. Suppose we want to send those headers:

```
Impersonate-User: jane.doe@example.com
Impersonate-Group: developers
Impersonate-Group: admins
Cache-Control: public, max-age=604800
```

It seems to me that neither of those proposals is breaking, so this change does not have to be in 2.0?

## First proposal: `add(fold: bool)`

```python3
headers = HTTPHeaderDict()

headers["Impersonate-User"] = "john.doe@example.com"
headers["Impersonate-User"] = "jane.doe@example.com"

headers.add("Cache-Control", "public")
headers.add("Cache-Control", "max-age=604800")

headers.add("Impersonate-Group", "developers", fold=False)
headers.add("Impersonate-Group", "admins", fold=False)
```

## Second proposal: `add(on_duplicate=overwrite|fold|sendmultiple)`

(Using strings in this example but it would probably be an enum instead)

```python3
headers = HTTPHeaderDict()

# we now have two ways to overwrite
headers["Impersonate-User"] = "john.doe@example.com"
headers.add("Impersonate-User",  "martin.doe@example.com", on_duplicate="overwrite")
headers["Impersonate-User"] = "jane.doe@example.com"

# on_duplicate=fold is the default behavior
headers.add("Cache-Control", "public")
headers.add("Cache-Control", "max-age=604800")

headers.add("Impersonate-Group", "developers", on_duplicate="sendmultiple")
headers.add("Impersonate-Group", "admins", on_duplicate="sendmultiple")
```
So I agree that these are both backwards compatible. I do think @sethmlarson had a good point about `__setitem__` already works to be `add(header_name, header_value, on_duplicate="overwrite")` but here's the thing, I think `__setitem__` could be confusing to have to switch to. I agree, one obvious way to do something is great, and I'm also a fan of "Oh shit, I didn't realize I was doing the wrong thing here, easier to add a parameter to do the right behaviour and iterate until I find the right output". 

Part of me wants a "fourth" option if we were to go with `on_duplicate` as well but I don't know that it matters for `HTTPHeaderDict` because the primary use for that case is as a server, not so much a client. I'll explain it anyway as food for thought.

Imagine that the default for `on_duplicate` is `auto`. That means for a sub-set of headers that we know are absolutely safe to send as multiple and should prefer not to be sent as a folded header (`Set-Cookie` is the most common one of these). For other headers `fold` is the right "automatic" behaviour.

If our header dict were used more widely, I'd be arguing for `auto` but I'm too sleep deprived to think of a client case that's similar that would require us to be smart for the user.
@haikuginger @shazow What are your thoughts on those APIs?
One suggestion I would consider is changing the term "fold" to "combine", since that's the term RFC 7230 uses and I think "fold" is not very beginner-programmer friendly (more of a functional programming term?).

If I was picking terms from scratch, my gut is to go with "replace", "combine", "separate".

If we wanted to go with UPSERT terminology, it's ignore/insert/update/merge. (Worth adding a 4th `ignore` option? What about a 5th `unique` option that throws?)

Other options I can think of that aren't great:
- Could have more `add`-like methods which have different names, such as `add_combined` or something like `append` or `insert`.
- Could have different HTTPHeaderDict implementations with different behaviours. (Is it easy to bring your own implementation rn or are we forcing people to use ours?)

In the spirit of making progress on this, my vote would be doing either of the current `add` proposals and renaming `fold` to `combine`. :)


Using `combine=False` instead of `fold=False` works for me too. This is the verb that RFC 7230 uses so is consistent there too. Great suggestion, @shazow!
Haha, ended up going down a rabbit hole on this one trying to find an interface that suits the more complicated cases that `Dict` and `List[Tuple[str, str]]` are insufficient for. If I were doing it from scratch (which we can think about for 2.0, because we're breaking a lot of stuff!), I might consider something like this:

```python3
class SingleValueHeader:
    value: str

    def try_add_value(self, value: str) -> bool:
        return False

    def serialized(self) -> str:
        return self.value

class MultiValueHeader:
    values: List[str]

    def try_add_value(self, value: str) -> bool:
        self.values.append(value)
        return True

    def serialized(self) -> str:
        return ','.join(self.values)


class HttpHeader(Protocol):
    
    def try_add_value(self, value: str) -> bool:
        """
        Returns True if the value was successfully merged into this header;
        returns False if this header cannot accept additional values
        """

    def serialize(self) -> str:
        """
        Return the string that this header should be serialized to in a request.
        """

class RequestHeaders:

    def all_by_name(self, name: str) -> List[HttpHeader]:
        """
        Return an iterable of all the `HttpHeader` entities going by the given
        name included in this collection of headers
        """

    def set_header(self, name: str, header: Union[str, HttpHeader]) -> None:
        """
        Ensures there is exactly one header with the provided name, and it has
        exactly the single serializable value provided (whether that value is
        a single string or a collection of strings that will be concatenated
        together when placing the request)
        """

    def serializable_kv_pairs(self) -> List[Tuple[str, str]]:
        """
        Return a list of string-string pairs that can be safely transmitted
        over the wire as part of an HTTP request
        """

    @overload  # When an HttpHeader object is passed, `mergeable` is disregarded
    def add_header(self, name: str, header: HttpHeader) -> None: ...
    @overload
    def add_header(self, name: str, header: str, mergeable: bool = False) -> None: ...
    @overload
    def add_header(
        self,
        name: str,
        header: Union[str, HttpHeader],
        mergeable: bool = False
    ) -> None:
        """
        In addition to any other headers that will be included in this request,
        send this header with the value specified; send it in addition to any
        headers that bear the same name but will be sent separately.

        If `mergeable` is True, the header will be created in such a way that
        `add_header_value` may add additional contents to it later on, but only
        if the value passed is a plain string; if it's `HttpHeader`, then trust
        it to handle its own business when `try_add_value` is called.
        """

    def add_header_value(self, name: str, value: str, mergeable: bool = False) -> None:
        """
        Ensure that the provided value is represented in the set of headers
        that will be serialized for this request.
        """

        if mergeable:
            # if this value can share a header with other values, get any existing
            # headers we might add this value to
            potential_existing_headers = self.all_by_name(name)

            # iterate over those headers...
            for header in potential_existing_headers:
                # ...trying to add the value to each in turn...
                if header.try_add_value(value):
                    # and finishing if we did it!
                    return

        # if we didn't find a header we could add this value to, or if it can't
        # be merged with others, create a new header that can accommodate it,
        # making sure that the new header follows this value's merge rules.
        return self.add_header(name, value, mergeable=mergeable)
```

I think the trick here is that for the case where we're adding values, we want to make sure that each value for a given name agrees with other already-added values about whether they can be merged together. That means that for the complex cases, we need to carry around a bit more than just strings.
I tried to the API described [in this comment](https://github.com/urllib3/urllib3/issues/2242#issuecomment-1023764816) in the PR #2652. 

The biggest messiness comes from backwards compatibility for the object. In particular, around the `items` method. I ended up with just one toggle to get this work, but we could go for zero if we are OK with the break on that specific method.

I would appreciate reviews on that PR to see if that makes sense.

Resolving patch:
diff --git a/src/urllib3/_collections.py b/src/urllib3/_collections.py
--- a/src/urllib3/_collections.py
+++ b/src/urllib3/_collections.py
@@ -174,14 +174,17 @@ class HTTPHeaderDictItemView(Set[Tuple[str, str]]):
     >>> d['X-Header-Name']
     'Value1, Value2, Value3'
 
-    However, if we iterate over an HTTPHeaderDict's items, we want to get a
-    distinct item for every different value of a header:
+    However, if we iterate over an HTTPHeaderDict's items, we will optionally combine
+    these values based on whether combine=True was called when building up the dictionary
 
+    >>> d = HTTPHeaderDict({"A": "1", "B": "foo"})
+    >>> d.add("A", "2", combine=True)
+    >>> d.add("B", "bar")
     >>> list(d.items())
     [
-        ('X-Header-Name', 'Value1')
-        ('X-Header-Name', 'Value2')
-        ('X-Header-Name', 'Value3')
+        ('A', '1, 2'),
+        ('B', 'foo'),
+        ('B', 'bar'),
     ]
 
     This class conforms to the interface required by the MutableMapping ABC while
@@ -273,6 +276,9 @@ def __contains__(self, key: object) -> bool:
             return key.lower() in self._container
         return False
 
+    def setdefault(self, key: str, default: str = "") -> str:
+        return super().setdefault(key, default)
+
     def __eq__(self, other: object) -> bool:
         maybe_constructable = ensure_can_construct_http_header_dict(other)
         if maybe_constructable is None:
@@ -301,14 +307,24 @@ def discard(self, key: str) -> None:
         except KeyError:
             pass
 
-    def add(self, key: str, val: str) -> None:
+    def add(self, key: str, val: str, *, combine: bool = False) -> None:
         """Adds a (name, value) pair, doesn't overwrite the value if it already
         exists.
 
+        If this is called with combine=True, instead of adding a new header value
+        as a distinct item during iteration, this will instead append the value to
+        any existing header value with a comma. If no existing header value exists
+        for the key, then the value will simply be added, ignoring the combine parameter.
+
         >>> headers = HTTPHeaderDict(foo='bar')
         >>> headers.add('Foo', 'baz')
         >>> headers['foo']
         'bar, baz'
+        >>> list(headers.items())
+        [('foo', 'bar'), ('foo', 'baz')]
+        >>> headers.add('foo', 'quz', combine=True)
+        >>> list(headers.items())
+        [('foo', 'bar, baz, quz')]
         """
         # avoid a bytes/str comparison by decoding before httplib
         if isinstance(key, bytes):
@@ -318,7 +334,13 @@ def add(self, key: str, val: str) -> None:
         # Keep the common case aka no item present as fast as possible
         vals = self._container.setdefault(key_lower, new_vals)
         if new_vals is not vals:
-            vals.append(val)
+            # if there are values here, then there is at least the initial
+            # key/value pair
+            assert len(vals) >= 2
+            if combine:
+                vals[-1] = vals[-1] + ", " + val
+            else:
+                vals.append(val)
 
     def extend(self, *args: ValidHTTPHeaderSource, **kwargs: str) -> None:
         """Generic import function for any type of header-like object.
diff --git a/src/urllib3/_request_methods.py b/src/urllib3/_request_methods.py
--- a/src/urllib3/_request_methods.py
+++ b/src/urllib3/_request_methods.py
@@ -2,6 +2,7 @@
 from typing import Any, Dict, Mapping, Optional, Sequence, Tuple, Union
 from urllib.parse import urlencode
 
+from ._collections import HTTPHeaderDict
 from .connection import _TYPE_BODY
 from .filepost import _TYPE_FIELDS, encode_multipart_formdata
 from .response import BaseHTTPResponse
@@ -188,7 +189,7 @@ def request_encode_body(
         if headers is None:
             headers = self.headers
 
-        extra_kw: Dict[str, Any] = {"headers": {}}
+        extra_kw: Dict[str, Any] = {"headers": HTTPHeaderDict(headers)}
         body: Union[bytes, str]
 
         if fields:
@@ -208,9 +209,8 @@ def request_encode_body(
                 )
 
             extra_kw["body"] = body
-            extra_kw["headers"] = {"Content-Type": content_type}
+            extra_kw["headers"].setdefault("Content-Type", content_type)
 
-        extra_kw["headers"].update(headers)
         extra_kw.update(urlopen_kw)
 
         return self.urlopen(method, url, **extra_kw)
diff --git a/src/urllib3/connection.py b/src/urllib3/connection.py
--- a/src/urllib3/connection.py
+++ b/src/urllib3/connection.py
@@ -73,7 +73,7 @@ class BaseSSLError(BaseException):  # type: ignore[no-redef]
 
 # When it comes time to update this value as a part of regular maintenance
 # (ie test_recent_date is failing) update it to ~6 months before the current date.
-RECENT_DATE = datetime.date(2020, 7, 1)
+RECENT_DATE = datetime.date(2022, 1, 1)
 
 _CONTAINS_CONTROL_CHAR_RE = re.compile(r"[^-!#$%&'*+.^_`|~0-9a-zA-Z]")
 


Test patch:
diff --git a/test/test_collections.py b/test/test_collections.py
--- a/test/test_collections.py
+++ b/test/test_collections.py
@@ -155,6 +155,14 @@ def test_create_from_kwargs(self) -> None:
         assert len(h) == 4
         assert "ab" in h
 
+    def test_setdefault(self) -> None:
+        h = HTTPHeaderDict(a="1")
+        assert h.setdefault("A", "3") == "1"
+        assert h.setdefault("b", "2") == "2"
+        assert h.setdefault("c") == ""
+        assert h["c"] == ""
+        assert h["b"] == "2"
+
     def test_create_from_dict(self) -> None:
         h = HTTPHeaderDict(dict(ab="1", cd="2", ef="3", gh="4"))
         assert len(h) == 4
@@ -252,6 +260,34 @@ def test_extend_from_container(self, d: HTTPHeaderDict) -> None:
         assert d["e"] == "foofoo"
         assert len(d) == 2
 
+    def test_header_repeat(self, d: HTTPHeaderDict) -> None:
+        d["other-header"] = "hello"
+        d.add("other-header", "world")
+
+        assert list(d.items()) == [
+            ("Cookie", "foo"),
+            ("Cookie", "bar"),
+            ("other-header", "hello"),
+            ("other-header", "world"),
+        ]
+
+        d.add("other-header", "!", combine=True)
+        expected_results = [
+            ("Cookie", "foo"),
+            ("Cookie", "bar"),
+            ("other-header", "hello"),
+            ("other-header", "world, !"),
+        ]
+
+        assert list(d.items()) == expected_results
+        # make sure the values persist over copys
+        assert list(d.copy().items()) == expected_results
+
+        other_dict = HTTPHeaderDict()
+        # we also need for extensions to properly maintain results
+        other_dict.extend(d)
+        assert list(other_dict.items()) == expected_results
+
     def test_extend_from_headerdict(self, d: HTTPHeaderDict) -> None:
         h = HTTPHeaderDict(Cookie="foo", e="foofoo")
         d.extend(h)
diff --git a/test/with_dummyserver/test_poolmanager.py b/test/with_dummyserver/test_poolmanager.py
--- a/test/with_dummyserver/test_poolmanager.py
+++ b/test/with_dummyserver/test_poolmanager.py
@@ -380,6 +380,46 @@ def test_headers_http_header_dict(self) -> None:
                 ["Extra", "extra"],
             ]
 
+    def test_headers_http_multi_header_multipart(self) -> None:
+        headers = HTTPHeaderDict()
+        headers.add("Multi", "1")
+        headers.add("Multi", "2")
+        old_headers = headers.copy()
+
+        with PoolManager(headers=headers) as http:
+            r = http.request(
+                "POST",
+                f"{self.base_url}/multi_headers",
+                fields={"k": "v"},
+                multipart_boundary="b",
+                encode_multipart=True,
+            )
+            returned_headers = r.json()["headers"]
+            assert returned_headers[4:] == [
+                ["Multi", "1"],
+                ["Multi", "2"],
+                ["Content-Type", "multipart/form-data; boundary=b"],
+            ]
+            # Assert that the previous headers weren't modified.
+            assert headers == old_headers
+
+            # Set a default value for the Content-Type
+            headers["Content-Type"] = "multipart/form-data; boundary=b; field=value"
+            r = http.request(
+                "POST",
+                f"{self.base_url}/multi_headers",
+                fields={"k": "v"},
+                multipart_boundary="b",
+                encode_multipart=True,
+            )
+            returned_headers = r.json()["headers"]
+            assert returned_headers[4:] == [
+                ["Multi", "1"],
+                ["Multi", "2"],
+                # Uses the set value, not the one that would be generated.
+                ["Content-Type", "multipart/form-data; boundary=b; field=value"],
+            ]
+
     def test_body(self) -> None:
         with PoolManager() as http:
             r = http.request("POST", f"{self.base_url}/echo", body=b"test")
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -1643,6 +1643,75 @@ def socket_handler(listener: socket.socket) -> None:
             ]
             assert expected_response_headers == actual_response_headers
 
+    @pytest.mark.parametrize(
+        "method_type, body_type",
+        [
+            ("GET", None),
+            ("POST", None),
+            ("POST", "bytes"),
+            ("POST", "bytes-io"),
+        ],
+    )
+    def test_headers_sent_with_add(
+        self, method_type: str, body_type: Optional[str]
+    ) -> None:
+        """
+        Confirm that when adding headers with combine=True that we simply append to the
+        most recent value, rather than create a new header line.
+        """
+        body: Union[None, bytes, io.BytesIO]
+        if body_type is None:
+            body = None
+        elif body_type == "bytes":
+            body = b"my-body"
+        elif body_type == "bytes-io":
+            body = io.BytesIO(b"bytes-io-body")
+            body.seek(0, 0)
+        else:
+            raise ValueError("Unknonw body type")
+
+        buffer: bytes = b""
+
+        def socket_handler(listener: socket.socket) -> None:
+            nonlocal buffer
+            sock = listener.accept()[0]
+            sock.settimeout(0)
+
+            start = time.time()
+            while time.time() - start < (LONG_TIMEOUT / 2):
+                try:
+                    buffer += sock.recv(65536)
+                except OSError:
+                    continue
+
+            sock.sendall(
+                b"HTTP/1.1 200 OK\r\n"
+                b"Server: example.com\r\n"
+                b"Content-Length: 0\r\n\r\n"
+            )
+            sock.close()
+
+        self._start_server(socket_handler)
+
+        headers = HTTPHeaderDict()
+        headers.add("A", "1")
+        headers.add("C", "3")
+        headers.add("B", "2")
+        headers.add("B", "3")
+        headers.add("A", "4", combine=False)
+        headers.add("C", "5", combine=True)
+        headers.add("C", "6")
+
+        with HTTPConnectionPool(self.host, self.port, retries=False) as pool:
+            r = pool.request(
+                method_type,
+                "/",
+                body=body,
+                headers=headers,
+            )
+            assert r.status == 200
+            assert b"A: 1\r\nA: 4\r\nC: 3, 5\r\nC: 6\r\nB: 2\r\nB: 3" in buffer
+
 
 class TestBrokenHeaders(SocketDummyServerTestCase):
     def _test_broken_header_parsing(