Repository: mitmproxy/mitmproxy

Issue:
Cannot access raw cookie header / incorrect folding
#### Problem Description

I just noticed that on my end the `cookie` header contains multiple comma separated cookies, which is not valid. The original request by Chromium had them all in a single header separated by semicolons. I assume mitmproxy splits the cookies into multiple headers and incorrectly puts them back together (by folding via comma) even though I explicitly want the rawest of raw headers via `h.fields`.

How do I get the actual raw header key / values without any manipulation by mitmproxy? The docs only mention issues with `set-cookie` and amusingly the library I'm using does [actually support folding](https://github.com/nfriedly/set-cookie-parser/blob/2628d67bd0718c38bd1c028ee0fce6cda56e55be/test/split-cookies-string.js) but [not for `cookie`](https://github.com/jshttp/cookie/issues/62).

For cookie we have https://datatracker.ietf.org/doc/html/rfc6265#section-5.4

> When the user agent generates an HTTP request, the user agent MUST
   NOT attach more than one Cookie header field.

and https://datatracker.ietf.org/doc/html/rfc7540#section-8.1.2.5

> The Cookie header field [COOKIE] uses a semi-colon (";") to delimit
   cookie-pairs (or "crumbs").  This header field doesn't follow the
   list construction rules in HTTP (see [RFC7230], Section 3.2.2), which
   prevents cookie-pairs from being separated into different name-value
   pairs.  This can significantly reduce compression efficiency as
   individual cookie-pairs are updated.

>   To allow for better compression efficiency, the Cookie header field
   MAY be split into separate header fields, each with one or more
   cookie-pairs.  If there are multiple Cookie header fields after
   decompression, these MUST be concatenated into a single octet string
   using the two-octet delimiter of 0x3B, 0x20 (the ASCII string "; ")
   before being passed into a non-HTTP/2 context, such as an HTTP/1.1
   connection, or a generic HTTP server application.

 >  Therefore, the following two lists of Cookie header fields are
   semantically equivalent.

>     cookie: a=b; c=d; e=f

>     cookie: a=b
>     cookie: c=d
>     cookie: e=f


#### Steps to reproduce the behavior:

I was using github.com in a private window:

```py
def request(flow):
    if flow.request.url == 'https://github.com/':
        print('=======================')
        print(flow.request.headers)
        print('------------------')
        print(flow.request.headers.items())
        print('------------------')
        print(flow.request.headers.fields)
        print('------------------')
        for name, value in flow.request.headers.items():
            if name == 'cookie':
                print(name, value)
        print('------------------')

        print('=======================')
```

```sh
mitmdump --quiet --scripts cookie.py
```

```
=======================
Headers[(b'cache-control', b'max-age=0'), (b'sec-ch-ua', b'"Chromium";v="91", " Not;A Brand";v="99"'), (b'sec-ch-ua-mobile', b'?0'), (b'upgrade-insecure-requests', b'1'), (b'user-agent', b'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'), (b'sec-fetch-site', b'none'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-user', b'?1'), (b'sec-fetch-dest', b'document'), (b'accept-encoding', b'gzip, deflate, br'), (b'accept-language', b'en-US,en;q=0.9'), (b'cookie', b'_octo=GH1.1.165320981.1622543172'), (b'cookie', b'logged_in=no'), (b'cookie', b'tz=Europe%2FBerlin'), (b'cookie', b'_gh_sess=*****'), (b'if-none-match', b'W/"e1519975261cdc6739507b7f0666ade0"')]
------------------
ItemsView(Headers[(b'cache-control', b'max-age=0'), (b'sec-ch-ua', b'"Chromium";v="91", " Not;A Brand";v="99"'), (b'sec-ch-ua-mobile', b'?0'), (b'upgrade-insecure-requests', b'1'), (b'user-agent', b'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'), (b'sec-fetch-site', b'none'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-user', b'?1'), (b'sec-fetch-dest', b'document'), (b'accept-encoding', b'gzip, deflate, br'), (b'accept-language', b'en-US,en;q=0.9'), (b'cookie', b'_octo=GH1.1.165320981.1622543172'), (b'cookie', b'logged_in=no'), (b'cookie', b'tz=Europe%2FBerlin'), (b'cookie', b'_gh_sess=*****'), (b'if-none-match', b'W/"e1519975261cdc6739507b7f0666ade0"')])
------------------
((b'cache-control', b'max-age=0'), (b'sec-ch-ua', b'"Chromium";v="91", " Not;A Brand";v="99"'), (b'sec-ch-ua-mobile', b'?0'), (b'upgrade-insecure-requests', b'1'), (b'user-agent', b'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36'), (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'), (b'sec-fetch-site', b'none'), (b'sec-fetch-mode', b'navigate'), (b'sec-fetch-user', b'?1'), (b'sec-fetch-dest', b'document'), (b'accept-encoding', b'gzip, deflate, br'), (b'accept-language', b'en-US,en;q=0.9'), (b'cookie', b'_octo=GH1.1.165320981.1622543172'), (b'cookie', b'logged_in=no'), (b'cookie', b'tz=Europe%2FBerlin'), (b'cookie', b'_gh_sess=*****'), (b'if-none-match', b'W/"e1519975261cdc6739507b7f0666ade0"'))
------------------
cookie _octo=GH1.1.165320981.1622543172, logged_in=no, tz=Europe%2FBerlin, _gh_sess=*****
------------------
=======================
```

You can see that even `fields` has them split. And that iterating `items()` will incorrectly fold the header with commas.

Chromium network tab (one header, multiple cookies with semicolon):

```
cookie: _octo=GH1.1.165320981.1622543172; logged_in=no; tz=Europe%2FBerlin; _gh_sess=*****
```

#### System Information

```
Mitmproxy: 7.0.0.dev (+451, commit 27883e7)
Python:    3.8.5
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Linux-5.8.0-53-generic-x86_64-with-glibc2.29
```


Discussion:
I'm using `connection_strategy=lazy` on my end so this shouldn't be one of those HTTP/1.1 vs HTTP/2.0 transparent interop issues? It might just be that the dev tools are lying :thinking: 

Ok, so disabling HTTP2 in mitmproxy behaves as I was expecting it and the dev tools are in fact lying. With HTTP/2.0 I get multiple cookie headers from Chromium.

So this is probably just a matter of documenting the issues with cookie similar to set-cookie? I have to sleep a night on that :sweat_smile: 
`Message.headers.fields` should always have the headers as received on-the-wire.
Do https://docs.mitmproxy.org/dev/api/mitmproxy/http.html#Headers and https://docs.mitmproxy.org/dev/api/mitmproxy/http.html#Headers.get_all explain the rest? 😃 

Resolving patch:
diff --git a/mitmproxy/http.py b/mitmproxy/http.py
--- a/mitmproxy/http.py
+++ b/mitmproxy/http.py
@@ -86,7 +86,7 @@ class Headers(multidict.MultiDict):  # type: ignore
     >>> h.fields
 
     Caveats:
-     - For use with the "Set-Cookie" header, either use `Response.cookies` or see `Headers.get_all`.
+     - For use with the "Set-Cookie" and "Cookie" headers, either use `Response.cookies` or see `Headers.get_all`.
     """
 
     def __init__(self, fields: Iterable[Tuple[bytes, bytes]] = (), **headers):
@@ -142,9 +142,12 @@ def __iter__(self) -> Iterator[str]:
     def get_all(self, name: Union[str, bytes]) -> List[str]:
         """
         Like `Headers.get`, but does not fold multiple headers into a single one.
-        This is useful for Set-Cookie headers, which do not support folding.
+        This is useful for Set-Cookie and Cookie headers, which do not support folding.
 
-        *See also:* <https://tools.ietf.org/html/rfc7230#section-3.2.2>
+        *See also:*
+         - <https://tools.ietf.org/html/rfc7230#section-3.2.2>
+         - <https://datatracker.ietf.org/doc/html/rfc6265#section-5.4>
+         - <https://datatracker.ietf.org/doc/html/rfc7540#section-8.1.2.5>
         """
         name = _always_bytes(name)
         return [


Test patch: