Repository: encode/httpx

Issue:
Response cookies are not ingested properly in 0.14.0
Hi. After last updatet when loading cookies from a file, it gives an error

**http/cookiejar.py", line 1947, in _really_load
domain_specified = domain.startswith(".")
AttributeError: 'NoneType' object has no attribute 'startswith'**

```python
import os
from http.cookiejar import LWPCookieJar
import httpx

h = httpx.Client(cookies=LWPCookieJar('test'))
if os.path.isfile('test'):
    h.cookies.jar.load()
h.get('https://www.google.ru')
h.cookies.jar.save()
```

**Cookies file - httpx 0.13.3**
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-08-21"; path="/"; domain=".google.ru"; path_spec; domain_dot; secure; expires="2020-09-07 21:47:19Z"; version=0
Set-Cookie3: NID="204=CQ898UjmES*****************************ctxJAE99ngLlZcojaaWLrqex-td7X00PooPgmVnW61VrtwCHIPSb_CPR8Ic6dpD_gRY_pBteV_4YSGU7Vu58V7uHPwNcPxZ6PMCAHt-RZSPgnxcoM"; path="/"; domain=".google.ru"; path_spec; domain_dot; expires="2021-02-07 21:47:19Z"; HttpOnly=None; version=0

**Cookies file - httpx 0.14.0**
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-08-21"; path="/"; domain=".google.ru"; path_spec; domain_dot; expires="2020-09-07 21:50:09Z"; HttpOnly=None; Secure, NID="204=NCk0**********************xnr8aaHwCrC-zMdQQ9bAS5QAp6rfv-yOiuwkYDhiN7kTyDASjlZVIJ0Zase-JQMH_YwCvaiS9I232kIRu36O684kwHD_nRXHeUVpTJTlS7mbniFmebf5nhE"; version=0




Discussion:
Heya, first things...

1. Could you provide the full traceback?
2. Could you double check and confirm the exact behaviour on 0.13?
3. I don't recall us making any changes in our cookie handling? Both versions are tagged on GitHub so you could review if there are any changes?
**1. Traceback, script runned on version 0.14.0**
_(manual demand for payment) djigit0s@R:/mnt/d/prj/manual demand for payment$ python test.py
(manual demand for payment) djigit0s@R:/mnt/d/prj/manual demand for payment$ python test.py_
/usr/lib/python3.7/http/cookiejar.py:1966: UserWarning: http.cookiejar bug!
Traceback (most recent call last):
  File "/usr/lib/python3.7/http/cookiejar.py", line 1947, in _really_load
    domain_specified = domain.startswith(".")
AttributeError: 'NoneType' object has no attribute 'startswith'

  _warn_unhandled_exception()
Traceback (most recent call last):
  File "/usr/lib/python3.7/http/cookiejar.py", line 1947, in _really_load
    domain_specified = domain.startswith(".")
AttributeError: 'NoneType' object has no attribute 'startswith'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    h.cookies.jar.load()
  File "/usr/lib/python3.7/http/cookiejar.py", line 1794, in load
    self._really_load(f, filename, ignore_discard, ignore_expires)
  File "/usr/lib/python3.7/http/cookiejar.py", line 1968, in _really_load
    (filename, line))
http.cookiejar.LoadError: invalid Set-Cookie3 format file 'test': '1P_JAR="2020-08-09-09"; path="/"; domain=".google.ru"; path_spec; domain_dot; expires="2020-09-08 09:44:05Z"; HttpOnly=None; Secure, NID="204=------------------------------yl_8tbSzcfb2Y6lmG7j8kLKLigynEVHW8PqDowWASEG_tCeKBBwx_qpTfUIt6klL4UoSaX6uVK32R9mIcXDK5kN2DRIgu07yYVJB3-CSq5SEoA"; version=0'
_(manual demand for payment) djigit0s@R:/mnt/d/prj/manual demand for payment$_

**2. Checked on version 0.13.0 and 0.13.3, script work.** 

pay attention in the first message I indicated the results of saving cookies they are different, as if deleted some cookies
Yeah, something's weird.

I ran your script against `0.14.0` (`master`), and `0.13.0`, printing the contents `dict(h.cookies)` after the first run of the script.

Looks like in 0.14.0 we're conflating the two cookies into one upon receiving the response...

`0.14.0`:

```console
$ python debug/cookies.py
{'1P_JAR': '2020-08-09-18'}
```

```log
# test-0-14.log
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-09-18"; path="/"; domain=".google.com"; path_spec; domain_dot; expires="2020-09-08 18:18:25Z"; HttpOnly=None; Secure, NID="204=jte-Aw2Xlnx61T6Jtp7pDe8FnyHBrbm4SOr3zAw0ew_P-Terd-JUiALnKz4dJZKaLrb3lg-qbzFdTUedj6tDARlSymHiE2rOXGghxG7P7Ca0FlgbMa6BEJdfBb1DUUvI5FTnchaPytDbLoDjsLJJckAld0fix9c8GAGCTeM8OrQ"; version=0
```

`0.13.0`:

```console
$ python debug/cookies.py
{'1P_JAR': '2020-08-09-18', 'NID': '204=za_rWjCBE8fM88zGZ_XZSVy6nUG99DpCv5BUxwGPBmiIQdXwEJMOjy7kteacGRw9doy-PIf1wxJXcn4tFHy_tZBtvKUbyJA1ewCDmbalWFUagYHJuzGD9SLxSw6sHZKggf33xy57-sQtBR3jQUWkTrhP4L4jQ0NrO1QHQDgGB54'}
```

```log
# test-0-13.log
#LWP-Cookies-2.0
Set-Cookie3: 1P_JAR="2020-08-09-18"; path="/"; domain=".google.com"; path_spec; domain_dot; secure; expires="2020-09-08 18:19:10Z"; version=0
Set-Cookie3: NID="204=fd_FJ8M55DBEyI76sXNFuLw8xWoBOBnfTBRxvy5Nz_hUGjWtA-XSWOI5MIT5JSIY78g2K_yvYokJuS8ZgoISf8oxX2vzzwSkSXeE-jM8FXH-UCTjs87T3i3pGdX8KVBbPjNk5hUUBuWRiB5ocxwqtBZaSupp0Tsc32usj9qc6NA"; path="/"; domain=".google.com"; path_spec; domain_dot; expires="2021-02-08 18:19:10Z"; HttpOnly=None; version=0
```
Doesn't seem to come from `httpcore` (tested against 0.9.1 with a bit of patching for the API changes…), and the git blame on `Cookies` shows no code changes, so… 🤷‍♂️ 
Ah-ha! Found it :-) When extracting cookies we're calling into `response.headers.items()`, which _used_ to return all headers, but now returns de-duplicated headers (via #1089).

Yet there seems to be a problem... Here's what `response.headers.items()` contains:

```python
('set-cookie', '1P_JAR=2020-08-09-18; expires=Tue, 08-Sep-2020 18:33:35 GMT; path=/; domain=.google.com; Secure, NID=204=KWdXOuypc86YvRfBSiWoW1dEXfSl_5qI7sxZY4umlk4J35yNTeNEkw15MRaujK6uYCwkrtjihTTXZPp285z_xDOUzrdHt4dj0Z5C0VOpbvdLwRdHatHAzQs77TsaiWY78a3qU9r7KP_RbSLvLl2hlhnWFR2Hp5nWKPsAcOhQgSg; expires=Mon, 08-Feb-2021 18:33:35 GMT; path=/; domain=.google.com; HttpOnly')
```

Notice how this is…?

* "`1P_JAR=2020-08-09-18; expires=Tue, 08-Sep-2020 18:33:35 GMT; path=/; domain=.google.com; Secure`"
* Plus "`, `"
* Plus "`NID=204=KWdXOuypc86YvRfBSiWoW1dEXfSl_5qI7sxZY4umlk4J35yNTeNEkw15MRaujK6uYCwkrtjihTTXZPp285z_xDOUzrdHt4dj0Z5C0VOpbvdLwRdHatHAzQs77TsaiWY78a3qU9r7KP_RbSLvLl2hlhnWFR2Hp5nWKPsAcOhQgSg; expires=Mon, 08-Feb-2021 18:33:35 GMT; path=/; domain=.google.com; HttpOnly`"

The values from the two `Set-Cookie` are indeed conflated into a single `Set-Cookie: <value_1>, <value_2>`, as per this line…

https://github.com/encode/httpx/blob/0e73be83a8e133521c8c0cd7426a33cda196eef0/httpx/_models.py#L367

This is not really the problem that's causing this bug though, but rather that we're still calling `.multi_items()` when processing response cookies, instead of calling into `response.headers.multi_items()` in order to properly get all separate `Set-Cookie` headers:

https://github.com/encode/httpx/blob/0e73be83a8e133521c8c0cd7426a33cda196eef0/httpx/_models.py#L1199

(Still not sure if concatenating headers into comma-separated lists shouldn't include some conditions on what header is being flattened. Eg for `Set-Cookie` we probably want to only keep the latest value, instead of concatenating all values which results in an invalid `Set-Cookie` value syntax.)
So to be clear, the fix is to change this line to use `.multi_items()`:

https://github.com/encode/httpx/blob/0e73be83a8e133521c8c0cd7426a33cda196eef0/httpx/_models.py#L1199

@djigit0s Does this sound legit to you? If you're up for throwing a PR together for this feel free to give it a go!

Resolving patch:
diff --git a/httpx/_models.py b/httpx/_models.py
--- a/httpx/_models.py
+++ b/httpx/_models.py
@@ -1196,6 +1196,9 @@ def __init__(self, response: Response):
 
         def info(self) -> email.message.Message:
             info = email.message.Message()
-            for key, value in self.response.headers.items():
+            for key, value in self.response.headers.multi_items():
+                #  Note that setting `info[key]` here is an "append" operation,
+                # not a "replace" operation.
+                # https://docs.python.org/3/library/email.compat32-message.html#email.message.Message.__setitem__
                 info[key] = value
             return info


Test patch:
diff --git a/tests/models/test_cookies.py b/tests/models/test_cookies.py
--- a/tests/models/test_cookies.py
+++ b/tests/models/test_cookies.py
@@ -1,10 +1,12 @@
+import http
+
 import pytest
 
-from httpx import CookieConflict, Cookies
+import httpx
 
 
 def test_cookies():
-    cookies = Cookies({"name": "value"})
+    cookies = httpx.Cookies({"name": "value"})
     assert cookies["name"] == "value"
     assert "name" in cookies
     assert len(cookies) == 1
@@ -19,8 +21,8 @@ def test_cookies():
 
 
 def test_cookies_update():
-    cookies = Cookies()
-    more_cookies = Cookies()
+    cookies = httpx.Cookies()
+    more_cookies = httpx.Cookies()
     more_cookies.set("name", "value", domain="example.com")
 
     cookies.update(more_cookies)
@@ -29,11 +31,11 @@ def test_cookies_update():
 
 
 def test_cookies_with_domain():
-    cookies = Cookies()
+    cookies = httpx.Cookies()
     cookies.set("name", "value", domain="example.com")
     cookies.set("name", "value", domain="example.org")
 
-    with pytest.raises(CookieConflict):
+    with pytest.raises(httpx.CookieConflict):
         cookies["name"]
 
     cookies.clear(domain="example.com")
@@ -41,10 +43,35 @@ def test_cookies_with_domain():
 
 
 def test_cookies_with_domain_and_path():
-    cookies = Cookies()
+    cookies = httpx.Cookies()
     cookies.set("name", "value", domain="example.com", path="/subpath/1")
     cookies.set("name", "value", domain="example.com", path="/subpath/2")
     cookies.clear(domain="example.com", path="/subpath/1")
     assert len(cookies) == 1
     cookies.delete("name", domain="example.com", path="/subpath/2")
     assert len(cookies) == 0
+
+
+def test_multiple_set_cookie():
+    jar = http.cookiejar.CookieJar()
+    headers = [
+        (
+            b"Set-Cookie",
+            b"1P_JAR=2020-08-09-18; expires=Tue, 08-Sep-2020 18:33:35 GMT; "
+            b"path=/; domain=.example.org; Secure",
+        ),
+        (
+            b"Set-Cookie",
+            b"NID=204=KWdXOuypc86YvRfBSiWoW1dEXfSl_5qI7sxZY4umlk4J35yNTeNEkw15"
+            b"MRaujK6uYCwkrtjihTTXZPp285z_xDOUzrdHt4dj0Z5C0VOpbvdLwRdHatHAzQs7"
+            b"7TsaiWY78a3qU9r7KP_RbSLvLl2hlhnWFR2Hp5nWKPsAcOhQgSg; expires=Mon, "
+            b"08-Feb-2021 18:33:35 GMT; path=/; domain=.example.org; HttpOnly",
+        ),
+    ]
+    request = httpx.Request("GET", "https://www.example.org")
+    response = httpx.Response(200, request=request, headers=headers)
+
+    cookies = httpx.Cookies(jar)
+    cookies.extract_cookies(response)
+
+    assert len(cookies) == 2