Repository: mitmproxy/mitmproxy

Issue:
Sticky cookies are improperly formatted.
##### Steps to reproduce the problem:

1. Go to http://www.html-kit.com/tools/cookietester/
2. Click 'Set Test Cookie'
3. Observe that one cookie is sent to the server.
4. Remove the cookie.
5. launch mitmproxy with `mitmproxy -t html-kit\.com` and tell your browser to use it as a proxy
6. Reload the page.
7. Click 'Set Test Cookie'
8. Observe that two 'cookies' are sent to the server.

##### Any other comments? What have you tried so far?
There appears to be a comma in the output of mitmproxy, even though it is surrounded by quotes. It's possible, then that this is a parsing fail on the tool's end caused by a difference in what's sent back for the format of the date. Still, should it really be changing that?

##### System information
Arch Linux, freshly updated.

Mitmproxy version: 2.0.1 (release version) 
Python version: 3.6.0
Platform: Linux-4.10.6-1-ARCH-x86_64-with-glibc2.3.4
SSL version: OpenSSL 1.0.2k  26 Jan 2017



Discussion:
Thanks for the report. It looks like we parse it correctly:

```python
>>> from mitmproxy.net.http import cookies
>>> cookies.parse_set_cookie_header("TestCookie_Name_201704075049=TestCookie_Value_035049; Domain=www.html-kit.com; Expires=Sun, 09-Apr-2017 08:50:54 GMT; Path=/")
[
    ('TestCookie_Name_201704075049', 'TestCookie_Value_035049', CookieAttrs[
        ('Domain', 'www.html-kit.com'), 
        ('Expires', 'Sun, 09-Apr-2017 08:50:54 GMT'), 
        ('Path', '/')
    ])
]
```
But for some reason [stickycookie.py](https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/addons/stickycookie.py) includes the attributes in the `Cookie` header. My understanding is that this is very wrong. We definitely should catch that in our tests.
So, I've got a hack that seems to fix it. If you replace:
`l.extend([cookies.format_cookie_header(c[name].items(multi=True)) for name in c.keys()])`
with
`l.extend([cookies.format_cookie_header([c[name].items(multi=True)[0]]) for name in c.keys()])`

I don't like this solution, though.

The issue appears to be with how the cookies are stored in the sticky cookie implementation, as well as, to an extent, the `format_cookie_header()` implementation. Sticky cookies stores them as a list of `(key, value)` tuple lists, instead of a list of `(key, value, CookieAttrs[(k,v),(k,v)])` tuples as seems to be used elsewhere. `format_cookie_header()` expects a list of `(key, value)` tuples for each cookie. It is noteworthy that this seems broken at the moment, because the way it should work as currently implemented should give me cookies for every attribute.

The changes I would reccomend would be to have `format_cookie_header()` take a list `(key, value, CookieAttrs[(k,v),(k,v)])` tuples, same as `format_set_cookie_header()` and change the cookie jar in sticky cookies to store cookies in this same format. If there are big issues with this, the hack I mentioned above can be applied.
I'm going to go ahead and submit the hack in a pull request, for now.

Resolving patch:
diff --git a/mitmproxy/addons/stickycookie.py b/mitmproxy/addons/stickycookie.py
--- a/mitmproxy/addons/stickycookie.py
+++ b/mitmproxy/addons/stickycookie.py
@@ -1,14 +1,14 @@
 import collections
 from http import cookiejar
+from typing import List, Tuple, Dict, Optional  # noqa
 
+from mitmproxy import http, flowfilter, ctx, exceptions
 from mitmproxy.net.http import cookies
 
-from mitmproxy import exceptions
-from mitmproxy import flowfilter
-from mitmproxy import ctx
+TOrigin = Tuple[str, int, str]
 
 
-def ckey(attrs, f):
+def ckey(attrs: Dict[str, str], f: http.HTTPFlow) -> TOrigin:
     """
         Returns a (domain, port, path) tuple.
     """
@@ -21,18 +21,18 @@ def ckey(attrs, f):
     return (domain, f.request.port, path)
 
 
-def domain_match(a, b):
-    if cookiejar.domain_match(a, b):
+def domain_match(a: str, b: str) -> bool:
+    if cookiejar.domain_match(a, b):  # type: ignore
         return True
-    elif cookiejar.domain_match(a, b.strip(".")):
+    elif cookiejar.domain_match(a, b.strip(".")):  # type: ignore
         return True
     return False
 
 
 class StickyCookie:
     def __init__(self):
-        self.jar = collections.defaultdict(dict)
-        self.flt = None
+        self.jar = collections.defaultdict(dict)  # type: Dict[TOrigin, Dict[str, str]]
+        self.flt = None  # type: Optional[flowfilter.TFilter]
 
     def configure(self, updated):
         if "stickycookie" in updated:
@@ -46,7 +46,7 @@ def configure(self, updated):
             else:
                 self.flt = None
 
-    def response(self, flow):
+    def response(self, flow: http.HTTPFlow):
         if self.flt:
             for name, (value, attrs) in flow.response.cookies.items(multi=True):
                 # FIXME: We now know that Cookie.py screws up some cookies with
@@ -63,24 +63,21 @@ def response(self, flow):
                         if not self.jar[dom_port_path]:
                             self.jar.pop(dom_port_path, None)
                     else:
-                        b = attrs.copy()
-                        b.insert(0, name, value)
-                        self.jar[dom_port_path][name] = b
+                        self.jar[dom_port_path][name] = value
 
-    def request(self, flow):
+    def request(self, flow: http.HTTPFlow):
         if self.flt:
-            l = []
+            cookie_list = []  # type: List[Tuple[str,str]]
             if flowfilter.match(self.flt, flow):
-                for domain, port, path in self.jar.keys():
+                for (domain, port, path), c in self.jar.items():
                     match = [
                         domain_match(flow.request.host, domain),
                         flow.request.port == port,
                         flow.request.path.startswith(path)
                     ]
                     if all(match):
-                        c = self.jar[(domain, port, path)]
-                        l.extend([cookies.format_cookie_header(c[name].items(multi=True)) for name in c.keys()])
-            if l:
+                        cookie_list.extend(c.items())
+            if cookie_list:
                 # FIXME: we need to formalise this...
-                flow.request.stickycookie = True
-                flow.request.headers["cookie"] = "; ".join(l)
+                flow.metadata["stickycookie"] = True
+                flow.request.headers["cookie"] = cookies.format_cookie_header(cookie_list)


Test patch:
diff --git a/test/mitmproxy/addons/test_dumper.py b/test/mitmproxy/addons/test_dumper.py
--- a/test/mitmproxy/addons/test_dumper.py
+++ b/test/mitmproxy/addons/test_dumper.py
@@ -68,7 +68,6 @@ def test_simple():
         ctx.configure(d, flow_detail=4)
         flow = tflow.tflow()
         flow.request = tutils.treq()
-        flow.request.stickycookie = True
         flow.client_conn = mock.MagicMock()
         flow.client_conn.address[0] = "foo"
         flow.response = tutils.tresp(content=None)
diff --git a/test/mitmproxy/addons/test_stickycookie.py b/test/mitmproxy/addons/test_stickycookie.py
--- a/test/mitmproxy/addons/test_stickycookie.py
+++ b/test/mitmproxy/addons/test_stickycookie.py
@@ -110,8 +110,8 @@ def test_response_overwrite(self):
             f.response.headers["Set-Cookie"] = c2
             sc.response(f)
             googlekey = list(sc.jar.keys())[0]
-            assert len(sc.jar[googlekey].keys()) == 1
-            assert list(sc.jar[googlekey]["somecookie"].items())[0][1] == "newvalue"
+            assert len(sc.jar[googlekey]) == 1
+            assert sc.jar[googlekey]["somecookie"] == "newvalue"
 
     def test_response_delete(self):
         sc = stickycookie.StickyCookie()