Repository: mitmproxy/mitmproxy

Issue:
deleting cookies (stickycookies) does not work
##### Steps to reproduce the problem:
1. start mitmdump with stickycookies: mitmdump -t "." and point a browser through it
2. visit https://httpbin.org/cookies/set?aaa=1 to set a cookie
3. visit https://httpbin.org/cookies/delete?aaa= to delete the cookie
4. visit https://httpbin.org/cookies to display the cookies
##### What is the expected behavior?

The expected behavior, according to RFC 6265, is that the cookie is deleted
##### What went wrong?

The cookie is not deleted
##### Any other comments?

The cookie library used by mitmproxy is Cookie.py from python, which is woefully outdated.
It implements RFC 2109 from 1997 and does not (yet?) support RFC 6265 from 2011.

RFC 6265 standardises both "expires" and "max-age" attributes which can be used
for expiration control of cookies. A cookie can be deleted by setting "expires" to a date in the past,
or "max-age" to 0.

A more up-to-date cookie library is probably beneficial here. I'm not sure if there are any
special reasons for depending on Cookie.py?
Is it better to attempt to fix cookie.py, or switch mitmproxy to another cookie library?

---

Mitmproxy Version: latest
Operating System:



Discussion:
Correct me if I'm wrong, but the entire point of `stickycookies` is that they don't get deleted, right?

Ar far as I interpreted stickycookies, they are persistent for the lifetime of the mitmproxy session, but should still follow cookie semantics. If the server requests that a cookie be removed (for instance if the user logs out explicitely), shouldn't mitmproxy then remove them? 

> If the server requests that a cookie be removed (for instance if the user logs out explicitely), shouldn't mitmproxy then remove them?

This is a good question. I think you are right and we should probably delete cookies if the server instructs us to do so.

> A more up-to-date cookie library is probably beneficial here. I'm not sure if there are any
> special reasons for depending on Cookie.py?
> Is it better to attempt to fix cookie.py, or switch mitmproxy to another cookie library?

@cortesi: you have cursed a few times about the cookie stuff IIRC. Can we get your expert opinion here? :smiley:

I agree that the cookies should be deleted when we're instructed to do so by the server. Python's built-in Cookie library is fractally terrible, and fails to parse many common cookies seen in the wild. I'd love to get rid of it entirely (which is why we have our own implementation). I'd be in favor of getting rid of it entirely.

@Kriechi worked on some parts of this during the Honeynet workshop. I think we now only depend on `http_cookiejar.domain_match` from the stdlib? (which I think is fine)

(Does not touch the deletion issue though)


Resolving patch:
diff --git a/mitmproxy/flow/modules.py b/mitmproxy/flow/modules.py
--- a/mitmproxy/flow/modules.py
+++ b/mitmproxy/flow/modules.py
@@ -320,10 +320,20 @@ def handle_response(self, f):
         for name, (value, attrs) in f.response.cookies.items(multi=True):
             # FIXME: We now know that Cookie.py screws up some cookies with
             # valid RFC 822/1123 datetime specifications for expiry. Sigh.
-            a = self.ckey(attrs, f)
-            if self.domain_match(f.request.host, a[0]):
-                b = attrs.with_insert(0, name, value)
-                self.jar[a][name] = b
+            dom_port_path = self.ckey(attrs, f)
+
+            if self.domain_match(f.request.host, dom_port_path[0]):
+                if cookies.is_expired(attrs):
+                    # Remove the cookie from jar
+                    self.jar[dom_port_path].pop(name, None)
+
+                    # If all cookies of a dom_port_path have been removed
+                    # then remove it from the jar itself
+                    if not self.jar[dom_port_path]:
+                        self.jar.pop(dom_port_path, None)
+                else:
+                    b = attrs.with_insert(0, name, value)
+                    self.jar[dom_port_path][name] = b
 
     def handle_request(self, f):
         l = []
diff --git a/netlib/http/cookies.py b/netlib/http/cookies.py
--- a/netlib/http/cookies.py
+++ b/netlib/http/cookies.py
@@ -1,7 +1,8 @@
 import collections
+import email.utils
 import re
+import time
 
-import email.utils
 from netlib import multidict
 
 """
@@ -260,3 +261,29 @@ def refresh_set_cookie_header(c, delta):
     if not ret:
         raise ValueError("Invalid Cookie")
     return ret
+
+
+def is_expired(cookie_attrs):
+    """
+        Determines whether a cookie has expired.
+
+        Returns: boolean
+    """
+
+    # See if 'expires' time is in the past
+    expires = False
+    if 'expires' in cookie_attrs:
+        e = email.utils.parsedate_tz(cookie_attrs["expires"])
+        if e:
+            exp_ts = email.utils.mktime_tz(e)
+            now_ts = time.time()
+            expires = exp_ts < now_ts
+
+    # or if Max-Age is 0
+    max_age = False
+    try:
+        max_age = int(cookie_attrs.get('Max-Age', 1)) == 0
+    except ValueError:
+        pass
+
+    return expires or max_age


Test patch:
diff --git a/test/mitmproxy/test_flow.py b/test/mitmproxy/test_flow.py
--- a/test/mitmproxy/test_flow.py
+++ b/test/mitmproxy/test_flow.py
@@ -55,14 +55,16 @@ def test_domain_match(self):
         assert s.domain_match("google.com", ".google.com")
 
     def test_response(self):
-        c = "SSID=mooo; domain=.google.com, FOO=bar; Domain=.google.com; Path=/; " \
+        c = (
+            "SSID=mooo; domain=.google.com, FOO=bar; Domain=.google.com; Path=/; "
             "Expires=Wed, 13-Jan-2021 22:23:01 GMT; Secure; "
+        )
 
         s, f = self._response(c, "host")
         assert not s.jar.keys()
 
         s, f = self._response(c, "www.google.com")
-        assert s.jar.keys()
+        assert list(s.jar.keys())[0] == ('.google.com', 80, '/')
 
         s, f = self._response("SSID=mooo", "www.google.com")
         assert list(s.jar.keys())[0] == ('www.google.com', 80, '/')
@@ -101,6 +103,16 @@ def test_response(self):
         assert len(s.jar[googlekey]) == 1
         assert list(s.jar[googlekey]["somecookie"].values())[0] == "newvalue"
 
+    def test_response_delete(self):
+        c = "duffer=zafar; Path=/", "www.google.com"
+
+        # Test that a cookie is be deleted
+        # by setting the expire time in the past
+        s, f = self._response(*c)
+        f.response.headers["Set-Cookie"] = "duffer=; Expires=Thu, 01-Jan-1970 00:00:00 GMT"
+        s.handle_response(f)
+        assert not s.jar.keys()
+
     def test_request(self):
         s, f = self._response("SSID=mooo", b"www.google.com")
         assert "cookie" not in f.request.headers
diff --git a/test/netlib/http/test_cookies.py b/test/netlib/http/test_cookies.py
--- a/test/netlib/http/test_cookies.py
+++ b/test/netlib/http/test_cookies.py
@@ -245,3 +245,24 @@ def test_refresh_cookie():
     assert cookies.refresh_set_cookie_header(c, 0)
     c = "foo/bar=bla"
     assert cookies.refresh_set_cookie_header(c, 0)
+
+
+def test_is_expired():
+    CA = cookies.CookieAttrs
+
+    # A cookie can be expired
+    # by setting the expire time in the past
+    assert cookies.is_expired(CA([("Expires", "Thu, 01-Jan-1970 00:00:00 GMT")]))
+
+    # or by setting Max-Age to 0
+    assert cookies.is_expired(CA([("Max-Age", "0")]))
+
+    # or both
+    assert cookies.is_expired(CA([("Expires", "Thu, 01-Jan-1970 00:00:00 GMT"), ("Max-Age", "0")]))
+
+    assert not cookies.is_expired(CA([("Expires", "Thu, 24-Aug-2063 00:00:00 GMT")]))
+    assert not cookies.is_expired(CA([("Max-Age", "1")]))
+    assert not cookies.is_expired(CA([("Expires", "Thu, 15-Jul-2068 00:00:00 GMT"), ("Max-Age", "1")]))
+
+    assert not cookies.is_expired(CA([("Max-Age", "nan")]))
+    assert not cookies.is_expired(CA([("Expires", "false")]))