kennethreitz/requests — Issue #440: Requests (via oreos.monkeys) quotes cookie *values* -- browsers do not
Author: mattklein
Created: 2012-02-18 19:35:12+00:00
State: open

I'm having an issue that has me questioning Requests's (Oreo's?) cookie handling. Using Requests, I submit a request that successfully authenticates me to a particular server. The response includes a cookie whose *value* contains an equals ("=") character. Because the cookie's value contains "=", which is not a "legal" char (as per _LegalChars in monkeys.py), Requests quotes the entire cookie value when it submits it in subsequent requests. The server, though, considers the quoted cookie to be different than the cookie it sent to the client -- so the client is no longer authenticated.

See this flow:

In [6]: session = requests.session(headers=_default_headers)

In [7]: data = {
...: 'userID': 'myuser',
...: 'password': 'mypassword',
...: }

In [8]: resp = session.post('https://www.discovercard.com/cardmembersvcs/loginlogout/app/signin', data=data)

In [9]: resp.headers
Out[9]:
{'cache-control': 'no-cache="set-cookie, set-cookie2"',
'connection': 'Keep-Alive',

'set-cookie': 'v1st=6B2AF394DCD43A57; path=/; expires=Wed, 19 Feb 2020 14:28:00 GMT; domain=.discovercard.com,
dfsedskey=62689366; Expires=Sun, 17 Feb 2013 17:25:06 GMT; Path=/; Domain=.discovercard.com,
PMData=PMV3EKKwYWpRB0ZAh7NnXgngf+fGUH7/c6RfKfa7Hiv5v2h1S1iS4MjexZtrnrPMErcNR0eyCzoWUfNFrj6bUkqxR0aQ==; Expires=Sun, 17 Feb 2013 17:25:06 GMT; Path=/; Domain=.discovercard.com; Secure,
dcsession=E7dc4gmtmOk9LkTB4xeTjRs82g8=; Path=/; Domain=.discovercard.com; Secure,
ACLOGIN=incntvtyp1=CBB; Expires=Sun, 17 Feb 2013 17:25:06 GMT; Path=/; Domain=.discovercard.com,
ACHASH=1338528939; Expires=Sun, 17 Feb 2013 17:25:06 GMT; Path=/; Domain=.discovercard.com, dcuserid=***CLEARED***; Path=/; Domain=.discovercard.com,
JSESSIONID=0001cT7y8RWiiQDuizJpy2gjQWw:16cfscb3m; Path=/',

}

So the *value* of the cookie with key "dcsession" is "E7dc4gmtmOk9LkTB4xeTjRs82g8=". The cookies dict is as expected:

In [10]: session.cookies['dcsession']
Out[10]: 'E7dc4gmtmOk9LkTB4xeTjRs82g8='

In [11]: resp.cookies['dcsession']
Out[11]: 'E7dc4gmtmOk9LkTB4xeTjRs82g8='

But when I make my next request to the server:

In [12]: resp = session.get('https://www.discovercard.com/cardmembersvcs/achome/homepage')

The response to that request indicates "session timed out or not established". This is because of how the cookie was sent in this request:

In [14]: resp.request.headers['Cookie']
Out[14]: 'ACHASH=1338528939; ACLOGIN="incntvtyp1=CBB"; JSESSIONID="0001cT7y8RWiiQDuizJpy2gjQWw:16cfscb3m";
PMData="PMV3EKKwYWpRB0ZAh7NnXgngf+fGUH7/c6RfKfa7Hiv5v2h1S1iS4MjexZtrnrPMErcNR0eyCzoWUfNFrj6bUkqxR0aQ==";
dcsession="E7dc4gmtmOk9LkTB4xeTjRs82g8="; dcuserid=***CLEARED***; dfsedskey=62689366; v1st=6B2AF394DCD43A57'

Because the server considers the quoted cookie being returned as being different from the non-quoted cookie it sent, the session is no longer authenticated.

The condition where the cookie comes back from the server containing the equals character is not rare or intermittent -- it always comes back like this, and so the authentication always fails. This is not broken for people using browsers! No browser I've tried (Chrome, Safari) quotes the cookie values, so there's no problem authenticating and carrying on a session. Obviously the server in this case is an established financial institution, presumably with many millions of users, so I think that the way Requests/Oreos is handling cookies is a problem.

From what I can tell from the code, Requests instantiates a monkeys.SimpleCookie. SimpleCookie decides that, since the value contains something other than what's in _LegalChars, it needs to quote the value. I can patch this so that it works for me any number of ways: adding "=" into _LegalChars, or preventing SimpleCookie from ever quoting a value, or having Requests use BaseCookie rather than SimpleCookie, or having my application re-construct the cookie header from the cookies before my next request, etc. But this does feel like a bug that should be fixed in the main branch for everybody. I'm happy to contribute a fix but am interested to know what Kenneth and/or anybody else thinks is the right way to do it.