CookieError in requests and not in urllib2

In `0.9.0`:

```
>>> import requests
>>> import urllib2
>>> url = 'http://www.miniclip.com/games/en/'
>>> len(urllib2.urlopen(url).read())
111106
>>> requests.get(url)
len(urllTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 50, in get
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 38, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/sessions.py", line 200, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 515, in send
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 197, in _build_response
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 177, in build
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/core.py", line 19, in dict_from_string
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 641, in load
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 674, in __ParseString
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 594, in __set
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 468, in set
requests.packages.oreos.monkeys.CookieError: Illegal key value: ad_is_enabled-position2:homepage
```


That's because urllib2 isn't reading the cookies unless you tell it to.


Another url:

```
>>> url = 'https://partner.microsoft.com'
>>> requests.get(url)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 50, in get
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 38, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/sessions.py", line 200, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 515, in send
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 254, in _build_response
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 515, in send
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 197, in _build_response
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 177, in build
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/core.py", line 19, in dict_from_string
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 641, in load
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 674, in __ParseString
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 594, in __set
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 468, in set
requests.packages.oreos.monkeys.CookieError: Illegal key value: 04:31:20&UserEBacExpression
>>> len(urllib2.urlopen(url).read())
25402
```


Apparently you can't have `:` in cookie keys. I'll try to add this to Oreos.


```
>>> url = 'http://www.swiss.com/'
>>> len(requests.get(url).content)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 50, in get
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 38, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/sessions.py", line 200, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 515, in send
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 197, in _build_response
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 177, in build
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/core.py", line 19, in dict_from_string
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 641, in load
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 674, in __ParseString
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 594, in __set
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/oreos/monkeys.py", line 468, in set
requests.packages.oreos.monkeys.CookieError: Illegal key value: ISAWPLB{34F3C496-0632-48BF-B442-6ADE6334837A}
>>> len(urllib2.urlopen(url).read())
79884
>>> 
```


Whew, there's plenty more. I'll stop there. :-)


Clearly I need to catch cookie errors :)


If you _really_ want me to, I can switch around my page fetching to record where requests fails and urllib2 succeeds, aggregate that over 2 hours (so every single page is fetched at least once), and report back with a number of discrepancies. It'll take some work, but if you're itching for broken test cases, there's no better dataset. Alternatively, you could hook this up yourself by just grabbing a list of URLs and running them through both, logging exceptions as they come through with discrepancies. And I can provide that dataset instead, ordered by popularity, of up to 1M urls if you like.


A simple (massive) list of the failures would be fantastic!


This really is a big headache
One more in the list!

> > > requests.get('http://www.luisaviaroma.com').cookies
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "requests/api.py", line 51, in get
> > >     return request('get', url, *_kwargs)
> > >   File "requests/api.py", line 39, in request
> > >     return s.request(method=method, url=url, *_kwargs)
> > >   File "requests/sessions.py", line 200, in request
> > >     r.send(prefetch=prefetch)
> > >   File "requests/models.py", line 513, in send
> > >     self._build_response(r)
> > >   File "requests/models.py", line 195, in _build_response
> > >     r = build(resp)
> > >   File "requests/models.py", line 175, in build
> > >     cookies = oreos.dict_from_string(cookie_header)
> > >   File "requests/packages/oreos/core.py", line 19, in dict_from_string
> > >     c.load(s)
> > >   File "requests/packages/oreos/monkeys.py", line 641, in load
> > >     self.__ParseString(rawdata)
> > >   File "requests/packages/oreos/monkeys.py", line 674, in __ParseString
> > >     self.__set(K, rval, cval)
> > >   File "requests/packages/oreos/monkeys.py", line 594, in __set
> > >     M.set(key, real_value, coded_value)
> > >   File "requests/packages/oreos/monkeys.py", line 468, in set
> > >     raise CookieError("Illegal key value: %s" % key)
> > > requests.packages.oreos.monkeys.CookieError: Illegal key value: 12:23:49&ladate


Oh and btw, same url, same thing, same worked on my friends laptop everything the same :S
I am so confused (and pissed)


This issue was being discussed last year over on the main Python bug tracker: http://bugs.python.org/issue2193

In summary, the simplest fix (which I've implemented in my fork of requests) is to simply add the colon and curly braces into the allowed characters in cookie keys. This works fine, but makes it possible to create cookies that contain those values as well, which is technically against the relevant RFC (RFC 2109). Of course, we can just do it and let the receiving party deal with it. Testing this fix is difficult, because HTTPBin is RFC compliant and won't return cookie keys with these characters in them.

The ideal fix is to accept them in keys, but not send them out. This will complicate the code a bit more, but is probably not the end of the world to implement. I haven't looked hard enough at the oreos code to know, but if we're interested in doing that I can take another look.

Basically, what is the consensus on the 'pythonic' thing to do? Do we fail, because the cookie we received is technically invalid; do we accept and send technically invalid cookies; or do we be liberal in what we accept and conservative in what we send?

[NB: The current state of Oreos is to accept other characters that are supposed to be forbidden, namely the square brackets. So in a sense the implementation is already non-compliant.]


Follow-up question. If we take the route of being liberal in what we accept but conservative in what we send, then what happens if we're redirected? For example, the http://www.swiss.com/ example above involves a redirect. We accept the malformed cookie, but trying to send it throws a CookieError.


CookieErrors shouldn't be happening.. but they are. They should be caught.


Just adding a "Me too" here - causing us some parsing problems at Readability.


The latest release should make the CookieErrors far less common, but they still need to be caught.


Ah, great. I updated to the latest version and I'm finding another issue though. I'll trawl the issues and if it doesn't exist already, report.


Feel free to email me, as always.


Hmmm, just encountered a CookieError =/, Is there any way to just ignore all cookie errors @kennethreitz? 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.6/site-packages/requests/api.py", line 79, in get
    :param url: URL for the new :class:`Request` object.
  File "/Library/Python/2.6/site-packages/requests/api.py", line 66, in request
    """Sends a HEAD request. Returns :class:`Response` object.
  File "/Library/Python/2.6/site-packages/requests/sessions.py", line 191, in request

  File "/Library/Python/2.6/site-packages/requests/models.py", line 462, in send
    r = self.proxy_auth(self)
  File "/Library/Python/2.6/site-packages/requests/models.py", line 242, in _build_response
    except KeyError:
  File "/Library/Python/2.6/site-packages/requests/models.py", line 462, in send
    r = self.proxy_auth(self)
  File "/Library/Python/2.6/site-packages/requests/models.py", line 186, in _build_response
    response.cookies = cookies
  File "/Library/Python/2.6/site-packages/requests/models.py", line 169, in build
    response.status_code = getattr(resp, 'status', None)
  File "/Library/Python/2.6/site-packages/requests/packages/oreos/core.py", line 19, in dict_from_string
    c.load(s)
  File "/Library/Python/2.6/site-packages/requests/packages/oreos/monkeys.py", line 641, in load
    map(Cookie.__setitem__, d.keys(), d.values())
  File "/Library/Python/2.6/site-packages/requests/packages/oreos/monkeys.py", line 674, in __ParseString
    M[ K ] = _unquote(V)
  File "/Library/Python/2.6/site-packages/requests/packages/oreos/monkeys.py", line 594, in __set
    def __set(self, key, real_value, coded_value):
  File "/Library/Python/2.6/site-packages/requests/packages/oreos/monkeys.py", line 468, in set
    if key.lower() in self._reserved:
requests.packages.oreos.monkeys.CookieError: Illegal key value: 10:54:14&CookieExpireDate


@dalanmiller 
How can we reproduce this?


@piotr-dobrogost Here's at least one sample url: http://mylikes.com/l/1tsn6

Like: 

```
>>> import requests
>>> requests.get("http://mylikes.com/l/1tsn6")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/sessions.py", line 200, in request
    r.send(prefetch=prefetch)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 557, in send
    self._build_response(r)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 263, in _build_response
    request.send()
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 557, in send
    self._build_response(r)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 203, in _build_response
    r = build(resp)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 184, in build
    cookies = dict_from_string(cookie_header)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/utils.py", line 30, in dict_from_string
    c.load(s)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/packages/oreos/monkeys.py", line 644, in load
    self.__ParseString(rawdata)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/packages/oreos/monkeys.py", line 677, in __ParseString
    self.__set(K, rval, cval)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/packages/oreos/monkeys.py", line 597, in __set
    M.set(key, real_value, coded_value)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/packages/oreos/monkeys.py", line 471, in set
    raise CookieError("Illegal key value: %s" % key)
requests.packages.oreos.monkeys.CookieError: Illegal key value: Message(userRedirectUrl)
```

I can scrape up a couple more in a bit.


One more: http://it.toolbox.com/blogs/db2zos/the-question-is-the-mainframe-still-the-right-answer-for-your-business-50380?ccy=us&cm=h&cpb=swg&cd=20120221&ct=tips&cr=buzzsheet&csr=systemz&cmp=smm


I hate cookies.


@piotr-dobrogost Try sending a get request with requests at http://nordstrom.com


@dalanmiller What version of requests are you using? When I hit http://nordstrom.com/, I find the illegal cookie key to be "13:05:29&CookieExpireDate", which is illegal because it contains colons. This was fixed in commit f72c13f .


@Lukasa According to pip I'm using Requests 0.10.4

```
...
requests==0.10.4
...
```

As well:

```
help(requests)
```

Results in 

```
...
VERSION
    0.10.4
...
```

I don't even need cookies for what I'm trying to do, is there anyway I can just set something to ignore them and errors?


The two failures given by @umbrae fail because the first one contains parentheses and the second one contains a question mark. I'm reluctant to just keep patching up failures when they arise, I'd be more inclined to rewrite the code to correctly handle all of the cookies. I'm looking to get around to it sometime in the next few days, but if someone else wants to tackle it they're welcome.

@dalanmiller Odd. If I launch a new virtualenv, pip install requests and then perform a get on http://nordstrom.com/ , I don't get a CookieError, and pip claims I'm running 0.10.4 as well.


@Lukasa Tried using a new virtualenv, then installing requests via pip and everything worked perfectly fine like you did. 

Going to try uninstalling / reinstalling requests.

Thank you for all your help! 

p.s.

A `pip uninstall requests` and then a `pip install requests --upgrade` did the trick! Now working! Thank you @Lukasa, @KennethReitz, @piotr-dobrogost. 


Fixed in d078b69aefc24ec6fc52fcaf4334c2573e5168fc


Thank you! :cake: