unicode in cookies

```
Set-Cookie: xx=\u4E2A\u4EBA; expires=Wed, 04-Apr-42 15:00:05 GMT; path=/; 
```

requests SimpleCookie

``` python
>>> s = SimpleCookie()
>>> s.load('xx=\u4E2A\u4EBA; expires=Wed, 04-Apr-42 15:00:05 GMT; path=/; ')
>>> s
<SimpleCookie: xx=''>
```

The result should be like this.

``` python
>>> s
<SimpleCookie: xx='\u4E2A\u4EBA'>
```


I've been studying this issue for a couple of days, I don't believe that this is a valid use case.  The RFC's for HTML require that the header be encoded in iso-8859-1, so there should never be a case where the header contains unencoded unicode characters as you've shown here.

Nonetheless, I'm thinking about whether it's possible to handle cases like this, which do seem to occur in the real world due to sloppy implementation by others.


@benmao 
What Python version are we talking about?

@rwzehner 
What does HTML spec have to do with HTTP headers?


Sorry, should have written HTTP spec, not HTML spec.  Check out RFC 2109, which defines the cookie spec and references RFC 2068.  Bottom line, although HTTP allows one to specify different encodings for the body of a reply, all of the header information must be Latin-1 encoded.

In any case, looking at the above situation further (with Python 2.7.1), the string passed in contains Unicode escapes, but does not have a unicode signifier on it, so Python does not properly interpret the escapes.  This could be fixed by decoding the string using the raw_unicode_escape encoding:

```
s.load('xx=\u4E2A\u4EBA; expires=Wed, 04-Apr-42 15:00:05 GMT; path=/; '.decode('raw_unicode_escape'))
```

It's a fairly easy change to make, but I haven't been able to think through whether it would potentially break if the user had already encoded their string using another technique.


@piotr-dobrogost Python Version: 2.7.2

I found monkeys.py is copy of Cookie.py which from Python Lib.

urllib2 + cookielib works well.

So I do not think "monkeys.py" have problem.


https://gist.github.com/2450467

I have write a test .


@benmao

> The result should be like this.

```
<SimpleCookie: xx='\u4E2A\u4EBA'>
```

I guess you meant 

```
<SimpleCookie: xx='\\u4E2A\\u4EBA'>
```

because backslash which is not part of escape sequence is doubled in string representation.

According to RFC 2109 backslash is not allowed in cookie's value unless the value is quoted. As [`Cookie`](http://docs.python.org/library/cookie.html) module implements this RFC (albeit with modifications) this could be the reason for not _accepting_ value `\u4E2A\u4EBA` as a valid cookie's value. The reason it works in `cookielib` is because it has its own `Cookie` class and apparently different parser - see [parse_ns_headers](http://hg.python.org/cpython/file/2.7/Lib/cookielib.py#l444) method. [Cookies](https://github.com/sashahart/cookies) lib doesn't allow this cookie value as well:

```
>>> import cookies
>>> cookies.Cookies.from_response('xx=\u4E2A\u4EBA; expires=Wed, 04-Apr-42 15:00:05 GMT; path=/; ')
Cookies(xx=u'')
```

However, there is a difference when using double quoted value of cookie:

```
>>> import Cookie
>>> s=Cookie.SimpleCookie()
>>> s.load('xx="\u4E2A\u4EBA"; expires=Wed, 04-Apr-42 15:00:05 GMT; path=/; ')
>>> s
<SimpleCookie: xx='u4E2Au4EBA'>

>>> import cookies
>>> cookies.Cookies.from_response('xx="\u4E2A\u4EBA"; expires=Wed, 04-Apr-42 15:00:05 GMT; path=/; ')
Cookies(xx=u'\\u4E2A\\u4EBA')
```

RFC 2109 defines cookie's value (`value`) as:

```
value           =       word
word            =       token | quoted-string
```

, where `quoted-string` is defined in RFC 2068 as:

```
quoted-string  = ( <"> *(qdtext) <"> )

qdtext         = <any TEXT except <">>

The backslash character ("\") may be used as a single-character quoting
mechanism only within quoted-string and comment constructs.

quoted-pair    = "\" CHAR
```

Because there's no further description of quoting mechanism I don't know which behavior is right (see https://github.com/sashahart/cookies/issues/1).

What's interesting is that RFC 6265 (which obsoletes RFC 2109 and which Cookies supposedly implements) states this:

```
cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                   ; US-ASCII characters excluding CTLs,
                   ; whitespace DQUOTE, comma, semicolon,
                   ; and backslash
```

which does not allow backslash in cookie's value at all. This is surprising as this RFC was meant to take into account existing practice and relax rules of previous RFCs.


@rwzehner 

> Bottom line, although HTTP allows one to specify different encodings for the body of a reply, all of the header information must be Latin-1 encoded.

No, it does not have to be Latin-1. See http://stackoverflow.com/a/818188/95735


I forked @benmao's gist at https://gist.github.com/2584020 , changing it to use the new `CookieJar` API.

```
[shivaram@good-fortune ~/workspace/requests]$ python ./benmao2.py 
urllib2 and cookielib test: True
requests test: True
```


Uh, sec, I did something wrong.


OK, the main problem was that the test case was querying two different URLs. urllib2 was querying `requests.sinaapp.com` and Requests was querying `request.sinaapp.com`.

Here is my corrected fork of the test: https://gist.github.com/2584138

Here is the output:

```
[shivaram@good-fortune ~/workspace/requests]$ python ./benmao.py 
urllib2 and cookielib test: True
requests.session test: True
```


Anyway, I think this is probably closeable, although @piotr-dobrogost and @rwzehner may want to weigh in on the standards-compliance issues. Basically, for better or worse, cookie parsing is now being deferred directly to `cookielib`.


+1


@sashahart 

Thanks for dropping in here and sharing your knowledge with us. It's good to have cookie expert on board.

As to the backslash I'm not done yet :) - I've just posted to ongoing thread at https://github.com/sashahart/cookies/issues/1

> As author of cookies.py and a close follower of the thrilling world of RFC cookie value syntax, I think letting cookielib handle it is a very sound choice for code focused on the client side of HTTP.

I think the main reason cookielib is used is because it implements cookie jars. However as you know its parsing is not perfect - that's why you created Cookies. The perfect solution would be to use your Cookies and write implementation of cookie jars.