codeinthehole/purl — pull request #12 by st4lk (2013-05-23 07:46:18+00:00)
Title: decode all urls to ascii

`urlencode.parse_qs` will return bad value, when input url is a unicode instance and contains encoded non-ascii data. Problem described in details here: [http://www.lexev.org/en/2013/parse-url-which-chontains-unicode-query-using-urlp/](http://www.lexev.org/en/2013/parse-url-which-chontains-unicode-query-using-urlp/)

Tests are attached.

In short:

>>> import urlparse, urllib
>>> value = urllib.quote(u'значение'.encode('utf8'))
>>> query = u"key=%s" % value
>>> query
u'key=%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5'
>>> query_dict = urlparse.parse_qs(query)
>>> query_dict
{u'key': [u'\xd0\xb7\xd0\xbd\xd0\xb0\xd1\x87\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5']}
>>> urllib.urlencode(query_dict, doseq=True)
Traceback (most recent call last):
File "", line 1, in
File "../urllib.py", line 1337, in urlencode
l.append(k + '=' + quote_plus(str(elt)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)

But, when construct query without **u**, it works fine:

>>> query = "key=%s" % value
>>> query_dict = urlparse.parse_qs(query)
>>> query_dict
{'key': ['\xd0\xb7\xd0\xbd\xd0\xb0\xd1\x87\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5']}
>>> urllib.urlencode(query_dict, doseq=True)
'key=%D0%B7%D0%BD%D0%B0%D1%87%D0%B5%D0%BD%D0%B8%D0%B5'