Pylons/webob — Issue #24: Issue with unicode post parameters?
Author: freeekanayaka
Created: 2012-01-20 09:44:33+00:00
State: open

I'm not sure this is a real issue, but while using the WSGI version of zope.testbrowser to post a form with a unicode field I got this with Python 2.7:

File "webob/request.py", line 1411, in readinto
data = bytes_(url_encode(self.vars), 'utf8')
File "/usr/lib/python2.7/urllib.py", line 1311, in urlencode
v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc9' in position 13: ordinal not in range(128)

The test browser code is something like:

input = browser.getControl(name="some-form-input")
input.value = u"1-001A1B2C3D4\xc95F6".encode("utf-8")
browser.getControl(name="some-form-button").click()

It turns out that Request.POST does the following:

else:
fs = cgi.FieldStorage(
fp=self.body_file,
environ=fs_environ,
keep_blank_values=True)
vars = MultiDict.from_fieldstorage(fs)

ctype = self._content_type_raw or 'application/x-www-form-urlencoded'
f = FakeCGIBody(vars, ctype)

where FieldStorage holds str values but MultiDict converts them to unicode, so now vars has unicode values. But FakeCGIBody.readinto then eventually runs:

if self.content_type.startswith(
'application/x-www-form-urlencoded'):
# TODO: check if bytes_ is necessary
data = bytes_(url_encode(self.vars), 'utf8')

where url_encode is urllib.urlencode, which breaks with unicode values.

Any take on this?

Thanks!