Pylons/webob — Issue #16: Request.charset defaults to UTF-8
Author: bquinlan-google
Created: 2011-12-14 03:11:51+00:00
State: open

Request.charset defaults to UTF-8 if no charset is specified. This seems like incorrect behavior.

See http://www.ietf.org/rfc/rfc2854.txt,:
"""
6. Charset default rules

The use of an explicit charset parameter is strongly recommended.
While [MIME] specifies "The default character set, which must be
assumed in the absence of a charset parameter, is US-ASCII." [HTTP]
Section 3.7.1, defines that "media subtypes of the 'text' type are
defined to have a default charset value of 'ISO-8859-1'". Section
19.3 of [HTTP] gives additional guidelines. Using an explicit
charset parameter will help avoid confusion.

Using an explicit charset parameter also takes into account that the
overwhelming majority of deployed browsers are set to use something
else than 'ISO-8859-1' as the default; the actual default is either a
corporate character encoding or character encodings widely deployed
in a certain national or regional community. For further
considerations, please also see Section 5.2 of [HTML40].
"""

It might be better to refuse to guess if no explicit charset is provided and let the client code deal with it.