symfony/symfony — pull request #7411 by mpdude (2013-03-18 11:39:55+00:00)
Title: Invalid multibyte sequences in request URL causes errors

__The simple/obvious problem__

When an URI like "/foo/%A0" is requested, the urlencoded "%A0" gets decoded before it shows up in PHP. Thus, the request uri contains a single byte value > 128 that is not valid in multibyte character encodings.

Symfony's Matcher might want to forward to the same address with a trailing slash added and creates a RedirectResponse. Internally, htmlspecialchars() is called which stumbles upon the invalid multibyte sequence (as it always tries to generate an UTF-8 response body) and emits a PHP warning.

__The more fundamental look__

The HTTP spec does not (at least I haven't found it until now; RFC 3986?) say anything regarding the character encoding of URLs in the HTTP *request*. For the response, the Content-Type header gives the encoding of the body, but there seems to be no such thing for the request.

That's a problem as [it does not make sense to have a string without knowing what encoding it uses](http://www.joelonsoftware.com/articles/Unicode.html).

Browsers seem to use the encoding of the current page for the URL when following a link from that page. That is, when an URL with a non-ascii (think of German umlauts) charcter is picked from a latin1 page, the request URL will be in latin1; when the same link appears on an utf-8 encoded page, the request is sent with a multibyte encoded character in that place.

This becomes a problem when the link is forwarded by other means - search engines, e-mail, messaging. In that case, the link-receiving and requesting browser might only see an umlaut (for example) in the URL but has no idea whether to request it with an latin1 or utf-8 encoded URL. Note that percent-encoding does not matter here as it only avoids sending bytes outside ASCII, but depending on the decision for latin1/utf-8, the percent-encoded URL will contain one, two (or even more) %.. sequences for every character.

In consequence, you can not be sure which encoding your request data (parameters, pathinfo, ...) will be in.

__Fixing this in the HttpFoundation?__

The HttpFoundation might be able to address this issue as it might apply magic (mb_detect_encoding() comes to mind) to figure out the encoding the requesting client used and convert the request to a desired (framework.charset?) encoding.

Thoughts?