rails/rails — pull request #3789 by benmmurphy (2011-11-28 22:09:21+00:00)
Title: Prevent parameters and uploaded filenames with invalid UTF-8 bytes from entering into rails.

Strings with invalid encodings can cause unexpected exceptions to occur in ruby. For example any regex that is run on a string containing invalid UTF-8 bytes will cause an exception. When these strings are passed to external systems which don't correctly handle invalid UTF-8 byte sequences it can cause unexpected behaviour.

Unfortunately, this commit breaks the previous handling of multipart/mixed files. The previous behaviour was to expose the whole of the unparsed content in a single parameter. Because the multipart/mixed parameter whose proper encoding is binary is indistinguishable from other parameters whose proper encoding is UTF-8 breakage is unavoidable. If we want to support multipart/mixed properly we need rack to do the parsing.

Also, because of the presence of the unicode tick on most requests this change will prevent any system sending a non-utf8 encoding of this character from interacting with the web application. It may be worth changing the check to happen on specific parameter access rather than check all parameters when a single parameter has been accessed. Or alternatively treating the tick field in a special way. At least then a client without proper UTF-8 support will mostly work until they need to send non-ascii characters.