LuminosoInsight/python-ftfy — pull request #16 by rspeer (2013-11-27 08:22:40+00:00)
Title: Proposed v3.1: adds bad_codecs and guess_bytes

`bad_codecs` is a new module for decoding text in encodings that Python would not otherwise deign to decode. Sloppy codecs include "utf-8-variants", the results of UTF-8 implementations that don't quite get it right, and "sloppy codecs", in which unassigned bytes fall through to Latin-1. Using this module, the sloppy decoding logic is moved out of the main body of the code, and it's also made available on its own through Python's codecs system.

UTF-8 variants such as CESU-8 or "Java modified UTF-8", and the Windows-1252 variant we call "sloppy-windows-1252", are bad codecs that are very popular on the Web.

This also adds a "guess_bytes" function, which applies a simple strategy to distinguish which of a few common encodings bytes are in. This improves the implementation command-line tool.