seomoz/reppy — pull request #14 by jmcarp (2014-07-10 04:44:47+00:00)
Title: Detect content encoding

This fixes a subtle issue with string encoding in `Rules::parse`. If `content` is a `bytes` type and starts with the UTF-8 or UTF-16 BOM, it's decoded to unicode correctly. But if it's just a plain `bytes` without a BOM, `parse` breaks in python 3.x. This PR falls back to `chardet` to guess the string encoding if no BOM is found and adds a regression test (`test_ascii`).

This is branched from https://github.com/seomoz/reppy/pull/12 to preserve other python 3 compatibility fixes and travis support, so the diffs include all those changes as well.