Handle bytes content without encoding; add regression test.

Resubmission of https://github.com/seomoz/reppy/pull/14, rebased off master.
Actually, I just realized that you're using chardet to sniff the encoding. I'm not sure that's a great idea. `robots.txt` files should be in ASCII since the URLs inside them are supposed to be URL encoded and relative paths.

The 1996 RFC for robots.txt specifically says the encoding should be ASCII. Google says that they do accept UTF-8 encoded characters, but in general, I don't think we want to support arbitrary encodings, and using chardet could be a large overhead compared to just assuming ASCII in these exotic cases.

Do you have a specific example of a `robots.txt` file that does not parse as ASCII or UTF-8?

I don't believe we've encountered such files in the wild, and we crawl quite a few sites using reppy right now.