scrapy/scrapy — pull request #97 by dangra (2012-02-28 16:50:01+00:00)
Title: W3lib encoding

After merging some changes to w3lib (scrapy/w3lib#3), and with current changes in this pull request Scrapy is able to pass all response encoding test cases that were in codebase before porting to w3lib.encoding.

Current code is a bit more complicate than expected because it tries to detect encoding without decoding body if not necessary. That's why detection can't be passed directly to `w3lib.encoding.html_to_unicode`. The counterpart is that logic for encoding detection is duplicated in `TextResponse` class and in `w3lib.encoding`.

I am opening this pull request to log our reviews and further ideas on how to improve and simplify w3lib integration.