scrapy/scrapy — pull request #756 by bijzz (2014-06-22 20:07:30+00:00)
Title: Changed feed to HTMLParser to unicode instead of str.

A lot of sites yielded `UnicodeDecodeError` when using `HtmlParserLinkExtractor().extract_links(response)`. When the `HTMLParser` receives the `response.body` as `unicode` the exceptions dissappear. Maybe you can still replicate this with one or the other url i posted on [stackedit](https://stackoverflow.com/questions/24351023/scrapy-linkextractors-fail) (but these might work on your system depending on the system default encoding).

Also have a look at the HTMLParser documentation in Python 2 [docs.python.org/2.7](https://docs.python.org/2.7/library/htmlparser.html?highlight=parser#HTMLParser.HTMLParser.feed) stating `data can be either unicode or str, but passing unicode is advised.`.