grangier/python-goose — pull request #151 by devkeh (2014-10-04 16:21:54+00:00)
Title: Fix issue with html pages containing <?xml> declarations

I had experienced a problem with html pages that started with

It seems the lxml.html parser is not able to deal with this properly as noted here. https://stackoverflow.com/questions/15302125/html-encoding-and-lxml-parsing

I have inserted a regex substitution to remove the tag from the raw html and the problem is now resolved.

```
Traceback (most recent call last):
File "", line 1, in
File "goose/__init__.py", line 56, in extract
return self.crawl(cc)
File "goose/__init__.py", line 63, in crawl
article = crawler.crawl(crawl_candiate)
File "goose/crawler.py", line 90, in crawl
doc = self.get_document(raw_html)
File "goose/crawler.py", line 176, in get_document
doc = self.parser.fromstring(raw_html)
File "goose/parsers.py", line 54, in fromstring
self.doc = lxml.html.fromstring(html)
File "/home/jeff/.virtualenvs/python-goose/lib/python2.7/site-packages/lxml/html/__init__.py", line 723, in fromstring
doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
File "/home/jeff/.virtualenvs/python-goose/lib/python2.7/site-packages/lxml/html/__init__.py", line 613, in document_fromstring
value = etree.fromstring(html, parser, **kw)
File "lxml.etree.pyx", line 3092, in lxml.etree.fromstring (src/lxml/lxml.etree.c:70691)
File "parser.pxi", line 1823, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:106654)
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
```