csyork/csbot — pull request #39 by alanbriolat (2012-12-04 15:49:20+00:00)
Title: LinkInfo: more Unicode shenanigans

The current use of the "requests"-decoded content and ``lxml.html.document_fromstring`` still has quite a few holes. One of these is handling cases where the only encoding indications are in ```` tags. A good example to test with is any Python module documentation, e.g. http://docs.python.org/library/urlparse.html. There are two em-dashes in the ``

``, the first is a UTF-8 literal, the second is an HTML entity.

```python
>>> r = requests.get('http://docs.python.org/library/urlparse.html')
>>> lxml.html.document_fromstring(r.content).find('.//title').text
u'20.16. urlparse \u2014 Parse URLs into components \u2014 Python v2.7.3 documentation'
>>> lxml.html.document_fromstring(r.text).find('.//title').text
u'20.16. urlparse \xe2\x80\x94 Parse URLs into components \u2014 Python v2.7.3 documentation'
```

lxml obviously does the correct thing when it has to do the decoding itself, because it is aware of the ``<meta>`` tags. Unfortunately, if we always passed ``r.content`` and the only indication of content type is in the HTTP Content-Type header, we'll still get it wrong in other cases.

Maybe it's worth checking to see what always using ``lxml.html.parse(url)`` would give us. Really we need a sensible way to unit test this against known HTTP responses.