rg3/youtube-dl — pull request #329 by FiloSottile (2012-04-10 21:37:47+00:00)
Title: So long lxml, and thanks for all the fish

There was a lot of confusion over HTML and XML parsing:

* `_unescapeHTML` did the same thing as `htmlentity_transform` but using the *undocumented* `HTMLParser.unescape`
* sometimes `_unescapeHTML` was used, sometimes the complex to read syntax of `htmlentity_transform` was used instead
* we depended on `lxml` (non public domain, MIT licensed) only to do what in Javascript is getElementById
* there wasn't a consistent way to handle HTML to make it human-readable

So here are my changes:

* now we have `get_element_by_id` that uses a custom `HTMLParser` to extract the HTML tag with a specific id, so *lxml is gone*
* `_unescapeHTML` now uses `htmlentity_transform` and I applied (and tested) it across the code
* `clean_html` does the tag-stripping and newline-handling needed to get a pretty text out of a bunch of HTML

Have a look at `./youtube-dl --get-description http://www.youtube.com/watch?v=BQ80NtYmktQ`

Before (without lxml):

```
Il nuovo pezzo di Mattia Sterpi Deejay Seguici su Facebook https://www.facebook.com/pages/Mattia-Sterpi-Deejay/121854024543063 e Twitter https://twitter.com/...
```

Before (with lxml):

```
Il nuovo pezzo di Mattia Sterpi DeejaySeguici su Facebook https://www.facebook.com/pages/Mattia-Sterpi-Deejay/121854024543063 e Twitter https://twitter.com/DjSterpiCompra il singolo su iTunes:http://itunes.apple.com/it/album/heart-of-mine-single/id494530985?ls=1Pubblicato: 21/01/2012℗ 2012 Il Chiosco
```

Now:

```
Il nuovo pezzo di Mattia Sterpi Deejay

Seguici su Facebook https://www.facebook.com/pages/Mattia-Sterpi-Deejay/121854024543063 e Twitter https://twitter.com/DjSterpi

Compra il singolo su iTunes:
http://itunes.apple.com/it/album/heart-of-mine-single/id494530985?ls=1

Pubblicato: 21/01/2012
℗ 2012 Il Chiosco
```