rgarcia/dochub — pull request #41 by keunwoo (2013-11-18 22:04:00+00:00)
Title: Fix mdn-css scraper for current MDN wiki format

The css-mdn scraper had stopped functioning some time ago against current MDN, producing empty output; see for example Issue #38. This CL fixes page scraping so that it extracts content successfully again.

Some improvements have been rolled in:
+ rewrites heading levels to better fit in dochub's page structure (MDN was using h2 for page titles)
+ rewrites relative a href and img src references (most MDN internal links are now host-relative)

Known deficiencies:
- Some MDN wiki pages have meaningful content before the first section, for example [animation-delay](https://developer.mozilla.org/en-US/docs/Web/CSS/animation-delay) or [-moz-box-flex](https://developer.mozilla.org/en-US/docs/Web/CSS/-moz-box-flex). This content is not currently captured at all. We should add a hack that stuffs all the content before the first section header into a headless section, but you would also need to write some logic to scrub the page's table of contents and nav breadcrumbs first.
- There are lots of dangling internal links scattered around. This can probably be fixed by accumulating all the page titles into a set and doing the HTML postprocessing as a second pass instead of doing it online. We will probably also want to have some blacklisting or other special treatment for certain subhierarchies, like .../docs/Web/CSS/Tools/* or .../docs/Web/CSS/Tutorials
- Clicking on internal links changes the view, but it does not scroll the main content pane to the target appropriately.

These deficiencies are not really introduced in this CL, but they are made more visible by it. In any case, this CL is a first step towards getting the CSS crawler working again. The problems above can be filed as issues and addressed by followup CLs.

Regenerated static data is not included in this CL. See keunwoo/dochub@14d951aaf8a512db302e8224794df5e21ae7f0c6 for sample static data regenerated from a scrape I ran this afternoon.