ecprice/newsdiffs — pull request #17 by mherdeg (2013-06-11 23:41:31+00:00)
Title: Add support for scraping "section fronts" in addition to the publication's home page

Plenty of NYT content never gets posted on the http://nytimes.com/ front page and is not automatically flagged for tracking. This content is lower profile relative to its front-page peers but might well be *more* susceptible to silent changes because, hey, who's going to notice?

This diff adds an optional extra parser setting, `feeder_pages`, which is a list of URLs that should be scraped in addition to a publication's home page when looking for new articles to index.

This diff also adds about 30 new "section front" pages for the NYT. Experimentally, parsing time to grab new URLs goes up by a factor of about 10 (from ~3 seconds to ~35 seconds); and the number of post-canonicalization generated URLs goes up by a factor of about 10 (from ~100 to ~800).