gt-big-data/retina-crawler — pull request #25 by supersam654 (2014-11-06 16:35:09+00:00)
Title: No newspaper

Completely removed Newspaper from the project.

Newspaper was really nice for getting us started, but it's not perfect and that's causing some major issues for everyone trying to use our data.

Instead, this code manually finds all relevant information from articles on CNN and NYTimes and properly fills out the article class. This replaces Newspaper's pretty good parsing + our very hacky "extra" parsing code.

Basically all of the changes work with the underlying libraries that Newspaper was using (requests and lxml). Therefore, there are no new dependencies.