gt-big-data/retina-crawler — pull request #21 by pviolette3 (2014-11-01 19:05:56+00:00)
Title: Deduplicate articles in the database

Every time an article is updated in CNN, CNN publishes an RSS entry for each previous version of the article. This caused us to insert many duplicate entries into our database.This diff should fix our duplication problems.

The first problem is an RSS feed duplicates articles by posting an entry for each update of an article. This fix to RSSFeedParser should handle this better.

The second problem is making sure we don't insert new entries for URLs we have already seen in a previous crawl. Now, in MongoDB if we receive an updated article that already exists in our database (whose URL we've already crawled), we update a history of the article instead of creating a new one. This way, all articles with the same URL only get 1 entry in the database.

This diff will change slightly the API between crawler and clusterer. @iRapha @mgruchacz you will want to look at recent_download_date and recent_pub_date instead of download_date and pub_date, to take into account that we can have several versions of the same article.