gt-big-data/retina-crawler — pull request #18 by pviolette3 (2014-10-30 01:42:15+00:00)
Title: No dups

add a set to enforce with an exception not crawling the same link twice. this should log the etag and rss feed, so we can try to reproduce the bug.