No dups

add a set to enforce with an exception not crawling the same link twice. this should log the etag and rss feed, so we can try to reproduce the bug.