stewartmckee/cobweb — pull request #12 by rojotek (2012-08-25 08:03:43+00:00)
Title: only try and crawl links on a page 1 time.

Hey Stew,

A minor pull request for you -- it does a uniq on the list of internal links to try and prevent pages from being added to the crawl set more than once.

Rob