stewartmckee/cobweb — pull request #9 by rojotek (2012-07-17 00:34:15+00:00)
Title: Improved handling of redirects

Stew,

I made another tweak to crawl_job to help with redirects.

I moved the url from the queued list to the crawled list - for both the original url, and the content url (to handle redirects).