cgiffard/node-simplecrawler — pull request #74 by tobli (2014-05-07 08:27:28+00:00)
Title: Add config option to follow initial redirects to a different domain.

If crawler.allowInitialDomainChange is set to true, and the initial url redirects to a different domain (e.g. a .com redirects to a canonical .net domain), crawler.host is updated to the new domain, to allow crawling to continue on that domain.