Added a configuration option to ignore robots.txt when that file
disallows a seed URL. When setIgnoreRobotsTxtForSeed is set to true, any
seed URL will be crawled regardless of robots.txt configuration.
However, should this page redirect, the robots.txt on the new location
will still be respected.