Domain processes urls, RobotsTxt limited evaluation of disallows