ParserBolt conditionally fetches robots.txt