add the RobotsTxt filter to the spider