scrapy/scrapy — pull request #896 by kmike (2014-09-22 18:14:42+00:00)
Title: [MRG] process robots.txt once

Currently RobotsTxtMiddleware processes requests to robots.txt it is sending. If an initial request was a redirect then 2 requests to robots.txt are sent. To fix that I've added `dont_obey_robotstxt` Request.meta flag; initially it was private, but as it can be useful for users it is made public. Also, an unused `self._spider_netlocs` set is deleted.