RobotsTxtMiddleware: remove unused attribute
dont_obey_robotstxt meta key; don't process requests to /robots.txt
Merge pull request #896 from scrapy/robotstxt-once

[MRG] process robots.txt once