robotstxt mid multi spider support removed
Merge pull request #419 from nramirezuy/robotstxt-multispider

robotstxt mid multi spider support removed