put spider_options and spider_info in robots()