Repository: scrapy/scrapy

Issue:
Using separate user agent for robots.txt
The current implementation of `RobotsTxtMiddleware` ([link](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/robotstxt.py)) uses `USER_AGENT` setting and `User-Agent` header of the request to determine the user agent to use for matching in `robots.txt`. There is no easy way for a user to send one user-agent header value but have a different user agent string evaluated to decide which pages to crawl. This use case seems to be more common that we thought previously for example - while [Googlebot](https://support.google.com/webmasters/answer/182072?hl=en) uses `"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"`  as `User-Agent` header of the request, it actually uses user agent `"Googlebot"` in `robots.txt` for deciding which pages to crawl. [Here is the an article describing this.](https://support.google.com/webmasters/answer/1061943?hl=en) The same seems to be the case with Bing - [link](https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0). Infact, Google made this distinction very clear in their proposed [robots.txt specification](https://github.com/google/robotstxt/blob/master/protocol-draft/draft-rep-wg-topic-00.txt) where they described the user agent used in `robots.txt` as the "product token".

The proposed solutions are implementing a new `ROBOTSTXT_USER_AGENT` setting, implementing a `robotstxt_user_agent` meta key for overriding user agent per request or both. Here is a thread that has previous discussion on the topic - https://github.com/scrapy/scrapy/pull/3796#discussion_r300036852.

I'm creating this issue as a place for discussion regarding whether to add this functionality into Scrapy? If yes, which route we should take?



Discussion:
I think `ROBOTSTXT_USER_AGENT` makes sense, falling back to the current behavior when undefined.

As for the meta key, although I proposed it, I’m not sure it’s worth it.

Resolving patch:
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -26,6 +26,7 @@ def __init__(self, crawler):
         if not crawler.settings.getbool('ROBOTSTXT_OBEY'):
             raise NotConfigured
         self._default_useragent = crawler.settings.get('USER_AGENT', 'Scrapy')
+        self._robotstxt_useragent = crawler.settings.get('ROBOTSTXT_USER_AGENT', None)
         self.crawler = crawler
         self._parsers = {}
         self._parserimpl = load_object(crawler.settings.get('ROBOTSTXT_PARSER'))
@@ -47,7 +48,10 @@ def process_request(self, request, spider):
     def process_request_2(self, rp, request, spider):
         if rp is None:
             return
-        useragent = request.headers.get(b'User-Agent', self._default_useragent)
+
+        useragent = self._robotstxt_useragent
+        if not useragent:
+            useragent = request.headers.get(b'User-Agent', self._default_useragent)
         if not rp.allowed(request.url, useragent):
             logger.debug("Forbidden by robots.txt: %(request)s",
                          {'request': request}, extra={'spider': spider})
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -246,6 +246,7 @@
 
 ROBOTSTXT_OBEY = False
 ROBOTSTXT_PARSER = 'scrapy.robotstxt.PythonRobotParser'
+ROBOTSTXT_USER_AGENT = None
 
 SCHEDULER = 'scrapy.core.scheduler.Scheduler'
 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleLifoDiskQueue'


Test patch:
diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -164,6 +164,15 @@ def ignore_request(request, spider):
         d.addCallback(lambda _: self.assertFalse(mw_module_logger.error.called))
         return d
 
+    def test_robotstxt_user_agent_setting(self):
+        crawler = self._get_successful_crawler()
+        crawler.settings.set('ROBOTSTXT_USER_AGENT', 'Examplebot')
+        crawler.settings.set('USER_AGENT', 'Mozilla/5.0 (X11; Linux x86_64)')
+        middleware = RobotsTxtMiddleware(crawler)
+        rp = mock.MagicMock(return_value=True)
+        middleware.process_request_2(rp, Request('http://site.local/allowed'), None)
+        rp.allowed.assert_called_once_with('http://site.local/allowed', 'Examplebot')
+
     def assertNotIgnored(self, request, middleware):
         spider = None  # not actually used
         dfd = maybeDeferred(middleware.process_request, request, spider)