Repository: scrapy/scrapy

Issue:
Using separate user agent for robots.txt
The current implementation of `RobotsTxtMiddleware` ([link](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/robotstxt.py)) uses `USER_AGENT` setting and `User-Agent` header of the request to determine the user agent to use for matching in `robots.txt`. There is no easy way for a user to send one user-agent header value but have a different user agent string evaluated to decide which pages to crawl. This use case seems to be more common that we thought previously for example - while [Googlebot](https://support.google.com/webmasters/answer/182072?hl=en) uses `"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"`  as `User-Agent` header of the request, it actually uses user agent `"Googlebot"` in `robots.txt` for deciding which pages to crawl. [Here is the an article describing this.](https://support.google.com/webmasters/answer/1061943?hl=en) The same seems to be the case with Bing - [link](https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0). Infact, Google made this distinction very clear in their proposed [robots.txt specification](https://github.com/google/robotstxt/blob/master/protocol-draft/draft-rep-wg-topic-00.txt) where they described the user agent used in `robots.txt` as the "product token".

The proposed solutions are implementing a new `ROBOTSTXT_USER_AGENT` setting, implementing a `robotstxt_user_agent` meta key for overriding user agent per request or both. Here is a thread that has previous discussion on the topic - https://github.com/scrapy/scrapy/pull/3796#discussion_r300036852.

I'm creating this issue as a place for discussion regarding whether to add this functionality into Scrapy? If yes, which route we should take?


Discussion:

Patch:
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 616b56101..93c91f18a 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -989,6 +989,12 @@ RobotsTxtMiddleware
     To make sure Scrapy respects robots.txt make sure the middleware is enabled
     and the :setting:`ROBOTSTXT_OBEY` setting is enabled.
 
+    The :setting:`ROBOTSTXT_USER_AGENT` setting can be used to specify the
+    user agent string to use for matching in the robots.txt_ file. If it
+    is ``None``, the User-Agent header you are sending with the request or the
+    :setting:`USER_AGENT` setting (in that order) will be used for determining
+    the user agent to use in the robots.txt_ file.
+
     This middleware has to be combined with a robots.txt_ parser.
 
     Scrapy ships with support for the following robots.txt_ parsers:
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 12606fe47..d3ad777f2 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1151,6 +1151,18 @@ Default: ``'scrapy.robotstxt.PythonRobotParser'``
 The parser backend to use for parsing ``robots.txt`` files. For more information see
 :ref:`topics-dlmw-robots`.
 
+.. setting:: ROBOTSTXT_USER_AGENT
+
+ROBOTSTXT_USER_AGENT
+^^^^^^^^^^^^^^^^^^^^
+
+Default: ``None``
+
+The user agent string to use for matching in the robots.txt file. If ``None``,
+the User-Agent header you are sending with the request or the
+:setting:`USER_AGENT` setting (in that order) will be used for determining
+the user agent to use in the robots.txt file.
+
 .. setting:: SCHEDULER
 
 SCHEDULER
@@ -1409,7 +1421,10 @@ USER_AGENT
 
 Default: ``"Scrapy/VERSION (+https://scrapy.org)"``
 
-The default User-Agent to use when crawling, unless overridden.
+The default User-Agent to use when crawling, unless overridden. This user agent is
+also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
+if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and
+there is no overridding User-Agent header specified for the request.
 
 
 Settings documented elsewhere:
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
index c5a60d355..6a5dfb79c 100644
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -26,6 +26,7 @@ class RobotsTxtMiddleware(object):
         if not crawler.settings.getbool('ROBOTSTXT_OBEY'):
             raise NotConfigured
         self._default_useragent = crawler.settings.get('USER_AGENT', 'Scrapy')
+        self._robotstxt_useragent = crawler.settings.get('ROBOTSTXT_USER_AGENT', None)
         self.crawler = crawler
         self._parsers = {}
         self._parserimpl = load_object(crawler.settings.get('ROBOTSTXT_PARSER'))
@@ -47,7 +48,10 @@ class RobotsTxtMiddleware(object):
     def process_request_2(self, rp, request, spider):
         if rp is None:
             return
-        useragent = request.headers.get(b'User-Agent', self._default_useragent)
+
+        useragent = self._robotstxt_useragent
+        if not useragent:
+            useragent = request.headers.get(b'User-Agent', self._default_useragent)
         if not rp.allowed(request.url, useragent):
             logger.debug("Forbidden by robots.txt: %(request)s",
                          {'request': request}, extra={'spider': spider})
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 81fee543f..efb257e2b 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -246,6 +246,7 @@ RETRY_PRIORITY_ADJUST = -1
 
 ROBOTSTXT_OBEY = False
 ROBOTSTXT_PARSER = 'scrapy.robotstxt.PythonRobotParser'
+ROBOTSTXT_USER_AGENT = None
 
 SCHEDULER = 'scrapy.core.scheduler.Scheduler'
 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleLifoDiskQueue'


Test patch:
diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
index 79f172848..fbc46cba4 100644
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -164,6 +164,15 @@ Disallow: /some/randome/page.html
         d.addCallback(lambda _: self.assertFalse(mw_module_logger.error.called))
         return d
 
+    def test_robotstxt_user_agent_setting(self):
+        crawler = self._get_successful_crawler()
+        crawler.settings.set('ROBOTSTXT_USER_AGENT', 'Examplebot')
+        crawler.settings.set('USER_AGENT', 'Mozilla/5.0 (X11; Linux x86_64)')
+        middleware = RobotsTxtMiddleware(crawler)
+        rp = mock.MagicMock(return_value=True)
+        middleware.process_request_2(rp, Request('http://site.local/allowed'), None)
+        rp.allowed.assert_called_once_with('http://site.local/allowed', 'Examplebot')
+
     def assertNotIgnored(self, request, middleware):
         spider = None  # not actually used
         dfd = maybeDeferred(middleware.process_request, request, spider)