Repository: scrapy/scrapy
Pull request #3982 (2019-08-28 08:19:14)

Fix JSONRequest naming
Change `JSONRequest` to `JsonRequest` to follow the same format/style as the other JSON classes (`JsonLinesItemExporter`, `JsonItemExporter` and `JsonWriterPipeline`) and the other response/request classes (`HtmlResponse`, `XmlResponse`, `XmlRpcRequest`)

fixes:
* issue: https://github.com/scrapy/scrapy/issues/3929
* old PR: https://github.com/scrapy/scrapy/pull/3930


Patch:
diff --git a/docs/conf.py b/docs/conf.py
index f49f79cd5..eba416cd6 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -252,6 +252,10 @@ coverage_ignore_pyobjects = [
 
     # Private exception used by the command-line interface implementation.
     r'^scrapy\.exceptions\.UsageError',
+
+    # Methods of BaseItemExporter subclasses are only documented in
+    # BaseItemExporter.
+    r'^scrapy\.exporters\.(?!BaseItemExporter\b)\w*?\.',
 ]
 
 
diff --git a/docs/news.rst b/docs/news.rst
index ce5b8b406..aac750601 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -80,8 +80,8 @@ New features
     provides a cleaner way to pass keyword arguments to callback methods
     (:issue:`1138`, :issue:`3563`)
 
-*   A new :class:`~scrapy.http.JSONRequest` class offers a more convenient way
-    to build JSON requests (:issue:`3504`, :issue:`3505`)
+*   A new :class:`JSONRequest <scrapy.http.JsonRequest>` class offers a more
+    convenient way to build JSON requests (:issue:`3504`, :issue:`3505`)
 
 *   A ``process_request`` callback passed to the :class:`~scrapy.spiders.Rule`
     constructor now receives the :class:`~scrapy.http.Response` object that
@@ -1269,8 +1269,8 @@ This 1.1 release brings a lot of interesting features and bug fixes:
     this behavior, update :setting:`ROBOTSTXT_OBEY` in ``settings.py`` file
     after creating a new project.
   - Exporters now work on unicode, instead of bytes by default (:issue:`1080`).
-    If you use ``PythonItemExporter``, you may want to update your code to
-    disable binary mode which is now deprecated.
+    If you use :class:`~scrapy.exporters.PythonItemExporter`, you may want to
+    update your code to disable binary mode which is now deprecated.
   - Accept XML node names containing dots as valid (:issue:`1533`).
   - When uploading files or images to S3 (with ``FilesPipeline`` or
     ``ImagesPipeline``), the default ACL policy is now "private" instead
@@ -1408,8 +1408,8 @@ Bugfixes
 - Fixed bug on ``XMLItemExporter`` with non-string fields in
   items (:issue:`1738`).
 - Fixed startproject command in OS X (:issue:`1635`).
-- Fixed PythonItemExporter and CSVExporter for non-string item
-  types (:issue:`1737`).
+- Fixed :class:`~scrapy.exporters.PythonItemExporter` and CSVExporter for
+  non-string item types (:issue:`1737`).
 - Various logging related fixes (:issue:`1294`, :issue:`1419`, :issue:`1263`,
   :issue:`1624`, :issue:`1654`, :issue:`1722`, :issue:`1726` and :issue:`1303`).
 - Fixed bug in ``utils.template.render_templatefile()`` (:issue:`1212`).
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 6aa714fb2..c25e12bda 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -989,6 +989,12 @@ RobotsTxtMiddleware
     To make sure Scrapy respects robots.txt make sure the middleware is enabled
     and the :setting:`ROBOTSTXT_OBEY` setting is enabled.
 
+    The :setting:`ROBOTSTXT_USER_AGENT` setting can be used to specify the
+    user agent string to use for matching in the robots.txt_ file. If it
+    is ``None``, the User-Agent header you are sending with the request or the
+    :setting:`USER_AGENT` setting (in that order) will be used for determining
+    the user agent to use in the robots.txt_ file.
+
     This middleware has to be combined with a robots.txt_ parser.
 
     Scrapy ships with support for the following robots.txt_ parsers:
diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst
index f5048d2da..11b3045ec 100644
--- a/docs/topics/exporters.rst
+++ b/docs/topics/exporters.rst
@@ -165,9 +165,9 @@ BaseItemExporter
       value unchanged except for ``unicode`` values which are encoded to
       ``str`` using the encoding declared in the :attr:`encoding` attribute.
 
-      :param field: the field being serialized. If a raw dict is being 
+      :param field: the field being serialized. If a raw dict is being
           exported (not :class:`~.Item`) *field* value is an empty dict.
-      :type field: :class:`~scrapy.item.Field` object or an empty dict 
+      :type field: :class:`~scrapy.item.Field` object or an empty dict
 
       :param name: the name of the field being serialized
       :type name: str
@@ -223,6 +223,12 @@ BaseItemExporter
       * ``indent<=0`` each item on its own line, no indentation
       * ``indent>0`` each item on its own line, indented with the provided numeric value
 
+PythonItemExporter
+------------------
+
+.. autoclass:: PythonItemExporter
+
+
 .. highlight:: none
 
 XmlItemExporter
@@ -410,3 +416,8 @@ JsonLinesItemExporter
    this exporter is well suited for serializing large amounts of data.
 
 .. _JSONEncoder: https://docs.python.org/2/library/json.html#json.JSONEncoder
+
+MarshalItemExporter
+-------------------
+
+.. autoclass:: MarshalItemExporter
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 9a5c65b0d..ad1b9af10 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -535,19 +535,19 @@ method for this job. Here's an example spider which uses it::
 
             # continue scraping with authenticated session...
 
-JSONRequest
+JsonRequest
 -----------
 
-The JSONRequest class extends the base :class:`Request` class with functionality for
+The JsonRequest class extends the base :class:`Request` class with functionality for
 dealing with JSON requests.
 
-.. class:: JSONRequest(url, [... data, dumps_kwargs])
+.. class:: JsonRequest(url, [... data, dumps_kwargs])
 
-   The :class:`JSONRequest` class adds two new argument to the constructor. The
+   The :class:`JsonRequest` class adds two new argument to the constructor. The
    remaining arguments are the same as for the :class:`Request` class and are
    not documented here.
 
-   Using the :class:`JSONRequest` will set the ``Content-Type`` header to ``application/json``
+   Using the :class:`JsonRequest` will set the ``Content-Type`` header to ``application/json``
    and ``Accept`` header to ``application/json, text/javascript, */*; q=0.01``
 
    :param data: is any JSON serializable object that needs to be JSON encoded and assigned to body.
@@ -562,7 +562,7 @@ dealing with JSON requests.
 
 .. _json.dumps: https://docs.python.org/3/library/json.html#json.dumps
 
-JSONRequest usage example
+JsonRequest usage example
 -------------------------
 
 Sending a JSON POST request with a JSON payload::
@@ -571,7 +571,7 @@ Sending a JSON POST request with a JSON payload::
        'name1': 'value1',
        'name2': 'value2',
    }
-   yield JSONRequest(url='http://www.example.com/post/action', data=data)
+   yield JsonRequest(url='http://www.example.com/post/action', data=data)
 
 
 Response objects
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 0cb81c43e..943ba13ee 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -952,7 +952,7 @@ LOGSTATS_INTERVAL
 Default: ``60.0``
 
 The interval (in seconds) between each logging printout of the stats 
-by :class:`~extensions.logstats.LogStats`.
+by :class:`~scrapy.extensions.logstats.LogStats`.
 
 .. setting:: MEMDEBUG_ENABLED
 
@@ -1170,6 +1170,18 @@ Default: ``'scrapy.robotstxt.PythonRobotParser'``
 The parser backend to use for parsing ``robots.txt`` files. For more information see
 :ref:`topics-dlmw-robots`.
 
+.. setting:: ROBOTSTXT_USER_AGENT
+
+ROBOTSTXT_USER_AGENT
+^^^^^^^^^^^^^^^^^^^^
+
+Default: ``None``
+
+The user agent string to use for matching in the robots.txt file. If ``None``,
+the User-Agent header you are sending with the request or the
+:setting:`USER_AGENT` setting (in that order) will be used for determining
+the user agent to use in the robots.txt file.
+
 .. setting:: SCHEDULER
 
 SCHEDULER
@@ -1428,7 +1440,10 @@ USER_AGENT
 
 Default: ``"Scrapy/VERSION (+https://scrapy.org)"``
 
-The default User-Agent to use when crawling, unless overridden.
+The default User-Agent to use when crawling, unless overridden. This user agent is
+also used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
+if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and
+there is no overridding User-Agent header specified for the request.
 
 
 Settings documented elsewhere:
diff --git a/scrapy/downloadermiddlewares/robotstxt.py b/scrapy/downloadermiddlewares/robotstxt.py
index c5a60d355..6a5dfb79c 100644
--- a/scrapy/downloadermiddlewares/robotstxt.py
+++ b/scrapy/downloadermiddlewares/robotstxt.py
@@ -26,6 +26,7 @@ class RobotsTxtMiddleware(object):
         if not crawler.settings.getbool('ROBOTSTXT_OBEY'):
             raise NotConfigured
         self._default_useragent = crawler.settings.get('USER_AGENT', 'Scrapy')
+        self._robotstxt_useragent = crawler.settings.get('ROBOTSTXT_USER_AGENT', None)
         self.crawler = crawler
         self._parsers = {}
         self._parserimpl = load_object(crawler.settings.get('ROBOTSTXT_PARSER'))
@@ -47,7 +48,10 @@ class RobotsTxtMiddleware(object):
     def process_request_2(self, rp, request, spider):
         if rp is None:
             return
-        useragent = request.headers.get(b'User-Agent', self._default_useragent)
+
+        useragent = self._robotstxt_useragent
+        if not useragent:
+            useragent = request.headers.get(b'User-Agent', self._default_useragent)
         if not rp.allowed(request.url, useragent):
             logger.debug("Forbidden by robots.txt: %(request)s",
                          {'request': request}, extra={'spider': spider})
diff --git a/scrapy/exporters.py b/scrapy/exporters.py
index 695c74fec..6fc87ed18 100644
--- a/scrapy/exporters.py
+++ b/scrapy/exporters.py
@@ -276,6 +276,13 @@ class PickleItemExporter(BaseItemExporter):
 
 
 class MarshalItemExporter(BaseItemExporter):
+    """Exports items in a Python-specific binary format (see
+    :mod:`marshal`).
+
+    :param file: The file-like object to use for exporting the data. Its
+                 ``write`` method should accept :class:`bytes` (a disk file
+                 opened in binary mode, a :class:`~io.BytesIO` object, etc)
+    """
 
     def __init__(self, file, **kwargs):
         self._configure(kwargs)
@@ -297,10 +304,13 @@ class PprintItemExporter(BaseItemExporter):
 
 
 class PythonItemExporter(BaseItemExporter):
-    """The idea behind this exporter is to have a mechanism to serialize items
-    to built-in python types so any serialization library (like
-    json, msgpack, binc, etc) can be used on top of it. Its main goal is to
-    seamless support what BaseItemExporter does plus nested items.
+    """This is a base class for item exporters that extends
+    :class:`BaseItemExporter` with support for nested items.
+
+    It serializes items to built-in Python types, so that any serialization
+    library (e.g. :mod:`json` or msgpack_) can be used on top of it.
+
+    .. _msgpack: https://pypi.org/project/msgpack/
     """
     def _configure(self, options, dont_fail=False):
         self.binary = options.pop('binary', True)
diff --git a/scrapy/http/__init__.py b/scrapy/http/__init__.py
index 4b2f7b33f..e6c58e1f1 100644
--- a/scrapy/http/__init__.py
+++ b/scrapy/http/__init__.py
@@ -10,7 +10,7 @@ from scrapy.http.headers import Headers
 from scrapy.http.request import Request
 from scrapy.http.request.form import FormRequest
 from scrapy.http.request.rpc import XmlRpcRequest
-from scrapy.http.request.json_request import JSONRequest
+from scrapy.http.request.json_request import JsonRequest
 
 from scrapy.http.response import Response
 from scrapy.http.response.html import HtmlResponse
diff --git a/scrapy/http/request/json_request.py b/scrapy/http/request/json_request.py
index 8f7a61a6d..f08b25280 100644
--- a/scrapy/http/request/json_request.py
+++ b/scrapy/http/request/json_request.py
@@ -1,5 +1,5 @@
 """
-This module implements the JSONRequest class which is a more convenient class
+This module implements the JsonRequest class which is a more convenient class
 (than Request) to generate JSON Requests.
 
 See documentation in docs/topics/request-response.rst
@@ -10,9 +10,10 @@ import json
 import warnings
 
 from scrapy.http.request import Request
+from scrapy.utils.deprecate import create_deprecated_class
 
 
-class JSONRequest(Request):
+class JsonRequest(Request):
     def __init__(self, *args, **kwargs):
         dumps_kwargs = copy.deepcopy(kwargs.pop('dumps_kwargs', {}))
         dumps_kwargs.setdefault('sort_keys', True)
@@ -31,7 +32,7 @@ class JSONRequest(Request):
             if 'method' not in kwargs:
                 kwargs['method'] = 'POST'
 
-        super(JSONRequest, self).__init__(*args, **kwargs)
+        super(JsonRequest, self).__init__(*args, **kwargs)
         self.headers.setdefault('Content-Type', 'application/json')
         self.headers.setdefault('Accept', 'application/json, text/javascript, */*; q=0.01')
 
@@ -46,8 +47,11 @@ class JSONRequest(Request):
         elif not body_passed and data_passed:
             kwargs['body'] = self._dumps(data)
 
-        return super(JSONRequest, self).replace(*args, **kwargs)
+        return super(JsonRequest, self).replace(*args, **kwargs)
 
     def _dumps(self, data):
         """Convert to JSON """
         return json.dumps(data, **self._dumps_kwargs)
+
+
+JSONRequest = create_deprecated_class("JSONRequest", JsonRequest)
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 742c8e8a1..5884dfc60 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -247,6 +247,7 @@ RETRY_PRIORITY_ADJUST = -1
 
 ROBOTSTXT_OBEY = False
 ROBOTSTXT_PARSER = 'scrapy.robotstxt.PythonRobotParser'
+ROBOTSTXT_USER_AGENT = None
 
 SCHEDULER = 'scrapy.core.scheduler.Scheduler'
 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleLifoDiskQueue'


Test patch:
diff --git a/tests/test_downloadermiddleware_robotstxt.py b/tests/test_downloadermiddleware_robotstxt.py
index 79f172848..fbc46cba4 100644
--- a/tests/test_downloadermiddleware_robotstxt.py
+++ b/tests/test_downloadermiddleware_robotstxt.py
@@ -164,6 +164,15 @@ Disallow: /some/randome/page.html
         d.addCallback(lambda _: self.assertFalse(mw_module_logger.error.called))
         return d
 
+    def test_robotstxt_user_agent_setting(self):
+        crawler = self._get_successful_crawler()
+        crawler.settings.set('ROBOTSTXT_USER_AGENT', 'Examplebot')
+        crawler.settings.set('USER_AGENT', 'Mozilla/5.0 (X11; Linux x86_64)')
+        middleware = RobotsTxtMiddleware(crawler)
+        rp = mock.MagicMock(return_value=True)
+        middleware.process_request_2(rp, Request('http://site.local/allowed'), None)
+        rp.allowed.assert_called_once_with('http://site.local/allowed', 'Examplebot')
+
     def assertNotIgnored(self, request, middleware):
         spider = None  # not actually used
         dfd = maybeDeferred(middleware.process_request, request, spider)
diff --git a/tests/test_http_request.py b/tests/test_http_request.py
index 60494d792..16d7a1cb8 100644
--- a/tests/test_http_request.py
+++ b/tests/test_http_request.py
@@ -11,7 +11,7 @@ from six.moves.urllib.parse import urlparse, parse_qs, unquote
 if six.PY3:
     from urllib.parse import unquote_to_bytes
 
-from scrapy.http import Request, FormRequest, XmlRpcRequest, JSONRequest, Headers, HtmlResponse
+from scrapy.http import Request, FormRequest, XmlRpcRequest, JsonRequest, Headers, HtmlResponse
 from scrapy.utils.python import to_bytes, to_native_str
 
 from tests import mock
@@ -1246,14 +1246,14 @@ class XmlRpcRequestTest(RequestTest):
         self._test_request(params=(u'pas£',), encoding='latin1')
 
 
-class JSONRequestTest(RequestTest):
-    request_class = JSONRequest
+class JsonRequestTest(RequestTest):
+    request_class = JsonRequest
     default_method = 'GET'
     default_headers = {b'Content-Type': [b'application/json'], b'Accept': [b'application/json, text/javascript, */*; q=0.01']}
 
     def setUp(self):
         warnings.simplefilter("always")
-        super(JSONRequestTest, self).setUp()
+        super(JsonRequestTest, self).setUp()
 
     def test_data(self):
         r1 = self.request_class(url="http://www.example.com/")
@@ -1407,7 +1407,7 @@ class JSONRequestTest(RequestTest):
 
     def tearDown(self):
         warnings.resetwarnings()
-        super(JSONRequestTest, self).tearDown()
+        super(JsonRequestTest, self).tearDown()
 
 
 if __name__ == "__main__":