Repository: scrapy/scrapy
Pull request: Import paths #2733
@Gallaecio please check this file and let me know if this is the way that I should carry this issue, so I will be able to continue with the other files.
Fixes #2733.

Problem statement:
Import Request in the Template file
I have been creating many spiders recently, and I noticed I had to add below line every time

`from scrapy.http import Request`

I think for most cases we need the Request to crawl to other pages, only for very simple spider we don't need that. Is it possible to include that line to the template?

Hints from issue discussion:
The intention is for users to use `scrapy.Request`, `import scrapy` is already in the template. Does it work for you?
Thanks for the quick the response. I didn't noticed `Request` has been declared on the top level, I checked the API document and always use it from` scrapy.http`.

That is fair enough, and please close this issue.
Yeah, it'd be nice to mention the shortcut in API docs for Request object: https://doc.scrapy.org/en/latest/topics/request-response.html.
I'd like to grab this if that's ok
Hello! Is this issue still open? Does @keremgocen carry this? If not, could you give me some guidance on what needs to be done (as it was reopened) in order to contribute? 
This is still open indeed, and it seems like there has been no attempt to address it after https://github.com/scrapy/scrapy/pull/2757, which seems abandoned.
Great. Could you give me more guidance on what I should do? I saw your comment https://github.com/scrapy/scrapy/pull/2757#discussion_r263744493 and I assume that I have to begin from this point. 
Yes.

It’s relatively hard to find what I mention in this comment, but referencing all import paths looks like this:

```python
.. class:: scrapy.http.request.Request
.. class:: scrapy.http.Request
.. class:: scrapy.Request(…)
```

Then there are some possible additional changes to make, but it’s up to you whether to make them or not, since they are not covered by this issue report:

- The same could be done for other classes that have a top-level import in https://github.com/scrapy/scrapy/blob/master/scrapy/__init__.py
- In addition to covering all import paths in the reference documentation, you could search the documentation (and even the docstrings in code) for references to the longer import paths, and switch those usages to the short import path, as the shortest import path should probably be the recommendation in the documentation.
So, the purpose is to change every long import path in the documentation files, such as: `.. class:: scrapy.http.request.Request` and `.. class:: scrapy.http.Request` to: `.. class:: scrapy.Request(…)` ?
And this change should be applied to every class declared as top-level in  https://github.com/scrapy/scrapy/blob/master/scrapy/__init__.py:
```
# Declare top-level shortcuts
from scrapy.spiders import Spider
from scrapy.http import Request, FormRequest
from scrapy.selector import Selector
from scrapy.item import Item, Field
```
For example, in [request-response.rst](https://github.com/scrapy/scrapy/blob/master/docs/topics/request-response.rst) line 42 (`:meth:` `~scrapy.spiders.Spider.parse` ` method will be used.`) should become: ` meth: ` `~scrapy.Spider.parse` `  method will be used.`  ?

Yes.

The long import paths should only appear once, right above the APi member reference documentation (`.. class:: scrapy.Request(…`), just so that it’s clear that those import paths exist, and so that documentation linking to the Scrapy documentation using InterSphinx continues to work even if it uses the long import paths for API references.

Commit message: Fix more import paths #2733

Patch to review:
diff --git a/docs/faq.rst b/docs/faq.rst
index 9709885f631..16903daeaed 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -149,7 +149,7 @@ How can I prevent memory errors due to many allowed domains?
 ------------------------------------------------------------
 
 If you have a spider with a long list of
-:attr:`~scrapy.spiders.Spider.allowed_domains` (e.g. 50,000+), consider
+:attr:`~scrapy.Spider.allowed_domains` (e.g. 50,000+), consider
 replacing the default
 :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` spider middleware
 with a :ref:`custom spider middleware <custom-spider-middleware>` that requires
@@ -157,7 +157,7 @@ less memory. For example:
 
 -   If your domain names are similar enough, use your own regular expression
     instead joining the strings in
-    :attr:`~scrapy.spiders.Spider.allowed_domains` into a complex regular
+    :attr:`~scrapy.Spider.allowed_domains` into a complex regular
     expression.
 
 -   If you can `meet the installation requirements`_, use pyre2_ instead of
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 740e47d0c15..438f3d6df9d 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -78,7 +78,7 @@ Our first Spider
 
 Spiders are classes that you define and that Scrapy uses to scrape information
 from a website (or a group of websites). They must subclass
-:class:`~scrapy.spiders.Spider` and define the initial requests to make,
+:class:`~scrapy.Spider` and define the initial requests to make,
 optionally how to follow links in the pages, and how to parse the downloaded
 page content to extract data.
 
@@ -107,26 +107,26 @@ This is the code for our first Spider. Save it in a file named
             self.log(f'Saved file {filename}')
 
 
-As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.spiders.Spider>`
+As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.Spider>`
 and defines some attributes and methods:
 
-* :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be
+* :attr:`~scrapy.Spider.name`: identifies the Spider. It must be
   unique within a project, that is, you can't set the same name for different
   Spiders.
 
-* :meth:`~scrapy.spiders.Spider.start_requests`: must return an iterable of
+* :meth:`~scrapy.Spider.start_requests`: must return an iterable of
   Requests (you can return a list of requests or write a generator function)
   which the Spider will begin to crawl from. Subsequent requests will be
   generated successively from these initial requests.
 
-* :meth:`~scrapy.spiders.Spider.parse`: a method that will be called to handle
+* :meth:`~scrapy.Spider.parse`: a method that will be called to handle
   the response downloaded for each of the requests made. The response parameter
   is an instance of :class:`~scrapy.http.TextResponse` that holds
   the page content and has further helpful methods to handle it.
 
-  The :meth:`~scrapy.spiders.Spider.parse` method usually parses the response, extracting
+  The :meth:`~scrapy.Spider.parse` method usually parses the response, extracting
   the scraped data as dicts and also finding new URLs to
-  follow and creating new requests (:class:`~scrapy.http.Request`) from them.
+  follow and creating new requests (:class:`~scrapy.Request`) from them.
 
 How to run our spider
 ---------------------
@@ -162,7 +162,7 @@ for the respective URLs, as our ``parse`` method instructs.
 What just happened under the hood?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects
+Scrapy schedules the :class:`scrapy.Request <scrapy.Request>` objects
 returned by the ``start_requests`` method of the Spider. Upon receiving a
 response for each one, it instantiates :class:`~scrapy.http.Response` objects
 and calls the callback method associated with the request (in this case, the
@@ -171,11 +171,11 @@ and calls the callback method associated with the request (in this case, the
 
 A shortcut to the start_requests method
 ---------------------------------------
-Instead of implementing a :meth:`~scrapy.spiders.Spider.start_requests` method
-that generates :class:`scrapy.Request <scrapy.http.Request>` objects from URLs,
-you can just define a :attr:`~scrapy.spiders.Spider.start_urls` class attribute
+Instead of implementing a :meth:`~scrapy.Spider.start_requests` method
+that generates :class:`scrapy.Request <scrapy.Request>` objects from URLs,
+you can just define a :attr:`~scrapy.Spider.start_urls` class attribute
 with a list of URLs. This list will then be used by the default implementation
-of :meth:`~scrapy.spiders.Spider.start_requests` to create the initial requests
+of :meth:`~scrapy.Spider.start_requests` to create the initial requests
 for your spider::
 
     import scrapy
@@ -194,9 +194,9 @@ for your spider::
             with open(filename, 'wb') as f:
                 f.write(response.body)
 
-The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle each
+The :meth:`~scrapy.Spider.parse` method will be called to handle each
 of the requests for those URLs, even though we haven't explicitly told Scrapy
-to do so. This happens because :meth:`~scrapy.spiders.Spider.parse` is Scrapy's
+to do so. This happens because :meth:`~scrapy.Spider.parse` is Scrapy's
 default callback method, which is called for requests without an explicitly
 assigned callback.
 
@@ -248,7 +248,7 @@ object:
 
 The result of running ``response.css('title')`` is a list-like object called
 :class:`~scrapy.selector.SelectorList`, which represents a list of
-:class:`~scrapy.selector.Selector` objects that wrap around XML/HTML elements
+:class:`~scrapy.Selector` objects that wrap around XML/HTML elements
 and allow you to run further queries to fine-grain the selection or extract the
 data.
 
@@ -670,7 +670,7 @@ the pagination links with the ``parse`` callback as we saw before.
 Here we're passing callbacks to
 :meth:`response.follow_all <scrapy.http.TextResponse.follow_all>` as positional
 arguments to make the code shorter; it also works for
-:class:`~scrapy.http.Request`.
+:class:`~scrapy.Request`.
 
 The ``parse_author`` callback defines a helper function to extract and cleanup the
 data from a CSS query and yields the Python dict with the author data.
diff --git a/docs/news.rst b/docs/news.rst
index 0ea412e753a..d5e9993fc3f 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -267,13 +267,13 @@ Backward-incompatible changes
 
 *   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` once again
     discards cookies defined in :attr:`Request.headers
-    <scrapy.http.Request.headers>`.
+    <scrapy.Request.headers>`.
 
     We decided to revert this bug fix, introduced in Scrapy 2.2.0, because it
     was reported that the current implementation could break existing code.
 
     If you need to set cookies for a request, use the :class:`Request.cookies
-    <scrapy.http.Request>` parameter.
+    <scrapy.Request>` parameter.
 
     A future version of Scrapy will include a new, better implementation of the
     reverted bug fix.
@@ -394,16 +394,16 @@ New features
     :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`
     or
     :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception`
-    with a custom :class:`~scrapy.http.Request` object assigned to
+    with a custom :class:`~scrapy.Request` object assigned to
     :class:`response.request <scrapy.http.Response.request>`:
 
     -   The response is handled by the callback of that custom
-        :class:`~scrapy.http.Request` object, instead of being handled by the
-        callback of the original :class:`~scrapy.http.Request` object
+        :class:`~scrapy.Request` object, instead of being handled by the
+        callback of the original :class:`~scrapy.Request` object
 
-    -   That custom :class:`~scrapy.http.Request` object is now sent as the
+    -   That custom :class:`~scrapy..Request` object is now sent as the
         ``request`` argument to the :signal:`response_received` signal, instead
-        of the original :class:`~scrapy.http.Request` object
+        of the original :class:`~scrapy.Request` object
 
     (:issue:`4529`, :issue:`4632`)
 
@@ -574,7 +574,7 @@ New features
 *   The :command:`parse` command now allows specifying an output file
     (:issue:`4317`, :issue:`4377`)
 
-*   :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
+*   :meth:`Request.from_curl <scrapy.Request.from_curl>` and
     :func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support
     ``--data-raw`` (:issue:`4612`)
 
@@ -590,7 +590,7 @@ Bug fixes
     :ref:`dataclass items <dataclass-items>` and :ref:`attr.s items
     <attrs-items>` (:issue:`4667`, :issue:`4668`)
 
-*   :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
+*   :meth:`Request.from_curl <scrapy.Request.from_curl>` and
     :func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request
     method to ``POST`` when a request body is specified and no request method
     is specified (:issue:`4612`)
@@ -681,7 +681,7 @@ Deprecations
     (:issue:`4546`, :issue:`4555`, :issue:`4579`)
 
 *   :class:`scrapy.item.BaseItem` is now deprecated, use
-    :class:`scrapy.item.Item` instead (:issue:`4534`)
+    :class:`scrapy.Item` instead (:issue:`4534`)
 
 
 New features
@@ -716,8 +716,8 @@ New features
 *   :ref:`Link extractors <topics-link-extractors>` are now serializable,
     as long as you do not use :ref:`lambdas <lambda>` for parameters; for
     example, you can now pass link extractors in :attr:`Request.cb_kwargs
-    <scrapy.http.Request.cb_kwargs>` or
-    :attr:`Request.meta <scrapy.http.Request.meta>` when :ref:`persisting
+    <scrapy.Request.cb_kwargs>` or
+    :attr:`Request.meta <scrapy.Request.meta>` when :ref:`persisting
     scheduled requests <topics-jobs>` (:issue:`4554`)
 
 *   Upgraded the :ref:`pickle protocol <pickle-protocols>` that Scrapy uses
@@ -736,20 +736,20 @@ Bug fixes
 
 *   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
     discards cookies defined in :attr:`Request.headers
-    <scrapy.http.Request.headers>` (:issue:`1992`, :issue:`2400`)
+    <scrapy.Request.headers>` (:issue:`1992`, :issue:`2400`)
 
 *   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
     re-encodes cookies defined as :class:`bytes` in the ``cookies`` parameter
-    of the ``__init__`` method of :class:`~scrapy.http.Request`
+    of the ``__init__`` method of :class:`~scrapy.Request`
     (:issue:`2400`, :issue:`3575`)
 
 *   When :setting:`FEEDS` defines multiple URIs, :setting:`FEED_STORE_EMPTY` is
     ``False`` and the crawl yields no items, Scrapy no longer stops feed
     exports after the first URI (:issue:`4621`, :issue:`4626`)
 
-*   :class:`~scrapy.spiders.Spider` callbacks defined using :doc:`coroutine
+*   :class:`~scrapy.Spider` callbacks defined using :doc:`coroutine
     syntax <topics/coroutines>` no longer need to return an iterable, and may
-    instead return a :class:`~scrapy.http.Request` object, an
+    instead return a :class:`~scrapy.Request` object, an
     :ref:`item <topics-items>`, or ``None`` (:issue:`4609`)
 
 *   The :command:`startproject` command now ensures that the generated project
@@ -791,7 +791,7 @@ Documentation
 
 *   The display-on-hover behavior of internal documentation references now also
     covers links to :ref:`commands <topics-commands>`, :attr:`Request.meta
-    <scrapy.http.Request.meta>` keys, :ref:`settings <topics-settings>` and
+    <scrapy.Request.meta>` keys, :ref:`settings <topics-settings>` and
     :ref:`signals <topics-signals>` (:issue:`4495`, :issue:`4563`)
 
 *   It is again possible to download the documentation for offline reading
@@ -918,7 +918,7 @@ New features
     (:issue:`3903`, :issue:`3940`)
 
 *   A warning is now issued when a value in
-    :attr:`~scrapy.spiders.Spider.allowed_domains` includes a port
+    :attr:`~scrapy.Spider.allowed_domains` includes a port
     (:issue:`50`, :issue:`3198`, :issue:`4413`)
 
 *   Zsh completion now excludes used option aliases from the completion list
@@ -932,7 +932,7 @@ Bug fixes
     callbacks that are spider attributes which are assigned a function with a
     different name (:issue:`4500`)
 
-*   ``None`` values in :attr:`~scrapy.spiders.Spider.allowed_domains` no longer
+*   ``None`` values in :attr:`~scrapy.Spider.allowed_domains` no longer
     cause a :exc:`TypeError` exception (:issue:`4410`)
 
 *   Zsh completion no longer allows options after arguments (:issue:`4438`)
@@ -941,7 +941,7 @@ Bug fixes
     (:issue:`4447`, :issue:`4448`)
 
 *   :meth:`Spider.make_requests_from_url
-    <scrapy.spiders.Spider.make_requests_from_url>`, deprecated in Scrapy
+    <scrapy.Spider.make_requests_from_url>`, deprecated in Scrapy
     1.4.0, now issues a warning when used (:issue:`4412`)
 
 
@@ -1147,10 +1147,10 @@ New features
 
 *   The new :attr:`Response.cb_kwargs <scrapy.http.Response.cb_kwargs>`
     attribute serves as a shortcut for :attr:`Response.request.cb_kwargs
-    <scrapy.http.Request.cb_kwargs>` (:issue:`4331`)
+    <scrapy.Request.cb_kwargs>` (:issue:`4331`)
 
 *   :meth:`Response.follow <scrapy.http.Response.follow>` now supports a
-    ``flags`` parameter, for consistency with :class:`~scrapy.http.Request`
+    ``flags`` parameter, for consistency with :class:`~scrapy.Request`
     (:issue:`4277`, :issue:`4279`)
 
 *   :ref:`Item loader processors <topics-loaders-processors>` can now be
@@ -1159,7 +1159,7 @@ New features
 *   :class:`~scrapy.spiders.Rule` now accepts an ``errback`` parameter
     (:issue:`4000`)
 
-*   :class:`~scrapy.http.Request` no longer requires a ``callback`` parameter
+*   :class:`~scrapy.Request` no longer requires a ``callback`` parameter
     when an ``errback`` parameter is specified (:issue:`3586`, :issue:`4008`)
 
 *   :class:`~scrapy.logformatter.LogFormatter` now supports some additional
@@ -1187,9 +1187,9 @@ New features
     uses ``yield`` but also returns a value, since the returned value would be
     lost (:issue:`3484`, :issue:`3869`)
 
-*   :class:`~scrapy.spiders.Spider` objects now raise an :exc:`AttributeError`
-    exception if they do not have a :class:`~scrapy.spiders.Spider.start_urls`
-    attribute nor reimplement :class:`~scrapy.spiders.Spider.start_requests`,
+*   :class:`~scrapy.Spider` objects now raise an :exc:`AttributeError`
+    exception if they do not have a :class:`~scrapy.Spider.start_urls`
+    attribute nor reimplement :class:`~scrapy.Spider.start_requests`,
     but have a ``start_url`` attribute (:issue:`4133`, :issue:`4170`)
 
 *   :class:`~scrapy.exporters.BaseItemExporter` subclasses may now use
@@ -1231,7 +1231,7 @@ Bug fixes
 *   Redirects to URLs starting with 3 slashes (``///``) are now supported
     (:issue:`4032`, :issue:`4042`)
 
-*   :class:`~scrapy.http.Request` no longer accepts strings as ``url`` simply
+*   :class:`~scrapy.Request` no longer accepts strings as ``url`` simply
     because they have a colon (:issue:`2552`, :issue:`4094`)
 
 *   The correct encoding is now used for attach names in
@@ -1277,7 +1277,7 @@ Documentation
     using :class:`~scrapy.crawler.CrawlerProcess` (:issue:`2149`,
     :issue:`2352`, :issue:`3146`, :issue:`3960`)
 
-*   Clarified the requirements for :class:`~scrapy.http.Request` objects
+*   Clarified the requirements for :class:`~scrapy.Request` objects
     :ref:`when using persistence <request-serialization>` (:issue:`4124`,
     :issue:`4139`)
 
@@ -1463,7 +1463,7 @@ Highlights:
 
 * Dropped Python 3.4 support and updated minimum requirements; made Python 3.8
   support official
-* New :meth:`Request.from_curl <scrapy.http.Request.from_curl>` class method
+* New :meth:`Request.from_curl <scrapy.Request.from_curl>` class method
 * New :setting:`ROBOTSTXT_PARSER` and :setting:`ROBOTSTXT_USER_AGENT` settings
 * New :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and
   :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` settings
@@ -1515,7 +1515,7 @@ See also :ref:`1.8-deprecation-removals` below.
 New features
 ~~~~~~~~~~~~
 
-*   A new :meth:`Request.from_curl <scrapy.http.Request.from_curl>` class
+*   A new :meth:`Request.from_curl <scrapy.Request.from_curl>` class
     method allows :ref:`creating a request from a cURL command
     <requests-from-curl>` (:issue:`2985`, :issue:`3862`)
 
@@ -1545,7 +1545,7 @@ New features
     after establishing HTTPS connections (:issue:`2111`, :issue:`3450`)
 
 *   Callbacks that receive keyword arguments
-    (see :attr:`Request.cb_kwargs <scrapy.http.Request.cb_kwargs>`) can now be
+    (see :attr:`Request.cb_kwargs <scrapy.Request.cb_kwargs>`) can now be
     tested using the new :class:`@cb_kwargs
     <scrapy.contracts.default.CallbackKeywordArgumentsContract>`
     :ref:`spider contract <topics-contracts>` (:issue:`3985`, :issue:`3988`)
@@ -1632,7 +1632,7 @@ Deprecations
 *   Use of the undocumented ``SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE`` environment
     variable is deprecated (:issue:`3910`)
 
-*   ``scrapy.item.DictItem`` is deprecated, use :class:`~scrapy.item.Item`
+*   ``scrapy.item.DictItem`` is deprecated, use :class:`~scrapy.Item`
     instead (:issue:`3999`)
 
 
@@ -1724,18 +1724,18 @@ Backward-incompatible changes
 *   :class:`~scrapy.crawler.Crawler`,
     :class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
     :class:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
-    no longer accept a :class:`~scrapy.spiders.Spider` subclass instance, they
-    only accept a :class:`~scrapy.spiders.Spider` subclass now.
+    no longer accept a :class:`~scrapy.Spider` subclass instance, they
+    only accept a :class:`~scrapy.Spider` subclass now.
 
-    :class:`~scrapy.spiders.Spider` subclass instances were never meant to
+    :class:`~scrapy.Spider` subclass instances were never meant to
     work, and they were not working as one would expect: instead of using the
-    passed :class:`~scrapy.spiders.Spider` subclass instance, their
-    :class:`~scrapy.spiders.Spider.from_crawler` method was called to generate
+    passed :class:`~scrapy.Spider` subclass instance, their
+    :class:`~scrapy.Spider.from_crawler` method was called to generate
     a new instance.
 
 *   Non-default values for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting
     may stop working. Scheduler priority queue classes now need to handle
-    :class:`~scrapy.http.Request` objects instead of arbitrary Python data
+    :class:`~scrapy.Request` objects instead of arbitrary Python data
     structures.
 
 *   An additional ``crawler`` parameter has been added to the ``__init__``
@@ -1757,7 +1757,7 @@ New features
     scheduling improvement on crawls targetting multiple web domains, at the
     cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)
 
-*   A new :attr:`Request.cb_kwargs <scrapy.http.Request.cb_kwargs>` attribute
+*   A new :attr:`Request.cb_kwargs <scrapy.Request.cb_kwargs>` attribute
     provides a cleaner way to pass keyword arguments to callback methods
     (:issue:`1138`, :issue:`3563`)
 
@@ -1792,7 +1792,7 @@ New features
     check runs from code <detecting-contract-check-runs>` (:issue:`3704`,
     :issue:`3739`)
 
-*   A new :meth:`Item.deepcopy() <scrapy.item.Item.deepcopy>` method makes it
+*   A new :meth:`Item.deepcopy() <scrapy.Item.deepcopy>` method makes it
     easier to :ref:`deep-copy items <copying-items>` (:issue:`1493`,
     :issue:`3671`)
 
@@ -1806,7 +1806,7 @@ New features
 *   :class:`~scrapy.crawler.Crawler`,
     :class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
     :class:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
-    now fail gracefully if they receive a :class:`~scrapy.spiders.Spider`
+    now fail gracefully if they receive a :class:`~scrapy.Spider`
     subclass instance instead of the subclass itself (:issue:`2283`,
     :issue:`3610`, :issue:`3872`)
 
@@ -1838,7 +1838,7 @@ Bug fixes
 *   Requests with private callbacks are now correctly unserialized from disk
     (:issue:`3790`)
 
-*   :meth:`FormRequest.from_response() <scrapy.http.FormRequest.from_response>`
+*   :meth:`FormRequest.from_response() <scrapy.FormRequest.from_response>`
     now handles invalid methods like major web browsers (:issue:`3777`,
     :issue:`3794`)
 
@@ -1918,13 +1918,13 @@ The following deprecated APIs have been removed (:issue:`3578`):
 
 *   From both ``scrapy.selector`` and ``scrapy.selector.lxmlsel``:
 
-    *   ``HtmlXPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``HtmlXPathSelector`` (use :class:`~scrapy.Selector`)
 
-    *   ``XmlXPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XmlXPathSelector`` (use :class:`~scrapy.Selector`)
 
-    *   ``XPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XPathSelector`` (use :class:`~scrapy..Selector`)
 
-    *   ``XPathSelectorList`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XPathSelectorList`` (use :class:`~scrapy.Selector`)
 
 *   From ``scrapy.selector.csstranslator``:
 
@@ -1934,7 +1934,7 @@ The following deprecated APIs have been removed (:issue:`3578`):
 
     *   ``ScrapyXPathExpr`` (use parsel.csstranslator.XPathExpr_)
 
-*   From :class:`~scrapy.selector.Selector`:
+*   From :class:`~scrapy.Selector`:
 
     *   ``_root`` (both the ``__init__`` method argument and the object property, use
         ``root``)
@@ -1951,14 +1951,14 @@ The following deprecated APIs have been removed (:issue:`3578`):
 
     *   ``x`` (use ``xpath``)
 
-*   ``scrapy.spiders.BaseSpider`` (use :class:`~scrapy.spiders.Spider`)
+*   ``scrapy.spiders.BaseSpider`` (use :class:`~scrapy.Spider`)
 
-*   From :class:`~scrapy.spiders.Spider` (and subclasses):
+*   From :class:`~scrapy.Spider` (and subclasses):
 
     *   ``DOWNLOAD_DELAY`` (use :ref:`download_delay
         <spider-download_delay-attribute>`)
 
-    *   ``set_crawler`` (use :meth:`~scrapy.spiders.Spider.from_crawler`)
+    *   ``set_crawler`` (use :meth:`~scrapy.Spider.from_crawler`)
 
 *   ``scrapy.spiders.spiders`` (use :class:`~scrapy.spiderloader.SpiderLoader`)
 
@@ -2464,7 +2464,7 @@ New Features
   (:issue:`2535`)
 - New :ref:`response.follow <response-follow-example>` shortcut
   for creating requests (:issue:`1940`)
-- Added ``flags`` argument and attribute to :class:`Request <scrapy.http.Request>`
+- Added ``flags`` argument and attribute to :class:`Request <scrapy.Request>`
   objects (:issue:`2047`)
 - Support Anonymous FTP (:issue:`2342`)
 - Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/<reason>``
@@ -2506,7 +2506,7 @@ Bug fixes
 - LinkExtractor now strips leading and trailing whitespaces from attributes
   (:issue:`2547`, fixes :issue:`1614`)
 - Properly handle whitespaces in action attribute in
-  :class:`~scrapy.http.FormRequest` (:issue:`2548`)
+  :class:`~scrapy.FormRequest` (:issue:`2548`)
 - Buffer CONNECT response bytes from proxy until all HTTP headers are received
   (:issue:`2495`, fixes :issue:`2491`)
 - FTP downloader now works on Python 3, provided you use Twisted>=17.1
@@ -2549,7 +2549,7 @@ Documentation
 
 - Binary mode is required for exporters (:issue:`2564`, fixes :issue:`2553`)
 - Mention issue with :meth:`FormRequest.from_response
-  <scrapy.http.FormRequest.from_response>` due to bug in lxml (:issue:`2572`)
+  <scrapy.FormRequest.from_response>` due to bug in lxml (:issue:`2572`)
 - Use single quotes uniformly in templates (:issue:`2596`)
 - Document :reqmeta:`ftp_user` and :reqmeta:`ftp_password` meta keys (:issue:`2587`)
 - Removed section on deprecated ``contrib/`` (:issue:`2636`)
@@ -3090,7 +3090,7 @@ Bugfixes
 - Support empty password for http_proxy config (:issue:`1274`).
 - Interpret ``application/x-json`` as ``TextResponse`` (:issue:`1333`).
 - Support link rel attribute with multiple values (:issue:`1201`).
-- Fixed ``scrapy.http.FormRequest.from_response`` when there is a ``<base>``
+- Fixed ``scrapy.FormRequest.from_response`` when there is a ``<base>``
   tag (:issue:`1564`).
 - Fixed :setting:`TEMPLATES_DIR` handling (:issue:`1575`).
 - Various ``FormRequest`` fixes (:issue:`1595`, :issue:`1596`, :issue:`1597`).
@@ -3264,7 +3264,7 @@ override any given setting. As an extension of that goal, we included a new
 level of priority for settings that act exclusively for a single spider,
 allowing them to redefine project settings.
 
-Start using it by defining a :attr:`~scrapy.spiders.Spider.custom_settings`
+Start using it by defining a :attr:`~scrapy.Spider.custom_settings`
 class variable in your spider::
 
     class MySpider(scrapy.Spider):
@@ -3298,8 +3298,8 @@ warnings to switch to the Python logging API entirely.
     logging.info('MESSAGE')
 
 Logging with spiders remains the same, but on top of the
-:meth:`~scrapy.spiders.Spider.log` method you’ll have access to a custom
-:attr:`~scrapy.spiders.Spider.logger` created for the spider to issue log
+:meth:`~scrapy.Spider.log` method you’ll have access to a custom
+:attr:`~scrapy.Spider.logger` created for the spider to issue log
 events:
 
 ::
@@ -3806,7 +3806,7 @@ Enhancements
   To restore old backend set ``HTTPCACHE_STORAGE`` to ``scrapy.contrib.httpcache.DbmCacheStorage``
 - Proxy \https:// urls using CONNECT method (:issue:`392`, :issue:`397`)
 - Add a middleware to crawl ajax crawleable pages as defined by google (:issue:`343`)
-- Rename scrapy.spider.BaseSpider to scrapy.spider.Spider (:issue:`510`, :issue:`519`)
+- Rename scrapy.spider.BaseSpider to scrapy.Spider (:issue:`510`, :issue:`519`)
 - Selectors register EXSLT namespaces by default (:issue:`472`)
 - Unify item loaders similar to selectors renaming (:issue:`461`)
 - Make ``RFPDupeFilter`` class easily subclassable (:issue:`533`)
@@ -4017,7 +4017,7 @@ Scrapy 0.18.0 (released 2013-08-09)
 - Moved persistent (on disk) queues to a separate project (queuelib_) which Scrapy now depends on
 - Add Scrapy commands using external libraries (:issue:`260`)
 - Added ``--pdb`` option to ``scrapy`` command line tool
-- Added :meth:`XPathSelector.remove_namespaces <scrapy.selector.Selector.remove_namespaces>` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`.
+- Added :meth:`XPathSelector.remove_namespaces <scrapy.Selector.remove_namespaces>` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`.
 - Several improvements to spider contracts
 - New default middleware named MetaRefreshMiddldeware that handles meta-refresh html tag redirections,
 - MetaRefreshMiddldeware and RedirectMiddleware have different priorities to address #62
@@ -4465,7 +4465,7 @@ Backward-incompatible changes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 - Changed ``scrapy.utils.response.get_meta_refresh()`` signature (:rev:`1804`)
-- Removed deprecated ``scrapy.item.ScrapedItem`` class - use ``scrapy.item.Item instead`` (:rev:`1838`)
+- Removed deprecated ``scrapy.item.ScrapedItem`` class - use ``scrapy.Item instead`` (:rev:`1838`)
 - Removed deprecated ``scrapy.xpath`` module - use ``scrapy.selector`` instead. (:rev:`1836`)
 - Removed deprecated ``core.signals.domain_open`` signal - use ``core.signals.domain_opened`` instead (:rev:`1822`)
 - ``log.msg()`` now receives a ``spider`` argument (:rev:`1822`)
diff --git a/docs/topics/api.rst b/docs/topics/api.rst
index 445b2979fa8..900b19c7a07 100644
--- a/docs/topics/api.rst
+++ b/docs/topics/api.rst
@@ -29,7 +29,7 @@ how you :ref:`configure the downloader middlewares
 .. class:: Crawler(spidercls, settings)
 
     The Crawler object must be instantiated with a
-    :class:`scrapy.spiders.Spider` subclass and a
+    :class:`scrapy.Spider` subclass and a
     :class:`scrapy.settings.Settings` object.
 
     .. attribute:: settings
@@ -196,7 +196,7 @@ SpiderLoader API
        match the request's url against the domains of the spiders.
 
        :param request: queried request
-       :type request: :class:`~scrapy.http.Request` instance
+       :type request: :class:`~scrapy.Request` instance
 
 .. _topics-api-signals:
 
diff --git a/docs/topics/contracts.rst b/docs/topics/contracts.rst
index e61421bf11a..ef296dc9e55 100644
--- a/docs/topics/contracts.rst
+++ b/docs/topics/contracts.rst
@@ -37,7 +37,7 @@ This callback is tested using three built-in contracts:
 
 .. class:: CallbackKeywordArgumentsContract
 
-    This contract (``@cb_kwargs``) sets the :attr:`cb_kwargs <scrapy.http.Request.cb_kwargs>`
+    This contract (``@cb_kwargs``) sets the :attr:`cb_kwargs <scrapy.Request.cb_kwargs>`
     attribute for the sample request. It must be a valid JSON dictionary.
     ::
 
@@ -88,7 +88,7 @@ override three methods:
     .. method:: Contract.adjust_request_args(args)
 
         This receives a ``dict`` as an argument containing default arguments
-        for request object. :class:`~scrapy.http.Request` is used by default,
+        for request object. :class:`~scrapy.Request` is used by default,
         but this can be changed with the ``request_cls`` attribute.
         If multiple contracts in chain have this attribute defined, the last one is used.
 
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 3b1549bd3e0..0904637b07f 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -15,7 +15,7 @@ Supported callables
 The following callables may be defined as coroutines using ``async def``, and
 hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
--   :class:`~scrapy.http.Request` callbacks.
+-   :class:`~scrapy.Request` callbacks.
 
     .. note:: The callback output is not processed until the whole callback
         finishes.
diff --git a/docs/topics/debug.rst b/docs/topics/debug.rst
index d75f1730147..4d452b4dfca 100644
--- a/docs/topics/debug.rst
+++ b/docs/topics/debug.rst
@@ -36,7 +36,7 @@ Consider the following Scrapy spider below::
 
 Basically this is a simple spider which parses two pages of items (the
 start_urls). Items also have a details page with additional information, so we
-use the ``cb_kwargs`` functionality of :class:`~scrapy.http.Request` to pass a
+use the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a
 partially populated item.
 
 
diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst
index c83b1a9d9ee..057b1ec6222 100644
--- a/docs/topics/developer-tools.rst
+++ b/docs/topics/developer-tools.rst
@@ -274,7 +274,7 @@ In more complex websites, it could be difficult to easily reproduce the
 requests, as we could need to add ``headers`` or ``cookies`` to make it work.
 In those cases you can export the requests in `cURL <https://curl.haxx.se/>`_
 format, by right-clicking on each of them in the network tool and using the
-:meth:`~scrapy.http.Request.from_curl()` method to generate an equivalent
+:meth:`~scrapy.Request.from_curl()` method to generate an equivalent
 request::
 
     from scrapy import Request
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index b539c23dffa..80c6c2c3770 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -76,7 +76,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
       middleware.
 
       :meth:`process_request` should either: return ``None``, return a
-      :class:`~scrapy.http.Response` object, return a :class:`~scrapy.http.Request`
+      :class:`~scrapy.Response` object, return a :class:`~scrapy.http.Request`
       object, or raise :exc:`~scrapy.exceptions.IgnoreRequest`.
 
       If it returns ``None``, Scrapy will continue processing this request, executing all
@@ -88,7 +88,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
       or the appropriate download function; it'll return that response. The :meth:`process_response`
       methods of installed middleware is always called on every response.
 
-      If it returns a :class:`~scrapy.http.Request` object, Scrapy will stop calling
+      If it returns a :class:`~scrapy.Request` object, Scrapy will stop calling
       process_request methods and reschedule the returned request. Once the newly returned
       request is performed, the appropriate middleware chain will be called on
       the downloaded response.
@@ -100,22 +100,22 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
       ignored and not logged (unlike other exceptions).
 
       :param request: the request being processed
-      :type request: :class:`~scrapy.http.Request` object
+      :type request: :class:`~scrapy.Request` object
 
       :param spider: the spider for which this request is intended
-      :type spider: :class:`~scrapy.spiders.Spider` object
+      :type spider: :class:`~scrapy.Spider` object
 
    .. method:: process_response(request, response, spider)
 
       :meth:`process_response` should either: return a :class:`~scrapy.http.Response`
-      object, return a :class:`~scrapy.http.Request` object or
+      object, return a :class:`~scrapy.Request` object or
       raise a :exc:`~scrapy.exceptions.IgnoreRequest` exception.
 
       If it returns a :class:`~scrapy.http.Response` (it could be the same given
       response, or a brand-new one), that response will continue to be processed
       with the :meth:`process_response` of the next middleware in the chain.
 
-      If it returns a :class:`~scrapy.http.Request` object, the middleware chain is
+      If it returns a :class:`~scrapy.Request` object, the middleware chain is
       halted and the returned request is rescheduled to be downloaded in the future.
       This is the same behavior as if a request is returned from :meth:`process_request`.
 
@@ -124,13 +124,13 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
       exception, it is ignored and not logged (unlike other exceptions).
 
       :param request: the request that originated the response
-      :type request: is a :class:`~scrapy.http.Request` object
+      :type request: is a :class:`~scrapy.Request` object
 
       :param response: the response being processed
       :type response: :class:`~scrapy.http.Response` object
 
       :param spider: the spider for which this response is intended
-      :type spider: :class:`~scrapy.spiders.Spider` object
+      :type spider: :class:`~scrapy.Spider` object
 
    .. method:: process_exception(request, exception, spider)
 
@@ -139,7 +139,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
       exception (including an :exc:`~scrapy.exceptions.IgnoreRequest` exception)
 
       :meth:`process_exception` should return: either ``None``,
-      a :class:`~scrapy.http.Response` object, or a :class:`~scrapy.http.Request` object.
+      a :class:`~scrapy.http.Response` object, or a :class:`~scrapy.Request` object.
 
       If it returns ``None``, Scrapy will continue processing this exception,
       executing any other :meth:`process_exception` methods of installed middleware,
@@ -149,19 +149,19 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
       method chain of installed middleware is started, and Scrapy won't bother calling
       any other :meth:`process_exception` methods of middleware.
 
-      If it returns a :class:`~scrapy.http.Request` object, the returned request is
+      If it returns a :class:`~scrapy.Request` object, the returned request is
       rescheduled to be downloaded in the future. This stops the execution of
       :meth:`process_exception` methods of the middleware the same as returning a
       response would.
 
       :param request: the request that generated the exception
-      :type request: is a :class:`~scrapy.http.Request` object
+      :type request: is a :class:`~scrapy.Request` object
 
       :param exception: the raised exception
       :type exception: an ``Exception`` object
 
       :param spider: the spider for which this request is intended
-      :type spider: :class:`~scrapy.spiders.Spider` object
+      :type spider: :class:`~scrapy.Spider` object
 
    .. method:: from_crawler(cls, crawler)
 
@@ -203,13 +203,13 @@ CookiesMiddleware
    browsers do.
 
    .. caution:: When non-UTF8 encoded byte sequences are passed to a
-      :class:`~scrapy.http.Request`, the ``CookiesMiddleware`` will log
+      :class:`~scrapy.Request`, the ``CookiesMiddleware`` will log
       a warning. Refer to :ref:`topics-logging-advanced-customization`
       to customize the logging behaviour.
 
    .. caution:: Cookies set via the ``Cookie`` header are not considered by the
       :ref:`cookies-mw`. If you need to set cookies for a request, use the
-      :class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
+      :class:`Request.cookies <scrapy.Request>` parameter. This is a known
       current limitation that is being worked on.
 
 The following settings can be used to configure the cookie middleware:
@@ -258,7 +258,7 @@ web server and received cookies in :class:`~scrapy.http.Response` will
 **not** be merged with the existing cookies.
 
 For more detailed information see the ``cookies`` parameter in
-:class:`~scrapy.http.Request`.
+:class:`~scrapy.Request`.
 
 .. setting:: COOKIES_DEBUG
 
@@ -501,7 +501,7 @@ defines the methods described below.
       the :signal:`open_spider <spider_opened>` signal.
 
       :param spider: the spider which has been opened
-      :type spider: :class:`~scrapy.spiders.Spider` object
+      :type spider: :class:`~scrapy.Spider` object
 
     .. method:: close_spider(spider)
 
@@ -509,27 +509,27 @@ defines the methods described below.
       the :signal:`close_spider <spider_closed>` signal.
 
       :param spider: the spider which has been closed
-      :type spider: :class:`~scrapy.spiders.Spider` object
+      :type spider: :class:`~scrapy.Spider` object
 
     .. method:: retrieve_response(spider, request)
 
       Return response if present in cache, or ``None`` otherwise.
 
       :param spider: the spider which generated the request
-      :type spider: :class:`~scrapy.spiders.Spider` object
+      :type spider: :class:`~scrapy.Spider` object
 
       :param request: the request to find cached response for
-      :type request: :class:`~scrapy.http.Request` object
+      :type request: :class:`~scrapy.Request` object
 
     .. method:: store_response(spider, request, response)
 
       Store the given response in the cache.
 
       :param spider: the spider for which the response is intended
-      :type spider: :class:`~scrapy.spiders.Spider` object
+      :type spider: :class:`~scrapy.Spider` object
 
       :param request: the corresponding request the spider generated
-      :type request: :class:`~scrapy.http.Request` object
+      :type request: :class:`~scrapy.Request` object
 
       :param response: the response to store in the cache
       :type response: :class:`~scrapy.http.Response` object
@@ -722,7 +722,7 @@ HttpProxyMiddleware
 .. class:: HttpProxyMiddleware
 
    This middleware sets the HTTP proxy to use for requests, by setting the
-   ``proxy`` meta value for :class:`~scrapy.http.Request` objects.
+   ``proxy`` meta value for :class:`~scrapy.Request` objects.
 
    Like the Python standard library module :mod:`urllib.request`, it obeys
    the following environment variables:
@@ -749,12 +749,12 @@ RedirectMiddleware
 .. reqmeta:: redirect_urls
 
 The urls which the request goes through (while being redirected) can be found
-in the ``redirect_urls`` :attr:`Request.meta <scrapy.http.Request.meta>` key.
+in the ``redirect_urls`` :attr:`Request.meta <scrapy.Request.meta>` key.
 
 .. reqmeta:: redirect_reasons
 
 The reason behind each redirect in :reqmeta:`redirect_urls` can be found in the
-``redirect_reasons`` :attr:`Request.meta <scrapy.http.Request.meta>` key. For
+``redirect_reasons`` :attr:`Request.meta <scrapy.Request.meta>` key. For
 example: ``[301, 302, 307, 'meta refresh']``.
 
 The format of a reason depends on the middleware that handled the corresponding
@@ -770,7 +770,7 @@ settings (see the settings documentation for more info):
 
 .. reqmeta:: dont_redirect
 
-If :attr:`Request.meta <scrapy.http.Request.meta>` has ``dont_redirect``
+If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_redirect``
 key set to True, the request will be ignored by this middleware.
 
 If you want to handle some redirect status codes in your spider, you can
@@ -783,7 +783,7 @@ responses (and pass them through to your spider) you can do this::
         handle_httpstatus_list = [301, 302]
 
 The ``handle_httpstatus_list`` key of :attr:`Request.meta
-<scrapy.http.Request.meta>` can also be used to specify which response codes to
+<scrapy.Request.meta>` can also be used to specify which response codes to
 allow on a per-request basis. You can also set the meta key
 ``handle_httpstatus_all`` to ``True`` if you want to allow any response code
 for a request.
@@ -889,7 +889,7 @@ settings (see the settings documentation for more info):
 
 .. reqmeta:: dont_retry
 
-If :attr:`Request.meta <scrapy.http.Request.meta>` has ``dont_retry`` key
+If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key
 set to True, the request will be ignored by this middleware.
 
 To retry requests from a spider callback, you can use the
@@ -919,7 +919,7 @@ Default: ``2``
 Maximum number of times to retry, in addition to the first download.
 
 Maximum number of retries can also be specified per-request using
-:reqmeta:`max_retry_times` attribute of :attr:`Request.meta <scrapy.http.Request.meta>`.
+:reqmeta:`max_retry_times` attribute of :attr:`Request.meta <scrapy.Request.meta>`.
 When initialized, the :reqmeta:`max_retry_times` meta key takes higher
 precedence over the :setting:`RETRY_TIMES` setting.
 
@@ -986,7 +986,7 @@ RobotsTxtMiddleware
 
 .. reqmeta:: dont_obey_robotstxt
 
-If :attr:`Request.meta <scrapy.http.Request.meta>` has
+If :attr:`Request.meta <scrapy.Request.meta>` has
 ``dont_obey_robotstxt`` key set to True
 the request will be ignored by this middleware even if
 :setting:`ROBOTSTXT_OBEY` is enabled.
diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst
index 495111b5636..aa32c8943f7 100644
--- a/docs/topics/dynamic-content.rst
+++ b/docs/topics/dynamic-content.rst
@@ -62,9 +62,9 @@ download the webpage with an HTTP client like curl_ or wget_ and see if the
 information can be found in the response they get.
 
 If they get a response with the desired data, modify your Scrapy
-:class:`~scrapy.http.Request` to match that of the other HTTP client. For
+:class:`~scrapy.Request` to match that of the other HTTP client. For
 example, try using the same user-agent string (:setting:`USER_AGENT`) or the
-same :attr:`~scrapy.http.Request.headers`.
+same :attr:`~scrapy.Request.headers`.
 
 If they also get a response without the desired data, you’ll need to take
 steps to make your request more similar to that of the web browser. See
@@ -81,14 +81,14 @@ Use the :ref:`network tool <topics-network-tool>` of your web browser to see
 how your web browser performs the desired request, and try to reproduce that
 request with Scrapy.
 
-It might be enough to yield a :class:`~scrapy.http.Request` with the same HTTP
+It might be enough to yield a :class:`~scrapy.Request` with the same HTTP
 method and URL. However, you may also need to reproduce the body, headers and
-form parameters (see :class:`~scrapy.http.FormRequest`) of that request.
+form parameters (see :class:`~scrapy.FormRequest`) of that request.
 
 As all major browsers allow to export the requests in `cURL
 <https://curl.haxx.se/>`_ format, Scrapy incorporates the method
-:meth:`~scrapy.http.Request.from_curl()` to generate an equivalent
-:class:`~scrapy.http.Request` from a cURL command. To get more information
+:meth:`~scrapy.Request.from_curl()` to generate an equivalent
+:class:`~scrapy.Request` from a cURL command. To get more information
 visit :ref:`request from curl <requests-from-curl>` inside the network
 tool section.
 
@@ -125,7 +125,7 @@ data from it depends on the type of response:
 
     If the desired data is inside HTML or XML code embedded within JSON data,
     you can load that HTML or XML code into a
-    :class:`~scrapy.selector.Selector` and then
+    :class:`~scrapy.Selector` and then
     :ref:`use it <topics-selectors>` as usual::
 
         selector = Selector(data['html'])
diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst
index 8648daded90..db6d5e341ae 100644
--- a/docs/topics/exporters.rst
+++ b/docs/topics/exporters.rst
@@ -172,7 +172,7 @@ BaseItemExporter
       :param field: the field being serialized. If the source :ref:`item object
           <item-types>` does not define field metadata, *field* is an empty
           :class:`dict`.
-      :type field: :class:`~scrapy.item.Field` object or a :class:`dict` instance
+      :type field: :class:`~scrapy.Field` object or a :class:`dict` instance
 
       :param name: the name of the field being serialized
       :type name: str
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index e772a461cf7..de0cf1c2cec 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -616,9 +616,9 @@ The function signature should be as follows:
    :type params: dict
 
    :param spider: source spider of the feed items
-   :type spider: scrapy.spiders.Spider
+   :type spider: scrapy.Spider
 
-For example, to include the :attr:`name <scrapy.spiders.Spider.name>` of the
+For example, to include the :attr:`name <scrapy.Spider.name>` of the
 source spider in the feed URI:
 
 #.  Define the following function somewhere in your project::
diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst
index 6287ee0ad07..5351a2293ba 100644
--- a/docs/topics/item-pipeline.rst
+++ b/docs/topics/item-pipeline.rst
@@ -42,7 +42,7 @@ Each item pipeline component is a Python class that must implement the following
    :type item: :ref:`item object <item-types>`
 
    :param spider: the spider which scraped the item
-   :type spider: :class:`~scrapy.spiders.Spider` object
+   :type spider: :class:`~scrapy.Spider` object
 
 Additionally, they may also implement the following methods:
 
@@ -51,14 +51,14 @@ Additionally, they may also implement the following methods:
    This method is called when the spider is opened.
 
    :param spider: the spider which was opened
-   :type spider: :class:`~scrapy.spiders.Spider` object
+   :type spider: :class:`~scrapy.Spider` object
 
 .. method:: close_spider(self, spider)
 
    This method is called when the spider is closed.
 
    :param spider: the spider which was closed
-   :type spider: :class:`~scrapy.spiders.Spider` object
+   :type spider: :class:`~scrapy.Spider` object
 
 .. method:: from_crawler(cls, crawler)
 
diff --git a/docs/topics/items.rst b/docs/topics/items.rst
index 65bf156ac22..466f86617b3 100644
--- a/docs/topics/items.rst
+++ b/docs/topics/items.rst
@@ -42,7 +42,8 @@ Item objects
 :class:`Item` provides a :class:`dict`-like API plus additional features that
 make it the most feature-complete item type:
 
-.. class:: Item([arg])
+.. class:: scrapy.item.Item([arg])
+.. class:: scrapy.Item([arg])
 
     :class:`Item` objects replicate the standard :class:`dict` API, including
     its ``__init__`` method.
@@ -199,7 +200,8 @@ It's important to note that the :class:`Field` objects used to declare the item
 do not stay assigned as class attributes. Instead, they can be accessed through
 the :attr:`Item.fields` attribute.
 
-.. class:: Field([arg])
+.. class:: scrapy.item.Field([arg])
+.. class:: scrapy.Field([arg])
 
     The :class:`Field` class is just an alias to the built-in :class:`dict` class and
     doesn't provide any extra functionality or attributes. In other words,
diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst
index d855d01334f..e49f37a2fdb 100644
--- a/docs/topics/jobs.rst
+++ b/docs/topics/jobs.rst
@@ -74,10 +74,10 @@ on cookies.
 Request serialization
 ---------------------
 
-For persistence to work, :class:`~scrapy.http.Request` objects must be
+For persistence to work, :class:`~scrapy.Request` objects must be
 serializable with :mod:`pickle`, except for the ``callback`` and ``errback``
 values passed to their ``__init__`` method, which must be methods of the
-running :class:`~scrapy.spiders.Spider` class.
+running :class:`~scrapy.Spider` class.
 
 If you wish to log the requests that couldn't be serialized, you can set the
 :setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.
diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst
index b895b95cbc1..4776527042c 100644
--- a/docs/topics/leaks.rst
+++ b/docs/topics/leaks.rst
@@ -27,7 +27,7 @@ Common causes of memory leaks
 
 It happens quite often (sometimes by accident, sometimes on purpose) that the
 Scrapy developer passes objects referenced in Requests (for example, using the
-:attr:`~scrapy.http.Request.cb_kwargs` or :attr:`~scrapy.http.Request.meta`
+:attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta`
 attributes or the request callback function) and that effectively bounds the
 lifetime of those referenced objects to the lifetime of the Request. This is,
 by far, the most common cause of memory leaks in Scrapy projects, and a quite
@@ -48,9 +48,9 @@ Too Many Requests?
 ------------------
 
 By default Scrapy keeps the request queue in memory; it includes
-:class:`~scrapy.http.Request` objects and all objects
-referenced in Request attributes (e.g. in :attr:`~scrapy.http.Request.cb_kwargs`
-and :attr:`~scrapy.http.Request.meta`).
+:class:`~scrapy.Request` objects and all objects
+referenced in Request attributes (e.g. in :attr:`~scrapy.Request.cb_kwargs`
+and :attr:`~scrapy.Request.meta`).
 While not necessarily a leak, this can take a lot of memory. Enabling
 :ref:`persistent job queue <topics-jobs>` could help keeping memory usage
 in control.
@@ -90,11 +90,11 @@ Which objects are tracked?
 The objects tracked by ``trackrefs`` are all from these classes (and all its
 subclasses):
 
-* :class:`scrapy.http.Request`
+* :class:`scrapy.Request`
 * :class:`scrapy.http.Response`
-* :class:`scrapy.item.Item`
-* :class:`scrapy.selector.Selector`
-* :class:`scrapy.spiders.Spider`
+* :class:`scrapy.Item`
+* :class:`scrapy.Selector`
+* :class:`scrapy.Spider`
 
 A real example
 --------------
diff --git a/docs/topics/logging.rst b/docs/topics/logging.rst
index 00806392a67..dda04dc4dc7 100644
--- a/docs/topics/logging.rst
+++ b/docs/topics/logging.rst
@@ -93,7 +93,7 @@ path::
 Logging from Spiders
 ====================
 
-Scrapy provides a :data:`~scrapy.spiders.Spider.logger` within each Spider
+Scrapy provides a :data:`~scrapy.Spider.logger` within each Spider
 instance, which can be accessed and used like this::
 
     import scrapy
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst
index 156897274a7..3438cb63794 100644
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@@ -259,7 +259,7 @@ respectively), the pipeline will put the results under the respective field
 When using :ref:`item types <item-types>` for which fields are defined beforehand,
 you must define both the URLs field and the results field. For example, when
 using the images pipeline, items must define both the ``image_urls`` and the
-``images`` field. For instance, using the :class:`~scrapy.item.Item` class::
+``images`` field. For instance, using the :class:`~scrapy.Item` class::
 
     import scrapy
 
@@ -424,7 +424,7 @@ See here the methods that you can override in your custom Files Pipeline:
       In addition to ``response``, this method receives the original
       :class:`request <scrapy.Request>`,
       :class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and 
-      :class:`item <scrapy.item.Item>`
+      :class:`item <scrapy.Item>`
 
       You can override this method to customize the download path of each file.
 
@@ -563,7 +563,7 @@ See here the methods that you can override in your custom Images Pipeline:
       In addition to ``response``, this method receives the original
       :class:`request <scrapy.Request>`,
       :class:`info <scrapy.pipelines.media.MediaPipeline.SpiderInfo>` and 
-      :class:`item <scrapy.item.Item>`
+      :class:`item <scrapy.Item>`
 
       You can override this method to customize the download path of each file.
 
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 732eba5870e..9ce8a531390 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -1,264 +1,264 @@
-.. _topics-practices:
-
-================
-Common Practices
-================
-
-This section documents common practices when using Scrapy. These are things
-that cover many topics and don't often fall into any other specific section.
-
-.. _run-from-script:
-
-Run Scrapy from a script
-========================
-
-You can use the :ref:`API <topics-api>` to run Scrapy from a script, instead of
-the typical way of running Scrapy via ``scrapy crawl``.
-
-Remember that Scrapy is built on top of the Twisted
-asynchronous networking library, so you need to run it inside the Twisted reactor.
-
-The first utility you can use to run your spiders is
-:class:`scrapy.crawler.CrawlerProcess`. This class will start a Twisted reactor
-for you, configuring the logging and setting shutdown handlers. This class is
-the one used by all Scrapy commands.
-
-Here's an example showing how to run a single spider with it.
-
-::
-
-    import scrapy
-    from scrapy.crawler import CrawlerProcess
-
-    class MySpider(scrapy.Spider):
-        # Your spider definition
-        ...
-
-    process = CrawlerProcess(settings={
-        "FEEDS": {
-            "items.json": {"format": "json"},
-        },
-    })
-
-    process.crawl(MySpider)
-    process.start() # the script will block here until the crawling is finished
-
-Define settings within dictionary in CrawlerProcess. Make sure to check :class:`~scrapy.crawler.CrawlerProcess`
-documentation to get acquainted with its usage details.
-
-If you are inside a Scrapy project there are some additional helpers you can
-use to import those components within the project. You can automatically import
-your spiders passing their name to :class:`~scrapy.crawler.CrawlerProcess`, and
-use ``get_project_settings`` to get a :class:`~scrapy.settings.Settings`
-instance with your project settings.
-
-What follows is a working example of how to do that, using the `testspiders`_
-project as example.
-
-::
-
-    from scrapy.crawler import CrawlerProcess
-    from scrapy.utils.project import get_project_settings
-
-    process = CrawlerProcess(get_project_settings())
-
-    # 'followall' is the name of one of the spiders of the project.
-    process.crawl('followall', domain='scrapy.org')
-    process.start() # the script will block here until the crawling is finished
-
-There's another Scrapy utility that provides more control over the crawling
-process: :class:`scrapy.crawler.CrawlerRunner`. This class is a thin wrapper
-that encapsulates some simple helpers to run multiple crawlers, but it won't
-start or interfere with existing reactors in any way.
-
-Using this class the reactor should be explicitly run after scheduling your
-spiders. It's recommended you use :class:`~scrapy.crawler.CrawlerRunner`
-instead of :class:`~scrapy.crawler.CrawlerProcess` if your application is
-already using Twisted and you want to run Scrapy in the same reactor.
-
-Note that you will also have to shutdown the Twisted reactor yourself after the
-spider is finished. This can be achieved by adding callbacks to the deferred
-returned by the :meth:`CrawlerRunner.crawl
-<scrapy.crawler.CrawlerRunner.crawl>` method.
-
-Here's an example of its usage, along with a callback to manually stop the
-reactor after ``MySpider`` has finished running.
-
-::
-
-    from twisted.internet import reactor
-    import scrapy
-    from scrapy.crawler import CrawlerRunner
-    from scrapy.utils.log import configure_logging
-
-    class MySpider(scrapy.Spider):
-        # Your spider definition
-        ...
-
-    configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
-    runner = CrawlerRunner()
-
-    d = runner.crawl(MySpider)
-    d.addBoth(lambda _: reactor.stop())
-    reactor.run() # the script will block here until the crawling is finished
-
-.. seealso:: :doc:`twisted:core/howto/reactor-basics`
-
-.. _run-multiple-spiders:
-
-Running multiple spiders in the same process
-============================================
-
-By default, Scrapy runs a single spider per process when you run ``scrapy
-crawl``. However, Scrapy supports running multiple spiders per process using
-the :ref:`internal API <topics-api>`.
-
-Here is an example that runs multiple spiders simultaneously:
-
-::
-
-    import scrapy
-    from scrapy.crawler import CrawlerProcess
-    from scrapy.utils.project import get_project_settings
-
-    class MySpider1(scrapy.Spider):
-        # Your first spider definition
-        ...
-
-    class MySpider2(scrapy.Spider):
-        # Your second spider definition
-        ...
-
-    settings = get_project_settings()
-    process = CrawlerProcess(settings)
-    process.crawl(MySpider1)
-    process.crawl(MySpider2)
-    process.start() # the script will block here until all crawling jobs are finished
-
-Same example using :class:`~scrapy.crawler.CrawlerRunner`:
-
-::
-
-    import scrapy
-    from twisted.internet import reactor
-    from scrapy.crawler import CrawlerRunner
-    from scrapy.utils.log import configure_logging
-    from scrapy.utils.project import get_project_settings
-
-    class MySpider1(scrapy.Spider):
-        # Your first spider definition
-        ...
-
-    class MySpider2(scrapy.Spider):
-        # Your second spider definition
-        ...
-
-    configure_logging()
-    settings = get_project_settings()
-    runner = CrawlerRunner(settings)
-    runner.crawl(MySpider1)
-    runner.crawl(MySpider2)
-    d = runner.join()
-    d.addBoth(lambda _: reactor.stop())
-
-    reactor.run() # the script will block here until all crawling jobs are finished
-
-Same example but running the spiders sequentially by chaining the deferreds:
-
-::
-
-    from twisted.internet import reactor, defer
-    from scrapy.crawler import CrawlerRunner
-    from scrapy.utils.log import configure_logging
-    from scrapy.utils.project import get_project_settings
-
-    class MySpider1(scrapy.Spider):
-        # Your first spider definition
-        ...
-
-    class MySpider2(scrapy.Spider):
-        # Your second spider definition
-        ...
-
-    configure_logging()
-    settings = get_project_settings()
-    runner = CrawlerRunner(settings)
-
-    @defer.inlineCallbacks
-    def crawl():
-        yield runner.crawl(MySpider1)
-        yield runner.crawl(MySpider2)
-        reactor.stop()
-
-    crawl()
-    reactor.run() # the script will block here until the last crawl call is finished
-
-.. seealso:: :ref:`run-from-script`.
-
-.. _distributed-crawls:
-
-Distributed crawls
-==================
-
-Scrapy doesn't provide any built-in facility for running crawls in a distribute
-(multi-server) manner. However, there are some ways to distribute crawls, which
-vary depending on how you plan to distribute them.
-
-If you have many spiders, the obvious way to distribute the load is to setup
-many Scrapyd instances and distribute spider runs among those.
-
-If you instead want to run a single (big) spider through many machines, what
-you usually do is partition the urls to crawl and send them to each separate
-spider. Here is a concrete example:
-
-First, you prepare the list of urls to crawl and put them into separate
-files/urls::
-
-    http://somedomain.com/urls-to-crawl/spider1/part1.list
-    http://somedomain.com/urls-to-crawl/spider1/part2.list
-    http://somedomain.com/urls-to-crawl/spider1/part3.list
-
-Then you fire a spider run on 3 different Scrapyd servers. The spider would
-receive a (spider) argument ``part`` with the number of the partition to
-crawl::
-
-    curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1
-    curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
-    curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
-
-.. _bans:
-
-Avoiding getting banned
-=======================
-
-Some websites implement certain measures to prevent bots from crawling them,
-with varying degrees of sophistication. Getting around those measures can be
-difficult and tricky, and may sometimes require special infrastructure. Please
-consider contacting `commercial support`_ if in doubt.
-
-Here are some tips to keep in mind when dealing with these kinds of sites:
-
-* rotate your user agent from a pool of well-known ones from browsers (google
-  around to get a list of them)
-* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
-  cookies to spot bot behaviour
-* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
-* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
-  directly
-* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
-  services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a
-  super proxy that you can attach your own proxies to.
-* use a highly distributed downloader that circumvents bans internally, so you
-  can just focus on parsing clean pages. One example of such downloaders is
-  `Zyte Smart Proxy Manager`_
-
-If you are still unable to prevent your bot getting banned, consider contacting
-`commercial support`_.
-
-.. _Tor project: https://www.torproject.org/
-.. _commercial support: https://scrapy.org/support/
-.. _ProxyMesh: https://proxymesh.com/
-.. _Google cache: http://www.googleguide.com/cached_pages.html
-.. _testspiders: https://github.com/scrapinghub/testspiders
-.. _scrapoxy: https://scrapoxy.io/
-.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/
+.. _topics-practices:
+
+================
+Common Practices
+================
+
+This section documents common practices when using Scrapy. These are things
+that cover many topics and don't often fall into any other specific section.
+
+.. _run-from-script:
+
+Run Scrapy from a script
+========================
+
+You can use the :ref:`API <topics-api>` to run Scrapy from a script, instead of
+the typical way of running Scrapy via ``scrapy crawl``.
+
+Remember that Scrapy is built on top of the Twisted
+asynchronous networking library, so you need to run it inside the Twisted reactor.
+
+The first utility you can use to run your spiders is
+:class:`scrapy.crawler.CrawlerProcess`. This class will start a Twisted reactor
+for you, configuring the logging and setting shutdown handlers. This class is
+the one used by all Scrapy commands.
+
+Here's an example showing how to run a single spider with it.
+
+::
+
+    import scrapy
+    from scrapy.crawler import CrawlerProcess
+
+    class MySpider(scrapy.Spider):
+        # Your spider definition
+        ...
+
+    process = CrawlerProcess(settings={
+        "FEEDS": {
+            "items.json": {"format": "json"},
+        },
+    })
+
+    process.crawl(MySpider)
+    process.start() # the script will block here until the crawling is finished
+
+Define settings within dictionary in CrawlerProcess. Make sure to check :class:`~scrapy.crawler.CrawlerProcess`
+documentation to get acquainted with its usage details.
+
+If you are inside a Scrapy project there are some additional helpers you can
+use to import those components within the project. You can automatically import
+your spiders passing their name to :class:`~scrapy.crawler.CrawlerProcess`, and
+use ``get_project_settings`` to get a :class:`~scrapy.settings.Settings`
+instance with your project settings.
+
+What follows is a working example of how to do that, using the `testspiders`_
+project as example.
+
+::
+
+    from scrapy.crawler import CrawlerProcess
+    from scrapy.utils.project import get_project_settings
+
+    process = CrawlerProcess(get_project_settings())
+
+    # 'followall' is the name of one of the spiders of the project.
+    process.crawl('followall', domain='scrapy.org')
+    process.start() # the script will block here until the crawling is finished
+
+There's another Scrapy utility that provides more control over the crawling
+process: :class:`scrapy.crawler.CrawlerRunner`. This class is a thin wrapper
+that encapsulates some simple helpers to run multiple crawlers, but it won't
+start or interfere with existing reactors in any way.
+
+Using this class the reactor should be explicitly run after scheduling your
+spiders. It's recommended you use :class:`~scrapy.crawler.CrawlerRunner`
+instead of :class:`~scrapy.crawler.CrawlerProcess` if your application is
+already using Twisted and you want to run Scrapy in the same reactor.
+
+Note that you will also have to shutdown the Twisted reactor yourself after the
+spider is finished. This can be achieved by adding callbacks to the deferred
+returned by the :meth:`CrawlerRunner.crawl
+<scrapy.crawler.CrawlerRunner.crawl>` method.
+
+Here's an example of its usage, along with a callback to manually stop the
+reactor after ``MySpider`` has finished running.
+
+::
+
+    from twisted.internet import reactor
+    import scrapy
+    from scrapy.crawler import CrawlerRunner
+    from scrapy.utils.log import configure_logging
+
+    class MySpider(scrapy.Spider):
+        # Your spider definition
+        ...
+
+    configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
+    runner = CrawlerRunner()
+
+    d = runner.crawl(MySpider)
+    d.addBoth(lambda _: reactor.stop())
+    reactor.run() # the script will block here until the crawling is finished
+
+.. seealso:: :doc:`twisted:core/howto/reactor-basics`
+
+.. _run-multiple-spiders:
+
+Running multiple spiders in the same process
+============================================
+
+By default, Scrapy runs a single spider per process when you run ``scrapy
+crawl``. However, Scrapy supports running multiple spiders per process using
+the :ref:`internal API <topics-api>`.
+
+Here is an example that runs multiple spiders simultaneously:
+
+::
+
+    import scrapy
+    from scrapy.crawler import CrawlerProcess
+    from scrapy.utils.project import get_project_settings
+
+    class MySpider1(scrapy.Spider):
+        # Your first spider definition
+        ...
+
+    class MySpider2(scrapy.Spider):
+        # Your second spider definition
+        ...
+
+    settings = get_project_settings()
+    process = CrawlerProcess(settings)
+    process.crawl(MySpider1)
+    process.crawl(MySpider2)
+    process.start() # the script will block here until all crawling jobs are finished
+
+Same example using :class:`~scrapy.crawler.CrawlerRunner`:
+
+::
+
+    import scrapy
+    from twisted.internet import reactor
+    from scrapy.crawler import CrawlerRunner
+    from scrapy.utils.log import configure_logging
+    from scrapy.utils.project import get_project_settings
+
+    class MySpider1(scrapy.Spider):
+        # Your first spider definition
+        ...
+
+    class MySpider2(scrapy.Spider):
+        # Your second spider definition
+        ...
+
+    configure_logging()
+    settings = get_project_settings()
+    runner = CrawlerRunner(settings)
+    runner.crawl(MySpider1)
+    runner.crawl(MySpider2)
+    d = runner.join()
+    d.addBoth(lambda _: reactor.stop())
+
+    reactor.run() # the script will block here until all crawling jobs are finished
+
+Same example but running the spiders sequentially by chaining the deferreds:
+
+::
+
+    from twisted.internet import reactor, defer
+    from scrapy.crawler import CrawlerRunner
+    from scrapy.utils.log import configure_logging
+    from scrapy.utils.project import get_project_settings
+
+    class MySpider1(scrapy.Spider):
+        # Your first spider definition
+        ...
+
+    class MySpider2(scrapy.Spider):
+        # Your second spider definition
+        ...
+
+    configure_logging()
+    settings = get_project_settings()
+    runner = CrawlerRunner(settings)
+
+    @defer.inlineCallbacks
+    def crawl():
+        yield runner.crawl(MySpider1)
+        yield runner.crawl(MySpider2)
+        reactor.stop()
+
+    crawl()
+    reactor.run() # the script will block here until the last crawl call is finished
+
+.. seealso:: :ref:`run-from-script`.
+
+.. _distributed-crawls:
+
+Distributed crawls
+==================
+
+Scrapy doesn't provide any built-in facility for running crawls in a distribute
+(multi-server) manner. However, there are some ways to distribute crawls, which
+vary depending on how you plan to distribute them.
+
+If you have many spiders, the obvious way to distribute the load is to setup
+many Scrapyd instances and distribute spider runs among those.
+
+If you instead want to run a single (big) spider through many machines, what
+you usually do is partition the urls to crawl and send them to each separate
+spider. Here is a concrete example:
+
+First, you prepare the list of urls to crawl and put them into separate
+files/urls::
+
+    http://somedomain.com/urls-to-crawl/spider1/part1.list
+    http://somedomain.com/urls-to-crawl/spider1/part2.list
+    http://somedomain.com/urls-to-crawl/spider1/part3.list
+
+Then you fire a spider run on 3 different Scrapyd servers. The spider would
+receive a (spider) argument ``part`` with the number of the partition to
+crawl::
+
+    curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1
+    curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
+    curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
+
+.. _bans:
+
+Avoiding getting banned
+=======================
+
+Some websites implement certain measures to prevent bots from crawling them,
+with varying degrees of sophistication. Getting around those measures can be
+difficult and tricky, and may sometimes require special infrastructure. Please
+consider contacting `commercial support`_ if in doubt.
+
+Here are some tips to keep in mind when dealing with these kinds of sites:
+
+* rotate your user agent from a pool of well-known ones from browsers (google
+  around to get a list of them)
+* disable cookies (see :setting:`COOKIES_ENABLED`) as some sites may use
+  cookies to spot bot behaviour
+* use download delays (2 or higher). See :setting:`DOWNLOAD_DELAY` setting.
+* if possible, use `Google cache`_ to fetch pages, instead of hitting the sites
+  directly
+* use a pool of rotating IPs. For example, the free `Tor project`_ or paid
+  services like `ProxyMesh`_. An open source alternative is `scrapoxy`_, a
+  super proxy that you can attach your own proxies to.
+* use a highly distributed downloader that circumvents bans internally, so you
+  can just focus on parsing clean pages. One example of such downloaders is
+  `Zyte Smart Proxy Manager`_
+
+If you are still unable to prevent your bot getting banned, consider contacting
+`commercial support`_.
+
+.. _Tor project: https://www.torproject.org/
+.. _commercial support: https://scrapy.org/support/
+.. _ProxyMesh: https://proxymesh.com/
+.. _Google cache: http://www.googleguide.com/cached_pages.html
+.. _testspiders: https://github.com/scrapinghub/testspiders
+.. _scrapoxy: https://scrapoxy.io/
+.. _Zyte Smart Proxy Manager: https://www.zyte.com/smart-proxy-manager/
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 500781c0553..51b155a37c0 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -39,7 +39,7 @@ Request objects
        request (once it's downloaded) as its first parameter. For more information
        see :ref:`topics-request-response-ref-request-callback-arguments` below.
        If a Request doesn't specify a callback, the spider's
-       :meth:`~scrapy.spiders.Spider.parse` method will be used.
+       :meth:`~scrapy.Spider.parse` method will be used.
        Note that if exceptions are raised during processing, errback is called instead.
 
     :type callback: collections.abc.Callable
@@ -64,7 +64,7 @@ Request objects
 
         .. caution:: Cookies set via the ``Cookie`` header are not considered by the
             :ref:`cookies-mw`. If you need to set cookies for a request, use the
-            :class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
+            :class:`Request.cookies <scrapy.Request>` parameter. This is a known
             current limitation that is being worked on.
 
     :type headers: dict
@@ -96,7 +96,7 @@ Request objects
 
         To create a request that does not send stored cookies and does not
         store received cookies, set the ``dont_merge_cookies`` key to ``True``
-        in :attr:`request.meta <scrapy.http.Request.meta>`.
+        in :attr:`request.meta <scrapy.Request.meta>`.
 
         Example of a request that sends manually-defined cookies and ignores
         cookie storage::
@@ -111,7 +111,7 @@ Request objects
 
         .. caution:: Cookies set via the ``Cookie`` header are not considered by the
             :ref:`cookies-mw`. If you need to set cookies for a request, use the
-            :class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
+            :class:`Request.cookies <scrapy.Request>` parameter. This is a known
             current limitation that is being worked on.
 
     :type cookies: dict or list
@@ -488,7 +488,9 @@ fields with form data from :class:`Response` objects.
 
 .. _lxml.html forms: https://lxml.de/lxmlhtml.html#forms
 
-.. class:: FormRequest(url, [formdata, ...])
+.. class:: scrapy.http.request.form.FormRequest
+.. class:: scrapy.http.FormRequest
+.. class:: scrapy.FormRequest(url, [formdata, ...])
 
     The :class:`FormRequest` class adds a new keyword parameter to the ``__init__`` method. The
     remaining arguments are the same as for the :class:`Request` class and are
@@ -685,7 +687,7 @@ Response objects
 
     :param request: the initial value of the :attr:`Response.request` attribute.
         This represents the :class:`Request` that generated this response.
-    :type request: scrapy.http.Request
+    :type request: scrapy.Request
 
     :param certificate: an object representing the server's SSL certificate.
     :type certificate: twisted.internet.ssl.Certificate
@@ -911,7 +913,7 @@ TextResponse objects
 
     .. attribute:: TextResponse.selector
 
-        A :class:`~scrapy.selector.Selector` instance using the response as
+        A :class:`~scrapy.Selector` instance using the response as
         target. The selector is lazily instantiated on first access.
 
     :class:`TextResponse` objects support the following methods in addition to
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index 9caba5ee583..574d4568c1d 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -48,7 +48,7 @@ Constructing selectors
 
 .. highlight:: python
 
-Response objects expose a :class:`~scrapy.selector.Selector` instance
+Response objects expose a :class:`~scrapy.Selector` instance
 on ``.selector`` attribute:
 
 >>> response.selector.xpath('//span/text()').get()
@@ -62,7 +62,7 @@ more shortcuts: ``response.xpath()`` and ``response.css()``:
 >>> response.css('span::text').get()
 'good'
 
-Scrapy selectors are instances of :class:`~scrapy.selector.Selector` class
+Scrapy selectors are instances of :class:`~scrapy.Selector` class
 constructed by passing either :class:`~scrapy.http.TextResponse` object or
 markup as a string (in ``text`` argument).
 
@@ -175,7 +175,7 @@ of ``None``:
 'not-found'
 
 Instead of using e.g. ``'@src'`` XPath it is possible to query for attributes
-using ``.attrib`` property of a :class:`~scrapy.selector.Selector`:
+using ``.attrib`` property of a :class:`~scrapy.Selector`:
 
 >>> [img.attrib['src'] for img in response.css('img')]
 ['image1_thumb.jpg',
@@ -383,7 +383,7 @@ ID, or when selecting an unique element on a page):
 Using selectors with regular expressions
 ----------------------------------------
 
-:class:`~scrapy.selector.Selector` also has a ``.re()`` method for extracting
+:class:`~scrapy.Selector` also has a ``.re()`` method for extracting
 data using regular expressions. However, unlike using ``.xpath()`` or
 ``.css()`` methods, ``.re()`` returns a list of strings. So you
 can't construct nested ``.re()`` calls.
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 1d5babcec64..79f6ddebd58 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -67,7 +67,7 @@ Example::
 
 Spiders (See the :ref:`topics-spiders` chapter for reference) can define their
 own settings that will take precedence and override the project ones. They can
-do so by setting their :attr:`~scrapy.spiders.Spider.custom_settings` attribute::
+do so by setting their :attr:`~scrapy.Spider.custom_settings` attribute::
 
     class MySpider(scrapy.Spider):
         name = 'myspider'
@@ -142,7 +142,7 @@ In a spider, the settings are available through ``self.settings``::
     The ``settings`` attribute is set in the base Spider class after the spider
     is initialized.  If you want to use the settings before the initialization
     (e.g., in your spider's ``__init__()`` method), you'll need to override the
-    :meth:`~scrapy.spiders.Spider.from_crawler` method.
+    :meth:`~scrapy.Spider.from_crawler` method.
 
 Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings`
 attribute of the Crawler that is passed to ``from_crawler`` method in
@@ -338,7 +338,7 @@ is non-zero, download delay is enforced per IP, not per domain.
 DEFAULT_ITEM_CLASS
 ------------------
 
-Default: ``'scrapy.item.Item'``
+Default: ``'scrapy.Item'``
 
 The default class that will be used for instantiating items in the :ref:`the
 Scrapy shell <topics-shell>`.
@@ -360,7 +360,7 @@ The default headers used for Scrapy HTTP Requests. They're populated in the
 
 .. caution:: Cookies set via the ``Cookie`` header are not considered by the
     :ref:`cookies-mw`. If you need to set cookies for a request, use the
-    :class:`Request.cookies <scrapy.http.Request>` parameter. This is a known
+    :class:`Request.cookies <scrapy.Request>` parameter. This is a known
     current limitation that is being worked on.
 
 .. setting:: DEPTH_LIMIT
@@ -384,8 +384,8 @@ Default: ``0``
 
 Scope: ``scrapy.spidermiddlewares.depth.DepthMiddleware``
 
-An integer that is used to adjust the :attr:`~scrapy.http.Request.priority` of
-a :class:`~scrapy.http.Request` based on its depth.
+An integer that is used to adjust the :attr:`~scrapy.Request.priority` of
+a :class:`~scrapy.Request` based on its depth.
 
 The priority of a request is adjusted as follows::
 
@@ -812,14 +812,14 @@ The default (``RFPDupeFilter``) filters based on request fingerprint using
 the ``scrapy.utils.request.request_fingerprint`` function. In order to change
 the way duplicates are checked you could subclass ``RFPDupeFilter`` and
 override its ``request_fingerprint`` method. This method should accept
-scrapy :class:`~scrapy.http.Request` object and return its fingerprint
+scrapy :class:`~scrapy.Request` object and return its fingerprint
 (a string).
 
 You can disable filtering of duplicate requests by setting
 :setting:`DUPEFILTER_CLASS` to ``'scrapy.dupefilters.BaseDupeFilter'``.
 Be very careful about this however, because you can get into crawling loops.
 It's usually a better idea to set the ``dont_filter`` parameter to
-``True`` on the specific :class:`~scrapy.http.Request` that should not be
+``True`` on the specific :class:`~scrapy.Request` that should not be
 filtered.
 
 .. setting:: DUPEFILTER_DEBUG
diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst
index b910fc453fa..8c90a506ca1 100644
--- a/docs/topics/shell.rst
+++ b/docs/topics/shell.rst
@@ -118,7 +118,7 @@ Available Scrapy objects
 
 The Scrapy shell automatically creates some convenient objects from the
 downloaded page, like the :class:`~scrapy.http.Response` object and the
-:class:`~scrapy.selector.Selector` objects (for both HTML and XML
+:class:`~scrapy.Selector` objects (for both HTML and XML
 content).
 
 Those objects are:
@@ -126,12 +126,12 @@ Those objects are:
 -    ``crawler`` - the current :class:`~scrapy.crawler.Crawler` object.
 
 -   ``spider`` - the Spider which is known to handle the URL, or a
-    :class:`~scrapy.spiders.Spider` object if there is no spider found for the
+    :class:`~scrapy.Spider` object if there is no spider found for the
     current URL
 
--   ``request`` - a :class:`~scrapy.http.Request` object of the last fetched
+-   ``request`` - a :class:`~scrapy.Request` object of the last fetched
     page. You can modify this request using
-    :meth:`~scrapy.http.Request.replace` or fetch a new request (without
+    :meth:`~scrapy.Request.replace` or fetch a new request (without
     leaving the shell) using the ``fetch`` shortcut.
 
 -   ``response`` - a :class:`~scrapy.http.Response` object containing the last
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index 3d838fb634b..f71f13c1a34 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -155,7 +155,7 @@ item_scraped
     :type item: :ref:`item object <item-types>`
 
     :param spider: the spider which scraped the item
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
     :param response: the response from where the item was scraped
     :type response: :class:`~scrapy.http.Response` object
@@ -175,7 +175,7 @@ item_dropped
     :type item: :ref:`item object <item-types>`
 
     :param spider: the spider which scraped the item
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
     :param response: the response from where the item was dropped
     :type response: :class:`~scrapy.http.Response` object
@@ -203,7 +203,7 @@ item_error
     :type response: :class:`~scrapy.http.Response` object
 
     :param spider: the spider which raised the exception
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
     :param failure: the exception raised
     :type failure: twisted.python.failure.Failure
@@ -223,7 +223,7 @@ spider_closed
     This signal supports returning deferreds from its handlers.
 
     :param spider: the spider which has been closed
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
     :param reason: a string which describes the reason why the spider was closed. If
         it was closed because the spider has completed scraping, the reason
@@ -247,7 +247,7 @@ spider_opened
     This signal supports returning deferreds from its handlers.
 
     :param spider: the spider which has been opened
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 spider_idle
 ~~~~~~~~~~~
@@ -271,7 +271,7 @@ spider_idle
     This signal does not support returning deferreds from its handlers.
 
     :param spider: the spider which has gone idle
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 .. note:: Scheduling some requests in your :signal:`spider_idle` handler does
     **not** guarantee that it can prevent the spider from being closed,
@@ -296,7 +296,7 @@ spider_error
     :type response: :class:`~scrapy.http.Response` object
 
     :param spider: the spider which raised the exception
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 Request signals
 ---------------
@@ -307,16 +307,16 @@ request_scheduled
 .. signal:: request_scheduled
 .. function:: request_scheduled(request, spider)
 
-    Sent when the engine schedules a :class:`~scrapy.http.Request`, to be
+    Sent when the engine schedules a :class:`~scrapy.Request`, to be
     downloaded later.
 
     This signal does not support returning deferreds from its handlers.
 
     :param request: the request that reached the scheduler
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider that yielded the request
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 request_dropped
 ~~~~~~~~~~~~~~~
@@ -324,16 +324,16 @@ request_dropped
 .. signal:: request_dropped
 .. function:: request_dropped(request, spider)
 
-    Sent when a :class:`~scrapy.http.Request`, scheduled by the engine to be
+    Sent when a :class:`~scrapy.Request`, scheduled by the engine to be
     downloaded later, is rejected by the scheduler.
 
     This signal does not support returning deferreds from its handlers.
 
     :param request: the request that reached the scheduler
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider that yielded the request
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 request_reached_downloader
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -341,15 +341,15 @@ request_reached_downloader
 .. signal:: request_reached_downloader
 .. function:: request_reached_downloader(request, spider)
 
-    Sent when a :class:`~scrapy.http.Request` reached downloader.
+    Sent when a :class:`~scrapy.Request` reached downloader.
 
     This signal does not support returning deferreds from its handlers.
 
     :param request: the request that reached downloader
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider that yielded the request
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 request_left_downloader
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -359,16 +359,16 @@ request_left_downloader
 
     .. versionadded:: 2.0
 
-    Sent when a :class:`~scrapy.http.Request` leaves the downloader, even in case of
+    Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
     failure.
 
     This signal does not support returning deferreds from its handlers.
 
     :param request: the request that reached the downloader
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider that yielded the request
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 bytes_received
 ~~~~~~~~~~~~~~
@@ -395,10 +395,10 @@ bytes_received
     :type data: :class:`bytes` object
 
     :param request: the request that generated the download
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider associated with the response
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 headers_received
 ~~~~~~~~~~~~~~~~
@@ -425,10 +425,10 @@ headers_received
     :type body_length: `int`
 
     :param request: the request that generated the download
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider associated with the response
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 Response signals
 ----------------
@@ -448,10 +448,10 @@ response_received
     :type response: :class:`~scrapy.http.Response` object
 
     :param request: the request that generated the response
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider for which the response is intended
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
 
 .. note:: The ``request`` argument might not contain the original request that
     reached the downloader, if a :ref:`topics-downloader-middleware` modifies
@@ -472,7 +472,7 @@ response_downloaded
     :type response: :class:`~scrapy.http.Response` object
 
     :param request: the request that generated the response
-    :type request: :class:`~scrapy.http.Request` object
+    :type request: :class:`~scrapy.Request` object
 
     :param spider: the spider for which the response is intended
-    :type spider: :class:`~scrapy.spiders.Spider` object
+    :type spider: :class:`~scrapy.Spider` object
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 11bbbb58d0e..f0158dc41a0 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -93,7 +93,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
         :type response: :class:`~scrapy.http.Response` object
 
         :param spider: the spider for which this response is intended
-        :type spider: :class:`~scrapy.spiders.Spider` object
+        :type spider: :class:`~scrapy.Spider` object
 
 
     .. method:: process_spider_output(response, result, spider)
@@ -102,7 +102,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
         it has processed the response.
 
         :meth:`process_spider_output` must return an iterable of
-        :class:`~scrapy.http.Request` objects and :ref:`item object
+        :class:`~scrapy.Request` objects and :ref:`item object
         <topics-items>`.
 
         :param response: the response which generated this output from the
@@ -110,11 +110,11 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
         :type response: :class:`~scrapy.http.Response` object
 
         :param result: the result returned by the spider
-        :type result: an iterable of :class:`~scrapy.http.Request` objects and
+        :type result: an iterable of :class:`~scrapy.Request` objects and
           :ref:`item object <topics-items>`
 
         :param spider: the spider whose result is being processed
-        :type spider: :class:`~scrapy.spiders.Spider` object
+        :type spider: :class:`~scrapy.Spider` object
 
     .. method:: process_spider_exception(response, exception, spider)
 
@@ -122,7 +122,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
         method (from a previous spider middleware) raises an exception.
 
         :meth:`process_spider_exception` should return either ``None`` or an
-        iterable of :class:`~scrapy.http.Request` objects and :ref:`item object
+        iterable of :class:`~scrapy.Request` objects and :ref:`item object
         <topics-items>`.
 
         If it returns ``None``, Scrapy will continue processing this exception,
@@ -142,7 +142,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
         :type exception: :exc:`Exception` object
 
         :param spider: the spider which raised the exception
-        :type spider: :class:`~scrapy.spiders.Spider` object
+        :type spider: :class:`~scrapy.Spider` object
 
     .. method:: process_start_requests(start_requests, spider)
 
@@ -152,7 +152,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
         items).
 
         It receives an iterable (in the ``start_requests`` parameter) and must
-        return another iterable of :class:`~scrapy.http.Request` objects.
+        return another iterable of :class:`~scrapy.Request` objects.
 
         .. note:: When implementing this method in your spider middleware, you
            should always return an iterable (that follows the input one) and
@@ -164,10 +164,10 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
            (like a time limit or item/page count).
 
         :param start_requests: the start requests
-        :type start_requests: an iterable of :class:`~scrapy.http.Request`
+        :type start_requests: an iterable of :class:`~scrapy.Request`
 
         :param spider: the spider to whom the start requests belong
-        :type spider: :class:`~scrapy.spiders.Spider` object
+        :type spider: :class:`~scrapy.Spider` object
 
     .. method:: from_crawler(cls, crawler)
 
@@ -251,7 +251,7 @@ this::
 .. reqmeta:: handle_httpstatus_all
 
 The ``handle_httpstatus_list`` key of :attr:`Request.meta
-<scrapy.http.Request.meta>` can also be used to specify which response codes to
+<scrapy.Request.meta>` can also be used to specify which response codes to
 allow on a per-request basis. You can also set the meta key ``handle_httpstatus_all``
 to ``True`` if you want to allow any response code for a request, and ``False`` to
 disable the effects of the ``handle_httpstatus_all`` key.
@@ -295,7 +295,7 @@ OffsiteMiddleware
    Filters out Requests for URLs outside the domains covered by the spider.
 
    This middleware filters out every request whose host names aren't in the
-   spider's :attr:`~scrapy.spiders.Spider.allowed_domains` attribute.
+   spider's :attr:`~scrapy.Spider.allowed_domains` attribute.
    All subdomains of any domain in the list are also allowed.
    E.g. the rule ``www.example.org`` will also allow ``bob.www.example.org``
    but not ``www2.example.com`` nor ``example.com``.
@@ -313,10 +313,10 @@ OffsiteMiddleware
    will be printed (but only for the first request filtered).
 
    If the spider doesn't define an
-   :attr:`~scrapy.spiders.Spider.allowed_domains` attribute, or the
+   :attr:`~scrapy.Spider.allowed_domains` attribute, or the
    attribute is empty, the offsite middleware will allow all requests.
 
-   If the request has the :attr:`~scrapy.http.Request.dont_filter` attribute
+   If the request has the :attr:`~scrapy.Request.dont_filter` attribute
    set, the offsite middleware will allow the request even if its domain is not
    listed in allowed domains.
 
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 2056664c7ca..c223461713c 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -1,828 +1,829 @@
-.. _topics-spiders:
-
-=======
-Spiders
-=======
-
-Spiders are classes which define how a certain site (or a group of sites) will be
-scraped, including how to perform the crawl (i.e. follow links) and how to
-extract structured data from their pages (i.e. scraping items). In other words,
-Spiders are the place where you define the custom behaviour for crawling and
-parsing pages for a particular site (or, in some cases, a group of sites).
-
-For spiders, the scraping cycle goes through something like this:
-
-1. You start by generating the initial Requests to crawl the first URLs, and
-   specify a callback function to be called with the response downloaded from
-   those requests.
-
-   The first requests to perform are obtained by calling the
-   :meth:`~scrapy.spiders.Spider.start_requests` method which (by default)
-   generates :class:`~scrapy.http.Request` for the URLs specified in the
-   :attr:`~scrapy.spiders.Spider.start_urls` and the
-   :attr:`~scrapy.spiders.Spider.parse` method as callback function for the
-   Requests.
-
-2. In the callback function, you parse the response (web page) and return
-   :ref:`item objects <topics-items>`,
-   :class:`~scrapy.http.Request` objects, or an iterable of these objects.
-   Those Requests will also contain a callback (maybe
-   the same) and will then be downloaded by Scrapy and then their
-   response handled by the specified callback.
-
-3. In callback functions, you parse the page contents, typically using
-   :ref:`topics-selectors` (but you can also use BeautifulSoup, lxml or whatever
-   mechanism you prefer) and generate items with the parsed data.
-
-4. Finally, the items returned from the spider will be typically persisted to a
-   database (in some :ref:`Item Pipeline <topics-item-pipeline>`) or written to
-   a file using :ref:`topics-feed-exports`.
-
-Even though this cycle applies (more or less) to any kind of spider, there are
-different kinds of default spiders bundled into Scrapy for different purposes.
-We will talk about those types here.
-
-.. module:: scrapy.spiders
-   :synopsis: Spiders base class, spider manager and spider middleware
-
-.. _topics-spiders-ref:
-
-scrapy.Spider
-=============
-
-.. class:: Spider()
-
-   This is the simplest spider, and the one from which every other spider
-   must inherit (including spiders that come bundled with Scrapy, as well as spiders
-   that you write yourself). It doesn't provide any special functionality. It just
-   provides a default :meth:`start_requests` implementation which sends requests from
-   the :attr:`start_urls` spider attribute and calls the spider's method ``parse``
-   for each of the resulting responses.
-
-   .. attribute:: name
-
-       A string which defines the name for this spider. The spider name is how
-       the spider is located (and instantiated) by Scrapy, so it must be
-       unique. However, nothing prevents you from instantiating more than one
-       instance of the same spider. This is the most important spider attribute
-       and it's required.
-
-       If the spider scrapes a single domain, a common practice is to name the
-       spider after the domain, with or without the `TLD`_. So, for example, a
-       spider that crawls ``mywebsite.com`` would often be called
-       ``mywebsite``.
-
-   .. attribute:: allowed_domains
-
-       An optional list of strings containing domains that this spider is
-       allowed to crawl. Requests for URLs not belonging to the domain names
-       specified in this list (or their subdomains) won't be followed if
-       :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled.
-
-       Let's say your target url is ``https://www.example.com/1.html``,
-       then add ``'example.com'`` to the list.
-
-   .. attribute:: start_urls
-
-       A list of URLs where the spider will begin to crawl from, when no
-       particular URLs are specified. So, the first pages downloaded will be those
-       listed here. The subsequent :class:`~scrapy.http.Request` will be generated successively from data
-       contained in the start URLs.
-
-   .. attribute:: custom_settings
-
-      A dictionary of settings that will be overridden from the project wide
-      configuration when running this spider. It must be defined as a class
-      attribute since the settings are updated before instantiation.
-
-      For a list of available built-in settings see:
-      :ref:`topics-settings-ref`.
-
-   .. attribute:: crawler
-
-      This attribute is set by the :meth:`from_crawler` class method after
-      initializating the class, and links to the
-      :class:`~scrapy.crawler.Crawler` object to which this spider instance is
-      bound.
-
-      Crawlers encapsulate a lot of components in the project for their single
-      entry access (such as extensions, middlewares, signals managers, etc).
-      See :ref:`topics-api-crawler` to know more about them.
-
-   .. attribute:: settings
-
-      Configuration for running this spider. This is a
-      :class:`~scrapy.settings.Settings` instance, see the
-      :ref:`topics-settings` topic for a detailed introduction on this subject.
-
-   .. attribute:: logger
-
-      Python logger created with the Spider's :attr:`name`. You can use it to
-      send log messages through it as described on
-      :ref:`topics-logging-from-spiders`.
-
-   .. method:: from_crawler(crawler, *args, **kwargs)
-
-       This is the class method used by Scrapy to create your spiders.
-
-       You probably won't need to override this directly because the default
-       implementation acts as a proxy to the :meth:`__init__` method, calling
-       it with the given arguments ``args`` and named arguments ``kwargs``.
-
-       Nonetheless, this method sets the :attr:`crawler` and :attr:`settings`
-       attributes in the new instance so they can be accessed later inside the
-       spider's code.
-
-       :param crawler: crawler to which the spider will be bound
-       :type crawler: :class:`~scrapy.crawler.Crawler` instance
-
-       :param args: arguments passed to the :meth:`__init__` method
-       :type args: list
-
-       :param kwargs: keyword arguments passed to the :meth:`__init__` method
-       :type kwargs: dict
-
-   .. method:: start_requests()
-
-       This method must return an iterable with the first Requests to crawl for
-       this spider. It is called by Scrapy when the spider is opened for
-       scraping. Scrapy calls it only once, so it is safe to implement
-       :meth:`start_requests` as a generator.
-
-       The default implementation generates ``Request(url, dont_filter=True)``
-       for each url in :attr:`start_urls`.
-
-       If you want to change the Requests used to start scraping a domain, this is
-       the method to override. For example, if you need to start by logging in using
-       a POST request, you could do::
-
-           class MySpider(scrapy.Spider):
-               name = 'myspider'
-
-               def start_requests(self):
-                   return [scrapy.FormRequest("http://www.example.com/login",
-                                              formdata={'user': 'john', 'pass': 'secret'},
-                                              callback=self.logged_in)]
-
-               def logged_in(self, response):
-                   # here you would extract links to follow and return Requests for
-                   # each of them, with another callback
-                   pass
-
-   .. method:: parse(response)
-
-       This is the default callback used by Scrapy to process downloaded
-       responses, when their requests don't specify a callback.
-
-       The ``parse`` method is in charge of processing the response and returning
-       scraped data and/or more URLs to follow. Other Requests callbacks have
-       the same requirements as the :class:`Spider` class.
-
-       This method, as well as any other Request callback, must return an
-       iterable of :class:`~scrapy.http.Request` and/or :ref:`item objects
-       <topics-items>`.
-
-       :param response: the response to parse
-       :type response: :class:`~scrapy.http.Response`
-
-   .. method:: log(message, [level, component])
-
-       Wrapper that sends a log message through the Spider's :attr:`logger`,
-       kept for backward compatibility. For more information see
-       :ref:`topics-logging-from-spiders`.
-
-   .. method:: closed(reason)
-
-       Called when the spider closes. This method provides a shortcut to
-       signals.connect() for the :signal:`spider_closed` signal.
-
-Let's see an example::
-
-    import scrapy
-
-
-    class MySpider(scrapy.Spider):
-        name = 'example.com'
-        allowed_domains = ['example.com']
-        start_urls = [
-            'http://www.example.com/1.html',
-            'http://www.example.com/2.html',
-            'http://www.example.com/3.html',
-        ]
-
-        def parse(self, response):
-            self.logger.info('A response from %s just arrived!', response.url)
-
-Return multiple Requests and items from a single callback::
-
-    import scrapy
-
-    class MySpider(scrapy.Spider):
-        name = 'example.com'
-        allowed_domains = ['example.com']
-        start_urls = [
-            'http://www.example.com/1.html',
-            'http://www.example.com/2.html',
-            'http://www.example.com/3.html',
-        ]
-
-        def parse(self, response):
-            for h3 in response.xpath('//h3').getall():
-                yield {"title": h3}
-
-            for href in response.xpath('//a/@href').getall():
-                yield scrapy.Request(response.urljoin(href), self.parse)
-
-Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly;
-to give data more structure you can use :class:`~scrapy.item.Item` objects::
-
-    import scrapy
-    from myproject.items import MyItem
-
-    class MySpider(scrapy.Spider):
-        name = 'example.com'
-        allowed_domains = ['example.com']
-
-        def start_requests(self):
-            yield scrapy.Request('http://www.example.com/1.html', self.parse)
-            yield scrapy.Request('http://www.example.com/2.html', self.parse)
-            yield scrapy.Request('http://www.example.com/3.html', self.parse)
-
-        def parse(self, response):
-            for h3 in response.xpath('//h3').getall():
-                yield MyItem(title=h3)
-
-            for href in response.xpath('//a/@href').getall():
-                yield scrapy.Request(response.urljoin(href), self.parse)
-
-.. _spiderargs:
-
-Spider arguments
-================
-
-Spiders can receive arguments that modify their behaviour. Some common uses for
-spider arguments are to define the start URLs or to restrict the crawl to
-certain sections of the site, but they can be used to configure any
-functionality of the spider.
-
-Spider arguments are passed through the :command:`crawl` command using the
-``-a`` option. For example::
-
-    scrapy crawl myspider -a category=electronics
-
-Spiders can access arguments in their `__init__` methods::
-
-    import scrapy
-
-    class MySpider(scrapy.Spider):
-        name = 'myspider'
-
-        def __init__(self, category=None, *args, **kwargs):
-            super(MySpider, self).__init__(*args, **kwargs)
-            self.start_urls = [f'http://www.example.com/categories/{category}']
-            # ...
-
-The default `__init__` method will take any spider arguments
-and copy them to the spider as attributes.
-The above example can also be written as follows::
-
-    import scrapy
-
-    class MySpider(scrapy.Spider):
-        name = 'myspider'
-
-        def start_requests(self):
-            yield scrapy.Request(f'http://www.example.com/categories/{self.category}')
-
-Keep in mind that spider arguments are only strings.
-The spider will not do any parsing on its own.
-If you were to set the ``start_urls`` attribute from the command line,
-you would have to parse it on your own into a list
-using something like :func:`ast.literal_eval` or :func:`json.loads`
-and then set it as an attribute.
-Otherwise, you would cause iteration over a ``start_urls`` string
-(a very common python pitfall)
-resulting in each character being seen as a separate url.
-
-A valid use case is to set the http auth credentials
-used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
-or the user agent
-used by :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`::
-
-    scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword -a user_agent=mybot
-
-Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
-See `Scrapyd documentation`_.
-
-.. _builtin-spiders:
-
-Generic Spiders
-===============
-
-Scrapy comes with some useful generic spiders that you can use to subclass
-your spiders from. Their aim is to provide convenient functionality for a few
-common scraping cases, like following all links on a site based on certain
-rules, crawling from `Sitemaps`_, or parsing an XML/CSV feed.
-
-For the examples used in the following spiders, we'll assume you have a project
-with a ``TestItem`` declared in a ``myproject.items`` module::
-
-    import scrapy
-
-    class TestItem(scrapy.Item):
-        id = scrapy.Field()
-        name = scrapy.Field()
-        description = scrapy.Field()
-
-
-.. currentmodule:: scrapy.spiders
-
-CrawlSpider
------------
-
-.. class:: CrawlSpider
-
-   This is the most commonly used spider for crawling regular websites, as it
-   provides a convenient mechanism for following links by defining a set of rules.
-   It may not be the best suited for your particular web sites or project, but
-   it's generic enough for several cases, so you can start from it and override it
-   as needed for more custom functionality, or just implement your own spider.
-
-   Apart from the attributes inherited from Spider (that you must
-   specify), this class supports a new attribute:
-
-   .. attribute:: rules
-
-       Which is a list of one (or more) :class:`Rule` objects.  Each :class:`Rule`
-       defines a certain behaviour for crawling the site. Rules objects are
-       described below. If multiple rules match the same link, the first one
-       will be used, according to the order they're defined in this attribute.
-
-   This spider also exposes an overrideable method:
-
-   .. method:: parse_start_url(response, **kwargs)
-
-      This method is called for each response produced for the URLs in
-      the spider's ``start_urls`` attribute. It allows to parse
-      the initial responses and must return either an
-      :ref:`item object <topics-items>`, a :class:`~scrapy.http.Request`
-      object, or an iterable containing any of them.
-
-Crawling rules
-~~~~~~~~~~~~~~
-
-.. autoclass:: Rule
-
-   ``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
-   defines how links will be extracted from each crawled page. Each produced link will
-   be used to generate a :class:`~scrapy.http.Request` object, which will contain the
-   link's text in its ``meta`` dictionary (under the ``link_text`` key).
-   If omitted, a default link extractor created with no arguments will be used,
-   resulting in all links being extracted.
-
-   ``callback`` is a callable or a string (in which case a method from the spider
-   object with that name will be used) to be called for each link extracted with
-   the specified link extractor. This callback receives a :class:`~scrapy.http.Response`
-   as its first argument and must return either a single instance or an iterable of
-   :ref:`item objects <topics-items>` and/or :class:`~scrapy.http.Request` objects
-   (or any subclass of them). As mentioned above, the received :class:`~scrapy.http.Response`
-   object will contain the text of the link that produced the :class:`~scrapy.http.Request`
-   in its ``meta`` dictionary (under the ``link_text`` key)
-
-   ``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
-   callback function.
-
-   ``follow`` is a boolean which specifies if links should be followed from each
-   response extracted with this rule. If ``callback`` is None ``follow`` defaults
-   to ``True``, otherwise it defaults to ``False``.
-
-   ``process_links`` is a callable, or a string (in which case a method from the
-   spider object with that name will be used) which will be called for each list
-   of links extracted from each response using the specified ``link_extractor``.
-   This is mainly used for filtering purposes.
-
-   ``process_request`` is a callable (or a string, in which case a method from
-   the spider object with that name will be used) which will be called for every
-   :class:`~scrapy.http.Request` extracted by this rule. This callable should
-   take said request as first argument and the :class:`~scrapy.http.Response`
-   from which the request originated as second argument. It must return a
-   ``Request`` object or ``None`` (to filter out the request).
-
-   ``errback`` is a callable or a string (in which case a method from the spider
-   object with that name will be used) to be called if any exception is
-   raised while processing a request generated by the rule.
-   It receives a :class:`Twisted Failure <twisted.python.failure.Failure>`
-   instance as first parameter.
-
-
-.. warning:: Because of its internal implementation, you must explicitly set
-   callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
-   unexpected behaviour can occur otherwise.
-
-   .. versionadded:: 2.0
-      The *errback* parameter.
-
-CrawlSpider example
-~~~~~~~~~~~~~~~~~~~
-
-Let's now take a look at an example CrawlSpider with rules::
-
-    import scrapy
-    from scrapy.spiders import CrawlSpider, Rule
-    from scrapy.linkextractors import LinkExtractor
-
-    class MySpider(CrawlSpider):
-        name = 'example.com'
-        allowed_domains = ['example.com']
-        start_urls = ['http://www.example.com']
-
-        rules = (
-            # Extract links matching 'category.php' (but not matching 'subsection.php')
-            # and follow links from them (since no callback means follow=True by default).
-            Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
-
-            # Extract links matching 'item.php' and parse them with the spider's method parse_item
-            Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
-        )
-
-        def parse_item(self, response):
-            self.logger.info('Hi, this is an item page! %s', response.url)
-            item = scrapy.Item()
-            item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
-            item['name'] = response.xpath('//td[@id="item_name"]/text()').get()
-            item['description'] = response.xpath('//td[@id="item_description"]/text()').get()
-            item['link_text'] = response.meta['link_text']
-            url = response.xpath('//td[@id="additional_data"]/@href').get()
-            return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item))
-
-        def parse_additional_page(self, response, item):
-            item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get()
-            return item
-
-
-This spider would start crawling example.com's home page, collecting category
-links, and item links, parsing the latter with the ``parse_item`` method. For
-each item response, some data will be extracted from the HTML using XPath, and
-an :class:`~scrapy.item.Item` will be filled with it.
-
-XMLFeedSpider
--------------
-
-.. class:: XMLFeedSpider
-
-    XMLFeedSpider is designed for parsing XML feeds by iterating through them by a
-    certain node name.  The iterator can be chosen from: ``iternodes``, ``xml``,
-    and ``html``.  It's recommended to use the ``iternodes`` iterator for
-    performance reasons, since the ``xml`` and ``html`` iterators generate the
-    whole DOM at once in order to parse it.  However, using ``html`` as the
-    iterator may be useful when parsing XML with bad markup.
-
-    To set the iterator and the tag name, you must define the following class
-    attributes:
-
-    .. attribute:: iterator
-
-        A string which defines the iterator to use. It can be either:
-
-           - ``'iternodes'`` - a fast iterator based on regular expressions
-
-           - ``'html'`` - an iterator which uses :class:`~scrapy.selector.Selector`.
-             Keep in mind this uses DOM parsing and must load all DOM in memory
-             which could be a problem for big feeds
-
-           - ``'xml'`` - an iterator which uses :class:`~scrapy.selector.Selector`.
-             Keep in mind this uses DOM parsing and must load all DOM in memory
-             which could be a problem for big feeds
-
-        It defaults to: ``'iternodes'``.
-
-    .. attribute:: itertag
-
-        A string with the name of the node (or element) to iterate in. Example::
-
-            itertag = 'product'
-
-    .. attribute:: namespaces
-
-        A list of ``(prefix, uri)`` tuples which define the namespaces
-        available in that document that will be processed with this spider. The
-        ``prefix`` and ``uri`` will be used to automatically register
-        namespaces using the
-        :meth:`~scrapy.selector.Selector.register_namespace` method.
-
-        You can then specify nodes with namespaces in the :attr:`itertag`
-        attribute.
-
-        Example::
-
-            class YourSpider(XMLFeedSpider):
-
-                namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]
-                itertag = 'n:url'
-                # ...
-
-    Apart from these new attributes, this spider has the following overrideable
-    methods too:
-
-    .. method:: adapt_response(response)
-
-        A method that receives the response as soon as it arrives from the spider
-        middleware, before the spider starts parsing it. It can be used to modify
-        the response body before parsing it. This method receives a response and
-        also returns a response (it could be the same or another one).
-
-    .. method:: parse_node(response, selector)
-
-        This method is called for the nodes matching the provided tag name
-        (``itertag``).  Receives the response and an
-        :class:`~scrapy.selector.Selector` for each node.  Overriding this
-        method is mandatory. Otherwise, you spider won't work.  This method
-        must return an :ref:`item object <topics-items>`, a
-        :class:`~scrapy.http.Request` object, or an iterable containing any of
-        them.
-
-    .. method:: process_results(response, results)
-
-        This method is called for each result (item or request) returned by the
-        spider, and it's intended to perform any last time processing required
-        before returning the results to the framework core, for example setting the
-        item IDs. It receives a list of results and the response which originated
-        those results. It must return a list of results (items or requests).
-
-
-.. warning:: Because of its internal implementation, you must explicitly set
-   callbacks for new requests when writing :class:`XMLFeedSpider`-based spiders;
-   unexpected behaviour can occur otherwise.
-
-
-XMLFeedSpider example
-~~~~~~~~~~~~~~~~~~~~~
-
-These spiders are pretty easy to use, let's have a look at one example::
-
-    from scrapy.spiders import XMLFeedSpider
-    from myproject.items import TestItem
-
-    class MySpider(XMLFeedSpider):
-        name = 'example.com'
-        allowed_domains = ['example.com']
-        start_urls = ['http://www.example.com/feed.xml']
-        iterator = 'iternodes'  # This is actually unnecessary, since it's the default value
-        itertag = 'item'
-
-        def parse_node(self, response, node):
-            self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.getall()))
-
-            item = TestItem()
-            item['id'] = node.xpath('@id').get()
-            item['name'] = node.xpath('name').get()
-            item['description'] = node.xpath('description').get()
-            return item
-
-Basically what we did up there was to create a spider that downloads a feed from
-the given ``start_urls``, and then iterates through each of its ``item`` tags,
-prints them out, and stores some random data in an :class:`~scrapy.item.Item`.
-
-CSVFeedSpider
--------------
-
-.. class:: CSVFeedSpider
-
-   This spider is very similar to the XMLFeedSpider, except that it iterates
-   over rows, instead of nodes. The method that gets called in each iteration
-   is :meth:`parse_row`.
-
-   .. attribute:: delimiter
-
-       A string with the separator character for each field in the CSV file
-       Defaults to ``','`` (comma).
-
-   .. attribute:: quotechar
-
-       A string with the enclosure character for each field in the CSV file
-       Defaults to ``'"'`` (quotation mark).
-
-   .. attribute:: headers
-
-       A list of the column names in the CSV file.
-
-   .. method:: parse_row(response, row)
-
-       Receives a response and a dict (representing each row) with a key for each
-       provided (or detected) header of the CSV file.  This spider also gives the
-       opportunity to override ``adapt_response`` and ``process_results`` methods
-       for pre- and post-processing purposes.
-
-CSVFeedSpider example
-~~~~~~~~~~~~~~~~~~~~~
-
-Let's see an example similar to the previous one, but using a
-:class:`CSVFeedSpider`::
-
-    from scrapy.spiders import CSVFeedSpider
-    from myproject.items import TestItem
-
-    class MySpider(CSVFeedSpider):
-        name = 'example.com'
-        allowed_domains = ['example.com']
-        start_urls = ['http://www.example.com/feed.csv']
-        delimiter = ';'
-        quotechar = "'"
-        headers = ['id', 'name', 'description']
-
-        def parse_row(self, response, row):
-            self.logger.info('Hi, this is a row!: %r', row)
-
-            item = TestItem()
-            item['id'] = row['id']
-            item['name'] = row['name']
-            item['description'] = row['description']
-            return item
-
-
-SitemapSpider
--------------
-
-.. class:: SitemapSpider
-
-    SitemapSpider allows you to crawl a site by discovering the URLs using
-    `Sitemaps`_.
-
-    It supports nested sitemaps and discovering sitemap urls from
-    `robots.txt`_.
-
-    .. attribute:: sitemap_urls
-
-        A list of urls pointing to the sitemaps whose urls you want to crawl.
-
-        You can also point to a `robots.txt`_ and it will be parsed to extract
-        sitemap urls from it.
-
-    .. attribute:: sitemap_rules
-
-        A list of tuples ``(regex, callback)`` where:
-
-        * ``regex`` is a regular expression to match urls extracted from sitemaps.
-          ``regex`` can be either a str or a compiled regex object.
-
-        * callback is the callback to use for processing the urls that match
-          the regular expression. ``callback`` can be a string (indicating the
-          name of a spider method) or a callable.
-
-        For example::
-
-            sitemap_rules = [('/product/', 'parse_product')]
-
-        Rules are applied in order, and only the first one that matches will be
-        used.
-
-        If you omit this attribute, all urls found in sitemaps will be
-        processed with the ``parse`` callback.
-
-    .. attribute:: sitemap_follow
-
-        A list of regexes of sitemap that should be followed. This is only
-        for sites that use `Sitemap index files`_ that point to other sitemap
-        files.
-
-        By default, all sitemaps are followed.
-
-    .. attribute:: sitemap_alternate_links
-
-        Specifies if alternate links for one ``url`` should be followed. These
-        are links for the same website in another language passed within
-        the same ``url`` block.
-
-        For example::
-
-            <url>
-                <loc>http://example.com/</loc>
-                <xhtml:link rel="alternate" hreflang="de" href="http://example.com/de"/>
-            </url>
-
-        With ``sitemap_alternate_links`` set, this would retrieve both URLs. With
-        ``sitemap_alternate_links`` disabled, only ``http://example.com/`` would be
-        retrieved.
-
-        Default is ``sitemap_alternate_links`` disabled.
-
-    .. method:: sitemap_filter(entries)
-
-        This is a filter function that could be overridden to select sitemap entries
-        based on their attributes.
-
-        For example::
-
-            <url>
-                <loc>http://example.com/</loc>
-                <lastmod>2005-01-01</lastmod>
-            </url>
-
-        We can define a ``sitemap_filter`` function to filter ``entries`` by date::
-
-            from datetime import datetime
-            from scrapy.spiders import SitemapSpider
-
-            class FilteredSitemapSpider(SitemapSpider):
-                name = 'filtered_sitemap_spider'
-                allowed_domains = ['example.com']
-                sitemap_urls = ['http://example.com/sitemap.xml']
-
-                def sitemap_filter(self, entries):
-                    for entry in entries:
-                        date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d')
-                        if date_time.year >= 2005:
-                            yield entry
-
-        This would retrieve only ``entries`` modified on 2005 and the following
-        years.
-
-        Entries are dict objects extracted from the sitemap document.
-        Usually, the key is the tag name and the value is the text inside it.
-
-        It's important to notice that:
-
-        - as the loc attribute is required, entries without this tag are discarded
-        - alternate links are stored in a list with the key ``alternate``
-          (see ``sitemap_alternate_links``)
-        - namespaces are removed, so lxml tags named as ``{namespace}tagname`` become only ``tagname``
-
-        If you omit this method, all entries found in sitemaps will be
-        processed, observing other attributes and their settings.
-
-
-SitemapSpider examples
-~~~~~~~~~~~~~~~~~~~~~~
-
-Simplest example: process all urls discovered through sitemaps using the
-``parse`` callback::
-
-    from scrapy.spiders import SitemapSpider
-
-    class MySpider(SitemapSpider):
-        sitemap_urls = ['http://www.example.com/sitemap.xml']
-
-        def parse(self, response):
-            pass # ... scrape item here ...
-
-Process some urls with certain callback and other urls with a different
-callback::
-
-    from scrapy.spiders import SitemapSpider
-
-    class MySpider(SitemapSpider):
-        sitemap_urls = ['http://www.example.com/sitemap.xml']
-        sitemap_rules = [
-            ('/product/', 'parse_product'),
-            ('/category/', 'parse_category'),
-        ]
-
-        def parse_product(self, response):
-            pass # ... scrape product ...
-
-        def parse_category(self, response):
-            pass # ... scrape category ...
-
-Follow sitemaps defined in the `robots.txt`_ file and only follow sitemaps
-whose url contains ``/sitemap_shop``::
-
-    from scrapy.spiders import SitemapSpider
-
-    class MySpider(SitemapSpider):
-        sitemap_urls = ['http://www.example.com/robots.txt']
-        sitemap_rules = [
-            ('/shop/', 'parse_shop'),
-        ]
-        sitemap_follow = ['/sitemap_shops']
-
-        def parse_shop(self, response):
-            pass # ... scrape shop here ...
-
-Combine SitemapSpider with other sources of urls::
-
-    from scrapy.spiders import SitemapSpider
-
-    class MySpider(SitemapSpider):
-        sitemap_urls = ['http://www.example.com/robots.txt']
-        sitemap_rules = [
-            ('/shop/', 'parse_shop'),
-        ]
-
-        other_urls = ['http://www.example.com/about']
-
-        def start_requests(self):
-            requests = list(super(MySpider, self).start_requests())
-            requests += [scrapy.Request(x, self.parse_other) for x in self.other_urls]
-            return requests
-
-        def parse_shop(self, response):
-            pass # ... scrape shop here ...
-
-        def parse_other(self, response):
-            pass # ... scrape other here ...
-
-.. _Sitemaps: https://www.sitemaps.org/index.html
-.. _Sitemap index files: https://www.sitemaps.org/protocol.html#index
-.. _robots.txt: https://www.robotstxt.org/
-.. _TLD: https://en.wikipedia.org/wiki/Top-level_domain
-.. _Scrapyd documentation: https://scrapyd.readthedocs.io/en/latest/
+.. _topics-spiders:
+
+=======
+Spiders
+=======
+
+Spiders are classes which define how a certain site (or a group of sites) will be
+scraped, including how to perform the crawl (i.e. follow links) and how to
+extract structured data from their pages (i.e. scraping items). In other words,
+Spiders are the place where you define the custom behaviour for crawling and
+parsing pages for a particular site (or, in some cases, a group of sites).
+
+For spiders, the scraping cycle goes through something like this:
+
+1. You start by generating the initial Requests to crawl the first URLs, and
+   specify a callback function to be called with the response downloaded from
+   those requests.
+
+   The first requests to perform are obtained by calling the
+   :meth:`~scrapy.Spider.start_requests` method which (by default)
+   generates :class:`~scrapy.Request` for the URLs specified in the
+   :attr:`~scrapy.Spider.start_urls` and the
+   :attr:`~scrapy.Spider.parse` method as callback function for the
+   Requests.
+
+2. In the callback function, you parse the response (web page) and return
+   :ref:`item objects <topics-items>`,
+   :class:`~scrapy.Request` objects, or an iterable of these objects.
+   Those Requests will also contain a callback (maybe
+   the same) and will then be downloaded by Scrapy and then their
+   response handled by the specified callback.
+
+3. In callback functions, you parse the page contents, typically using
+   :ref:`topics-selectors` (but you can also use BeautifulSoup, lxml or whatever
+   mechanism you prefer) and generate items with the parsed data.
+
+4. Finally, the items returned from the spider will be typically persisted to a
+   database (in some :ref:`Item Pipeline <topics-item-pipeline>`) or written to
+   a file using :ref:`topics-feed-exports`.
+
+Even though this cycle applies (more or less) to any kind of spider, there are
+different kinds of default spiders bundled into Scrapy for different purposes.
+We will talk about those types here.
+
+.. module:: scrapy.spiders
+   :synopsis: Spiders base class, spider manager and spider middleware
+
+.. _topics-spiders-ref:
+
+scrapy.Spider
+=============
+
+.. class:: scrapy.spiders.Spider()
+.. class:: scrapy.Spider()
+
+   This is the simplest spider, and the one from which every other spider
+   must inherit (including spiders that come bundled with Scrapy, as well as spiders
+   that you write yourself). It doesn't provide any special functionality. It just
+   provides a default :meth:`start_requests` implementation which sends requests from
+   the :attr:`start_urls` spider attribute and calls the spider's method ``parse``
+   for each of the resulting responses.
+
+   .. attribute:: name
+
+       A string which defines the name for this spider. The spider name is how
+       the spider is located (and instantiated) by Scrapy, so it must be
+       unique. However, nothing prevents you from instantiating more than one
+       instance of the same spider. This is the most important spider attribute
+       and it's required.
+
+       If the spider scrapes a single domain, a common practice is to name the
+       spider after the domain, with or without the `TLD`_. So, for example, a
+       spider that crawls ``mywebsite.com`` would often be called
+       ``mywebsite``.
+
+   .. attribute:: allowed_domains
+
+       An optional list of strings containing domains that this spider is
+       allowed to crawl. Requests for URLs not belonging to the domain names
+       specified in this list (or their subdomains) won't be followed if
+       :class:`~scrapy.spidermiddlewares.offsite.OffsiteMiddleware` is enabled.
+
+       Let's say your target url is ``https://www.example.com/1.html``,
+       then add ``'example.com'`` to the list.
+
+   .. attribute:: start_urls
+
+       A list of URLs where the spider will begin to crawl from, when no
+       particular URLs are specified. So, the first pages downloaded will be those
+       listed here. The subsequent :class:`~scrapy.Request` will be generated successively from data
+       contained in the start URLs.
+
+   .. attribute:: custom_settings
+
+      A dictionary of settings that will be overridden from the project wide
+      configuration when running this spider. It must be defined as a class
+      attribute since the settings are updated before instantiation.
+
+      For a list of available built-in settings see:
+      :ref:`topics-settings-ref`.
+
+   .. attribute:: crawler
+
+      This attribute is set by the :meth:`from_crawler` class method after
+      initializating the class, and links to the
+      :class:`~scrapy.crawler.Crawler` object to which this spider instance is
+      bound.
+
+      Crawlers encapsulate a lot of components in the project for their single
+      entry access (such as extensions, middlewares, signals managers, etc).
+      See :ref:`topics-api-crawler` to know more about them.
+
+   .. attribute:: settings
+
+      Configuration for running this spider. This is a
+      :class:`~scrapy.settings.Settings` instance, see the
+      :ref:`topics-settings` topic for a detailed introduction on this subject.
+
+   .. attribute:: logger
+
+      Python logger created with the Spider's :attr:`name`. You can use it to
+      send log messages through it as described on
+      :ref:`topics-logging-from-spiders`.
+
+   .. method:: from_crawler(crawler, *args, **kwargs)
+
+       This is the class method used by Scrapy to create your spiders.
+
+       You probably won't need to override this directly because the default
+       implementation acts as a proxy to the :meth:`__init__` method, calling
+       it with the given arguments ``args`` and named arguments ``kwargs``.
+
+       Nonetheless, this method sets the :attr:`crawler` and :attr:`settings`
+       attributes in the new instance so they can be accessed later inside the
+       spider's code.
+
+       :param crawler: crawler to which the spider will be bound
+       :type crawler: :class:`~scrapy.crawler.Crawler` instance
+
+       :param args: arguments passed to the :meth:`__init__` method
+       :type args: list
+
+       :param kwargs: keyword arguments passed to the :meth:`__init__` method
+       :type kwargs: dict
+
+   .. method:: start_requests()
+
+       This method must return an iterable with the first Requests to crawl for
+       this spider. It is called by Scrapy when the spider is opened for
+       scraping. Scrapy calls it only once, so it is safe to implement
+       :meth:`start_requests` as a generator.
+
+       The default implementation generates ``Request(url, dont_filter=True)``
+       for each url in :attr:`start_urls`.
+
+       If you want to change the Requests used to start scraping a domain, this is
+       the method to override. For example, if you need to start by logging in using
+       a POST request, you could do::
+
+           class MySpider(scrapy.Spider):
+               name = 'myspider'
+
+               def start_requests(self):
+                   return [scrapy.FormRequest("http://www.example.com/login",
+                                              formdata={'user': 'john', 'pass': 'secret'},
+                                              callback=self.logged_in)]
+
+               def logged_in(self, response):
+                   # here you would extract links to follow and return Requests for
+                   # each of them, with another callback
+                   pass
+
+   .. method:: parse(response)
+
+       This is the default callback used by Scrapy to process downloaded
+       responses, when their requests don't specify a callback.
+
+       The ``parse`` method is in charge of processing the response and returning
+       scraped data and/or more URLs to follow. Other Requests callbacks have
+       the same requirements as the :class:`Spider` class.
+
+       This method, as well as any other Request callback, must return an
+       iterable of :class:`~scrapy.Request` and/or :ref:`item objects
+       <topics-items>`.
+
+       :param response: the response to parse
+       :type response: :class:`~scrapy.http.Response`
+
+   .. method:: log(message, [level, component])
+
+       Wrapper that sends a log message through the Spider's :attr:`logger`,
+       kept for backward compatibility. For more information see
+       :ref:`topics-logging-from-spiders`.
+
+   .. method:: closed(reason)
+
+       Called when the spider closes. This method provides a shortcut to
+       signals.connect() for the :signal:`spider_closed` signal.
+
+Let's see an example::
+
+    import scrapy
+
+
+    class MySpider(scrapy.Spider):
+        name = 'example.com'
+        allowed_domains = ['example.com']
+        start_urls = [
+            'http://www.example.com/1.html',
+            'http://www.example.com/2.html',
+            'http://www.example.com/3.html',
+        ]
+
+        def parse(self, response):
+            self.logger.info('A response from %s just arrived!', response.url)
+
+Return multiple Requests and items from a single callback::
+
+    import scrapy
+
+    class MySpider(scrapy.Spider):
+        name = 'example.com'
+        allowed_domains = ['example.com']
+        start_urls = [
+            'http://www.example.com/1.html',
+            'http://www.example.com/2.html',
+            'http://www.example.com/3.html',
+        ]
+
+        def parse(self, response):
+            for h3 in response.xpath('//h3').getall():
+                yield {"title": h3}
+
+            for href in response.xpath('//a/@href').getall():
+                yield scrapy.Request(response.urljoin(href), self.parse)
+
+Instead of :attr:`~.start_urls` you can use :meth:`~.start_requests` directly;
+to give data more structure you can use :class:`~scrapy.Item` objects::
+
+    import scrapy
+    from myproject.items import MyItem
+
+    class MySpider(scrapy.Spider):
+        name = 'example.com'
+        allowed_domains = ['example.com']
+
+        def start_requests(self):
+            yield scrapy.Request('http://www.example.com/1.html', self.parse)
+            yield scrapy.Request('http://www.example.com/2.html', self.parse)
+            yield scrapy.Request('http://www.example.com/3.html', self.parse)
+
+        def parse(self, response):
+            for h3 in response.xpath('//h3').getall():
+                yield MyItem(title=h3)
+
+            for href in response.xpath('//a/@href').getall():
+                yield scrapy.Request(response.urljoin(href), self.parse)
+
+.. _spiderargs:
+
+Spider arguments
+================
+
+Spiders can receive arguments that modify their behaviour. Some common uses for
+spider arguments are to define the start URLs or to restrict the crawl to
+certain sections of the site, but they can be used to configure any
+functionality of the spider.
+
+Spider arguments are passed through the :command:`crawl` command using the
+``-a`` option. For example::
+
+    scrapy crawl myspider -a category=electronics
+
+Spiders can access arguments in their `__init__` methods::
+
+    import scrapy
+
+    class MySpider(scrapy.Spider):
+        name = 'myspider'
+
+        def __init__(self, category=None, *args, **kwargs):
+            super(MySpider, self).__init__(*args, **kwargs)
+            self.start_urls = [f'http://www.example.com/categories/{category}']
+            # ...
+
+The default `__init__` method will take any spider arguments
+and copy them to the spider as attributes.
+The above example can also be written as follows::
+
+    import scrapy
+
+    class MySpider(scrapy.Spider):
+        name = 'myspider'
+
+        def start_requests(self):
+            yield scrapy.Request(f'http://www.example.com/categories/{self.category}')
+
+Keep in mind that spider arguments are only strings.
+The spider will not do any parsing on its own.
+If you were to set the ``start_urls`` attribute from the command line,
+you would have to parse it on your own into a list
+using something like :func:`ast.literal_eval` or :func:`json.loads`
+and then set it as an attribute.
+Otherwise, you would cause iteration over a ``start_urls`` string
+(a very common python pitfall)
+resulting in each character being seen as a separate url.
+
+A valid use case is to set the http auth credentials
+used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
+or the user agent
+used by :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`::
+
+    scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword -a user_agent=mybot
+
+Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
+See `Scrapyd documentation`_.
+
+.. _builtin-spiders:
+
+Generic Spiders
+===============
+
+Scrapy comes with some useful generic spiders that you can use to subclass
+your spiders from. Their aim is to provide convenient functionality for a few
+common scraping cases, like following all links on a site based on certain
+rules, crawling from `Sitemaps`_, or parsing an XML/CSV feed.
+
+For the examples used in the following spiders, we'll assume you have a project
+with a ``TestItem`` declared in a ``myproject.items`` module::
+
+    import scrapy
+
+    class TestItem(scrapy.Item):
+        id = scrapy.Field()
+        name = scrapy.Field()
+        description = scrapy.Field()
+
+
+.. currentmodule:: scrapy.spiders
+
+CrawlSpider
+-----------
+
+.. class:: CrawlSpider
+
+   This is the most commonly used spider for crawling regular websites, as it
+   provides a convenient mechanism for following links by defining a set of rules.
+   It may not be the best suited for your particular web sites or project, but
+   it's generic enough for several cases, so you can start from it and override it
+   as needed for more custom functionality, or just implement your own spider.
+
+   Apart from the attributes inherited from Spider (that you must
+   specify), this class supports a new attribute:
+
+   .. attribute:: rules
+
+       Which is a list of one (or more) :class:`Rule` objects.  Each :class:`Rule`
+       defines a certain behaviour for crawling the site. Rules objects are
+       described below. If multiple rules match the same link, the first one
+       will be used, according to the order they're defined in this attribute.
+
+   This spider also exposes an overrideable method:
+
+   .. method:: parse_start_url(response, **kwargs)
+
+      This method is called for each response produced for the URLs in
+      the spider's ``start_urls`` attribute. It allows to parse
+      the initial responses and must return either an
+      :ref:`item object <topics-items>`, a :class:`~scrapy.Request`
+      object, or an iterable containing any of them.
+
+Crawling rules
+~~~~~~~~~~~~~~
+
+.. autoclass:: Rule
+
+   ``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
+   defines how links will be extracted from each crawled page. Each produced link will
+   be used to generate a :class:`~scrapy.Request` object, which will contain the
+   link's text in its ``meta`` dictionary (under the ``link_text`` key).
+   If omitted, a default link extractor created with no arguments will be used,
+   resulting in all links being extracted.
+
+   ``callback`` is a callable or a string (in which case a method from the spider
+   object with that name will be used) to be called for each link extracted with
+   the specified link extractor. This callback receives a :class:`~scrapy.http.Response`
+   as its first argument and must return either a single instance or an iterable of
+   :ref:`item objects <topics-items>` and/or :class:`~scrapy.Request` objects
+   (or any subclass of them). As mentioned above, the received :class:`~scrapy.http.Response`
+   object will contain the text of the link that produced the :class:`~scrapy.Request`
+   in its ``meta`` dictionary (under the ``link_text`` key)
+
+   ``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
+   callback function.
+
+   ``follow`` is a boolean which specifies if links should be followed from each
+   response extracted with this rule. If ``callback`` is None ``follow`` defaults
+   to ``True``, otherwise it defaults to ``False``.
+
+   ``process_links`` is a callable, or a string (in which case a method from the
+   spider object with that name will be used) which will be called for each list
+   of links extracted from each response using the specified ``link_extractor``.
+   This is mainly used for filtering purposes.
+
+   ``process_request`` is a callable (or a string, in which case a method from
+   the spider object with that name will be used) which will be called for every
+   :class:`~scrapy.Request` extracted by this rule. This callable should
+   take said request as first argument and the :class:`~scrapy.http.Response`
+   from which the request originated as second argument. It must return a
+   ``Request`` object or ``None`` (to filter out the request).
+
+   ``errback`` is a callable or a string (in which case a method from the spider
+   object with that name will be used) to be called if any exception is
+   raised while processing a request generated by the rule.
+   It receives a :class:`Twisted Failure <twisted.python.failure.Failure>`
+   instance as first parameter.
+
+
+.. warning:: Because of its internal implementation, you must explicitly set
+   callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
+   unexpected behaviour can occur otherwise.
+
+   .. versionadded:: 2.0
+      The *errback* parameter.
+
+CrawlSpider example
+~~~~~~~~~~~~~~~~~~~
+
+Let's now take a look at an example CrawlSpider with rules::
+
+    import scrapy
+    from scrapy.spiders import CrawlSpider, Rule
+    from scrapy.linkextractors import LinkExtractor
+
+    class MySpider(CrawlSpider):
+        name = 'example.com'
+        allowed_domains = ['example.com']
+        start_urls = ['http://www.example.com']
+
+        rules = (
+            # Extract links matching 'category.php' (but not matching 'subsection.php')
+            # and follow links from them (since no callback means follow=True by default).
+            Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
+
+            # Extract links matching 'item.php' and parse them with the spider's method parse_item
+            Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
+        )
+
+        def parse_item(self, response):
+            self.logger.info('Hi, this is an item page! %s', response.url)
+            item = scrapy.Item()
+            item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
+            item['name'] = response.xpath('//td[@id="item_name"]/text()').get()
+            item['description'] = response.xpath('//td[@id="item_description"]/text()').get()
+            item['link_text'] = response.meta['link_text']
+            url = response.xpath('//td[@id="additional_data"]/@href').get()
+            return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item))
+
+        def parse_additional_page(self, response, item):
+            item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get()
+            return item
+
+
+This spider would start crawling example.com's home page, collecting category
+links, and item links, parsing the latter with the ``parse_item`` method. For
+each item response, some data will be extracted from the HTML using XPath, and
+an :class:`~scrapy.Item` will be filled with it.
+
+XMLFeedSpider
+-------------
+
+.. class:: XMLFeedSpider
+
+    XMLFeedSpider is designed for parsing XML feeds by iterating through them by a
+    certain node name.  The iterator can be chosen from: ``iternodes``, ``xml``,
+    and ``html``.  It's recommended to use the ``iternodes`` iterator for
+    performance reasons, since the ``xml`` and ``html`` iterators generate the
+    whole DOM at once in order to parse it.  However, using ``html`` as the
+    iterator may be useful when parsing XML with bad markup.
+
+    To set the iterator and the tag name, you must define the following class
+    attributes:
+
+    .. attribute:: iterator
+
+        A string which defines the iterator to use. It can be either:
+
+           - ``'iternodes'`` - a fast iterator based on regular expressions
+
+           - ``'html'`` - an iterator which uses :class:`~scrapy.Selector`.
+             Keep in mind this uses DOM parsing and must load all DOM in memory
+             which could be a problem for big feeds
+
+           - ``'xml'`` - an iterator which uses :class:`~scrapy.Selector`.
+             Keep in mind this uses DOM parsing and must load all DOM in memory
+             which could be a problem for big feeds
+
+        It defaults to: ``'iternodes'``.
+
+    .. attribute:: itertag
+
+        A string with the name of the node (or element) to iterate in. Example::
+
+            itertag = 'product'
+
+    .. attribute:: namespaces
+
+        A list of ``(prefix, uri)`` tuples which define the namespaces
+        available in that document that will be processed with this spider. The
+        ``prefix`` and ``uri`` will be used to automatically register
+        namespaces using the
+        :meth:`~scrapy.Selector.register_namespace` method.
+
+        You can then specify nodes with namespaces in the :attr:`itertag`
+        attribute.
+
+        Example::
+
+            class YourSpider(XMLFeedSpider):
+
+                namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]
+                itertag = 'n:url'
+                # ...
+
+    Apart from these new attributes, this spider has the following overrideable
+    methods too:
+
+    .. method:: adapt_response(response)
+
+        A method that receives the response as soon as it arrives from the spider
+        middleware, before the spider starts parsing it. It can be used to modify
+        the response body before parsing it. This method receives a response and
+        also returns a response (it could be the same or another one).
+
+    .. method:: parse_node(response, selector)
+
+        This method is called for the nodes matching the provided tag name
+        (``itertag``).  Receives the response and an
+        :class:`~scrapy.Selector` for each node.  Overriding this
+        method is mandatory. Otherwise, you spider won't work.  This method
+        must return an :ref:`item object <topics-items>`, a
+        :class:`~scrapy.Request` object, or an iterable containing any of
+        them.
+
+    .. method:: process_results(response, results)
+
+        This method is called for each result (item or request) returned by the
+        spider, and it's intended to perform any last time processing required
+        before returning the results to the framework core, for example setting the
+        item IDs. It receives a list of results and the response which originated
+        those results. It must return a list of results (items or requests).
+
+
+.. warning:: Because of its internal implementation, you must explicitly set
+   callbacks for new requests when writing :class:`XMLFeedSpider`-based spiders;
+   unexpected behaviour can occur otherwise.
+
+
+XMLFeedSpider example
+~~~~~~~~~~~~~~~~~~~~~
+
+These spiders are pretty easy to use, let's have a look at one example::
+
+    from scrapy.spiders import XMLFeedSpider
+    from myproject.items import TestItem
+
+    class MySpider(XMLFeedSpider):
+        name = 'example.com'
+        allowed_domains = ['example.com']
+        start_urls = ['http://www.example.com/feed.xml']
+        iterator = 'iternodes'  # This is actually unnecessary, since it's the default value
+        itertag = 'item'
+
+        def parse_node(self, response, node):
+            self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.getall()))
+
+            item = TestItem()
+            item['id'] = node.xpath('@id').get()
+            item['name'] = node.xpath('name').get()
+            item['description'] = node.xpath('description').get()
+            return item
+
+Basically what we did up there was to create a spider that downloads a feed from
+the given ``start_urls``, and then iterates through each of its ``item`` tags,
+prints them out, and stores some random data in an :class:`~scrapy.Item`.
+
+CSVFeedSpider
+-------------
+
+.. class:: CSVFeedSpider
+
+   This spider is very similar to the XMLFeedSpider, except that it iterates
+   over rows, instead of nodes. The method that gets called in each iteration
+   is :meth:`parse_row`.
+
+   .. attribute:: delimiter
+
+       A string with the separator character for each field in the CSV file
+       Defaults to ``','`` (comma).
+
+   .. attribute:: quotechar
+
+       A string with the enclosure character for each field in the CSV file
+       Defaults to ``'"'`` (quotation mark).
+
+   .. attribute:: headers
+
+       A list of the column names in the CSV file.
+
+   .. method:: parse_row(response, row)
+
+       Receives a response and a dict (representing each row) with a key for each
+       provided (or detected) header of the CSV file.  This spider also gives the
+       opportunity to override ``adapt_response`` and ``process_results`` methods
+       for pre- and post-processing purposes.
+
+CSVFeedSpider example
+~~~~~~~~~~~~~~~~~~~~~
+
+Let's see an example similar to the previous one, but using a
+:class:`CSVFeedSpider`::
+
+    from scrapy.spiders import CSVFeedSpider
+    from myproject.items import TestItem
+
+    class MySpider(CSVFeedSpider):
+        name = 'example.com'
+        allowed_domains = ['example.com']
+        start_urls = ['http://www.example.com/feed.csv']
+        delimiter = ';'
+        quotechar = "'"
+        headers = ['id', 'name', 'description']
+
+        def parse_row(self, response, row):
+            self.logger.info('Hi, this is a row!: %r', row)
+
+            item = TestItem()
+            item['id'] = row['id']
+            item['name'] = row['name']
+            item['description'] = row['description']
+            return item
+
+
+SitemapSpider
+-------------
+
+.. class:: SitemapSpider
+
+    SitemapSpider allows you to crawl a site by discovering the URLs using
+    `Sitemaps`_.
+
+    It supports nested sitemaps and discovering sitemap urls from
+    `robots.txt`_.
+
+    .. attribute:: sitemap_urls
+
+        A list of urls pointing to the sitemaps whose urls you want to crawl.
+
+        You can also point to a `robots.txt`_ and it will be parsed to extract
+        sitemap urls from it.
+
+    .. attribute:: sitemap_rules
+
+        A list of tuples ``(regex, callback)`` where:
+
+        * ``regex`` is a regular expression to match urls extracted from sitemaps.
+          ``regex`` can be either a str or a compiled regex object.
+
+        * callback is the callback to use for processing the urls that match
+          the regular expression. ``callback`` can be a string (indicating the
+          name of a spider method) or a callable.
+
+        For example::
+
+            sitemap_rules = [('/product/', 'parse_product')]
+
+        Rules are applied in order, and only the first one that matches will be
+        used.
+
+        If you omit this attribute, all urls found in sitemaps will be
+        processed with the ``parse`` callback.
+
+    .. attribute:: sitemap_follow
+
+        A list of regexes of sitemap that should be followed. This is only
+        for sites that use `Sitemap index files`_ that point to other sitemap
+        files.
+
+        By default, all sitemaps are followed.
+
+    .. attribute:: sitemap_alternate_links
+
+        Specifies if alternate links for one ``url`` should be followed. These
+        are links for the same website in another language passed within
+        the same ``url`` block.
+
+        For example::
+
+            <url>
+                <loc>http://example.com/</loc>
+                <xhtml:link rel="alternate" hreflang="de" href="http://example.com/de"/>
+            </url>
+
+        With ``sitemap_alternate_links`` set, this would retrieve both URLs. With
+        ``sitemap_alternate_links`` disabled, only ``http://example.com/`` would be
+        retrieved.
+
+        Default is ``sitemap_alternate_links`` disabled.
+
+    .. method:: sitemap_filter(entries)
+
+        This is a filter function that could be overridden to select sitemap entries
+        based on their attributes.
+
+        For example::
+
+            <url>
+                <loc>http://example.com/</loc>
+                <lastmod>2005-01-01</lastmod>
+            </url>
+
+        We can define a ``sitemap_filter`` function to filter ``entries`` by date::
+
+            from datetime import datetime
+            from scrapy.spiders import SitemapSpider
+
+            class FilteredSitemapSpider(SitemapSpider):
+                name = 'filtered_sitemap_spider'
+                allowed_domains = ['example.com']
+                sitemap_urls = ['http://example.com/sitemap.xml']
+
+                def sitemap_filter(self, entries):
+                    for entry in entries:
+                        date_time = datetime.strptime(entry['lastmod'], '%Y-%m-%d')
+                        if date_time.year >= 2005:
+                            yield entry
+
+        This would retrieve only ``entries`` modified on 2005 and the following
+        years.
+
+        Entries are dict objects extracted from the sitemap document.
+        Usually, the key is the tag name and the value is the text inside it.
+
+        It's important to notice that:
+
+        - as the loc attribute is required, entries without this tag are discarded
+        - alternate links are stored in a list with the key ``alternate``
+          (see ``sitemap_alternate_links``)
+        - namespaces are removed, so lxml tags named as ``{namespace}tagname`` become only ``tagname``
+
+        If you omit this method, all entries found in sitemaps will be
+        processed, observing other attributes and their settings.
+
+
+SitemapSpider examples
+~~~~~~~~~~~~~~~~~~~~~~
+
+Simplest example: process all urls discovered through sitemaps using the
+``parse`` callback::
+
+    from scrapy.spiders import SitemapSpider
+
+    class MySpider(SitemapSpider):
+        sitemap_urls = ['http://www.example.com/sitemap.xml']
+
+        def parse(self, response):
+            pass # ... scrape item here ...
+
+Process some urls with certain callback and other urls with a different
+callback::
+
+    from scrapy.spiders import SitemapSpider
+
+    class MySpider(SitemapSpider):
+        sitemap_urls = ['http://www.example.com/sitemap.xml']
+        sitemap_rules = [
+            ('/product/', 'parse_product'),
+            ('/category/', 'parse_category'),
+        ]
+
+        def parse_product(self, response):
+            pass # ... scrape product ...
+
+        def parse_category(self, response):
+            pass # ... scrape category ...
+
+Follow sitemaps defined in the `robots.txt`_ file and only follow sitemaps
+whose url contains ``/sitemap_shop``::
+
+    from scrapy.spiders import SitemapSpider
+
+    class MySpider(SitemapSpider):
+        sitemap_urls = ['http://www.example.com/robots.txt']
+        sitemap_rules = [
+            ('/shop/', 'parse_shop'),
+        ]
+        sitemap_follow = ['/sitemap_shops']
+
+        def parse_shop(self, response):
+            pass # ... scrape shop here ...
+
+Combine SitemapSpider with other sources of urls::
+
+    from scrapy.spiders import SitemapSpider
+
+    class MySpider(SitemapSpider):
+        sitemap_urls = ['http://www.example.com/robots.txt']
+        sitemap_rules = [
+            ('/shop/', 'parse_shop'),
+        ]
+
+        other_urls = ['http://www.example.com/about']
+
+        def start_requests(self):
+            requests = list(super(MySpider, self).start_requests())
+            requests += [scrapy.Request(x, self.parse_other) for x in self.other_urls]
+            return requests
+
+        def parse_shop(self, response):
+            pass # ... scrape shop here ...
+
+        def parse_other(self, response):
+            pass # ... scrape other here ...
+
+.. _Sitemaps: https://www.sitemaps.org/index.html
+.. _Sitemap index files: https://www.sitemaps.org/protocol.html#index
+.. _robots.txt: https://www.robotstxt.org/
+.. _TLD: https://en.wikipedia.org/wiki/Top-level_domain
+.. _Scrapyd documentation: https://scrapyd.readthedocs.io/en/latest/


Code review comments:
diff_hunk: @@ -394,16 +394,16 @@ New features
     :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`
     or
     :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception`
-    with a custom :class:`~scrapy.http.Request` object assigned to
+    with a custom :class:`~scrapy.Request` object assigned to
     :class:`response.request <scrapy.http.Response.request>`:
 
     -   The response is handled by the callback of that custom
-        :class:`~scrapy.http.Request` object, instead of being handled by the
-        callback of the original :class:`~scrapy.http.Request` object
+        :class:`~scrapy.Request` object, instead of being handled by the
+        callback of the original :class:`~scrapy.Request` object
 
-    -   That custom :class:`~scrapy.http.Request` object is now sent as the
+    -   That custom :class:`~scrapy..Request` object is now sent as the
path: docs/news.rst
text: @user1:
```suggestion
    -   That custom :class:`~scrapy.Request` object is now sent as the
```

diff_hunk: @@ -1918,13 +1918,13 @@ The following deprecated APIs have been removed (:issue:`3578`):
 
 *   From both ``scrapy.selector`` and ``scrapy.selector.lxmlsel``:
 
-    *   ``HtmlXPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``HtmlXPathSelector`` (use :class:`~scrapy.Selector`)
 
-    *   ``XmlXPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XmlXPathSelector`` (use :class:`~scrapy.Selector`)
 
-    *   ``XPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XPathSelector`` (use :class:`~scrapy..Selector`)
path: docs/news.rst
text: @user1:
```suggestion
    *   ``XPathSelector`` (use :class:`~scrapy.Selector`)
```

@user2:
Thank you for pointing out these errors. If you find anything else please let me know.