Repository: scrapy/scrapy

Issue:
Ability to retry a request from inside a spider callback
There are situations where websites return 200 responses but the content is not available due to bans or temporal issues which can be fixed by retrying requests.

There should be an easier way to retry requests inside spider callbacks, which should ideally reuse the code in Retry downloader middleware.

I see two approaches for this.

1) Introduce new exception called RetryRequest which can be raised inside a spider callback to indicate a retry. I personally prefer this but the implementation of this is a little untidy due to this bug #220

    ```
    from scrapy.exceptions import RetryRequest

    def parse(self, response):
        if response.xpath('//title[text()="Content not found"]'):
            raise RetryRequest('Missing content')
    ```
2) Introduce a new class RetryRequest which wraps a request that needs to be retried. A RetryRequest can be yielded from a spider callback to indicate a retry

    ```
    from scrapy.http import RetryRequest

    def parse(self, response):
        if response.xpath('//title[text()="Content not found"]'):
            yield RetryRequest(response.request, reason='Missing content')
    ```

Will be sending two PRs for the two approaches. Happy to hear about any other alternatives too.
Retry request from spider callback
New approach for #3590

This,
1) provides a method retry_request() in response class that will return a request instance that is marked as to be retried
2) A spidermiddleware (RetryMiddleware) will check whether this request can be retried, do the necessary modifications and retry the request

Resolves #3590


Discussion:
What aspects of the [current approach](https://stackoverflow.com/a/28640118/939364) don’t you like? It can be done with a one-liner (`yield response.request.replace(dont_filter=True)`).
@Gallaecio I miss everything the retry middleware currently does.

You can't infinitely do this. you need to count the retries and give up after a certain retry attempts
You need to have stats and log messages

IMO there should be a easier way to do above things than re-implement the same code in every project/spider.
I think the idea is nice, though I think you should focus on only one approach.
IMHO, number 1 is better because it makes more sense to retry when some specific exception happened.
I don't think a special kind of Request would be the best approach here.

Also, I think that we should focus on only one of them because _There should be one-- and preferably only one --obvious way to do it._
I frequently, comfortably, and successfully use the `raise RetryRequest("a reason")` approach. It counts and limits retries, logs the _reason_, and, in general, does what it should without much work from the developer. As the OP mentioned, corrupted content over 200 responses is frequent enough.
What about attaching the middleware routine to the response:

    class RetryMiddleware(scrapy.downloadermiddlewares.retry.RetryMiddleware):

        def process_response(self, request, response, spider):
            """Attach a .retry() method to every response"""
            response = super().process_response(request, response, spider)

            def retry(**kwargs):
                reason = kwargs.pop('reason', 'response_retry_called')
                retryreq = request.replace(**kwargs)
                return self._retry(retryreq, reason, spider)

            if isinstance(response, scrapy.http.Response):
                response.retry = retry

            return response

Then it could be used like:

    def parse(self, response):
        if response.xpath('//title[text()="Content not found"]'):
            yield response.retry(reason='Missing content')

@stav, from the API user point of view, I don't know if `response.retry` gives enough meaning that we are, actually, retrying the request.
What about:

            yield response.request.retry(reason='Missing content')
I like `yield response.request.retry(reason='Missing content')`, except it will often require a `return` afterwards.

At any rate, the solution should accept the same parameters as `request.replace()`, do whatever is necessary so the new request is not detected as a duplicate, and keep a retry count and limit.

It could be both, with `raise RetryRequest()` implemented in terms of `Request.retry()`.
> it will often require a `return` afterwards

We could also just have:

    return [response.request.retry(reason='Missing content')]

> It could be both, with `raise RetryRequest()`

What benefit does allowing `parse` to raise an exception give us?  I'm always confused with whose responsibility it is to catch exceptions, so this is probably just my ignorance :confused: 

> What benefit does allowing `parse` to raise an exception give us?

Good question!

While implementing session pooling I wrote a session-aware `retry_request()`, and never missed not having an exception for the functionality.
We can call [_retry](https://github.com/scrapy/scrapy/blob/1.5.0/scrapy/downloadermiddlewares/retry.py#L63-L91) method of Retry middleware from the spider code

```
class Myspider(scrapy.Spider):
.....
   def start_requests(self):
        downloader_middlewares = self.crawler.engine.downloader.middleware.middlewares
        self.RetryMiddleware = [middleware for middleware in downloader_middlewares if "RetryMiddleware" in str(type(middleware))][0]
.....

    def parse(self, response):
        if response.xpath('//title[text()="Content not found"]'):
            yield self.RetryMiddleware._retry(response.request,"some reason",self)
            return
```
@GeorgeA92, I don't like this approach :(
My problem is that we are calling a private method from the outside.
@ejulio  I agree.
There are 2 non private methods where `_retry `method called.

[process_requests ](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/retry.py#L50-L56)calls `_retry` if response code listed in `retry_http_codes` (from `RETRY_HTTP_CODES` setting). 
This is not our case because 200 response is not in `retry_http_codes `and response needs to pass all middlewares (including `RetryMiddleware`) in order to reach spider callback (and retry request from spider callback).

[process_exception ](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/retry.py#L58-L91)calls `_retry `if exception argument listed in `EXCEPTIONS_TO_RETRY `which is stored as tuple.
https://github.com/scrapy/scrapy/blob/c72ab1d4ba5dad3c68b12c473fa55b7f1f144834/scrapy/downloadermiddlewares/retry.py#L34-L37

We can use non private `process_exception `method instead of `_retry `in spider code after following:
1.define new `Exception` 
2.add our custom `Exception `in `EXCEPTION_TO_RETRY `in `RetryMiddleware` object
3.call `process_exception ` method with our new `exception `as argument
Code is slightly different:

```
class ContentNotFoundException(Exception):
    """
    ContentNotFound
    """

class Myspider(scrapy.Spider):
.....
   def start_requests(self):
        downloader_middlewares = self.crawler.engine.downloader.middleware.middlewares
        self.RetryMiddleware = [middleware for middleware in downloader_middlewares if "RetryMiddleware" in str(type(middleware))][0]
	self.RetryMiddleware.EXCEPTIONS_TO_RETRY = tuple(list(self.RetryMiddleware.EXCEPTIONS_TO_RETRY)+[ContentNotFoundException])
.....

    def parse(self, response):
        if response.xpath('//title[text()="Content not found"]'):
            yield self.RetryMiddleware.process_exception(response.request, ContentNotFoundException(), self)
            return
```
After going through the different ideas, I opted for Response.retry_request() over raising a RetryException() to indicate a retry.

This new method will just return a request that is marked as to be retried and actual modification needed to retry the request is been handled in a new spidermiddleware. The reason is you need access to spider settings to determine whether the request has reached the maximum retry attempts etc.

please have a look at https://github.com/scrapy/scrapy/pull/3685 
Any progress on this? 
Any update on this? 

I also encounter the issue mentioned by @kasun 
use try and except to handle retry request, and for counting the number of retry set a variable in the "meta", update that in the except section. 

that is wrong?? @kasun 
Any update on this?
# [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/3685?src=pr&el=h1) Report
> Merging [#3685](https://codecov.io/gh/scrapy/scrapy/pull/3685?src=pr&el=desc) into [master](https://codecov.io/gh/scrapy/scrapy/commit/c72ab1d4ba5dad3c68b12c473fa55b7f1f144834&el=desc) will **increase** coverage by `0.14%`.
> The diff coverage is `97.67%`.


```diff
@@            Coverage Diff             @@
##           master    #3685      +/-   ##
==========================================
+ Coverage   84.52%   84.67%   +0.14%     
==========================================
  Files         167      169       +2     
  Lines        9410     9566     +156     
  Branches     1397     1425      +28     
==========================================
+ Hits         7954     8100     +146     
- Misses       1199     1206       +7     
- Partials      257      260       +3     
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/3685?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [scrapy/settings/default\_settings.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L3NldHRpbmdzL2RlZmF1bHRfc2V0dGluZ3MucHk=) | `98.63% <ø> (ø)` | |
| [scrapy/http/response/\_\_init\_\_.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L2h0dHAvcmVzcG9uc2UvX19pbml0X18ucHk=) | `90.90% <60.00%> (-2.54%)` | :arrow_down: |
| [scrapy/downloadermiddlewares/retry.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9yZXRyeS5weQ==) | `94.59% <100.00%> (-1.24%)` | :arrow_down: |
| [scrapy/http/request/\_\_init\_\_.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L2h0dHAvcmVxdWVzdC9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | |
| [scrapy/spidermiddlewares/retry.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L3NwaWRlcm1pZGRsZXdhcmVzL3JldHJ5LnB5) | `100.00% <100.00%> (ø)` | |
| [scrapy/utils/retry.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L3V0aWxzL3JldHJ5LnB5) | `100.00% <100.00%> (ø)` | |
| [scrapy/crawler.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L2NyYXdsZXIucHk=) | `91.60% <0.00%> (-0.48%)` | :arrow_down: |
| [scrapy/linkextractors/sgml.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L2xpbmtleHRyYWN0b3JzL3NnbWwucHk=) | `96.80% <0.00%> (ø)` | |
| [scrapy/linkextractors/\_\_init\_\_.py](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree#diff-c2NyYXB5L2xpbmtleHRyYWN0b3JzL19faW5pdF9fLnB5) | `100.00% <0.00%> (ø)` | |
| ... and [3 more](https://codecov.io/gh/scrapy/scrapy/pull/3685/diff?src=pr&el=tree-more) | |

Any update?
Any updates?

Resolving patch:
diff --git a/scrapy/downloadermiddlewares/retry.py b/scrapy/downloadermiddlewares/retry.py
--- a/scrapy/downloadermiddlewares/retry.py
+++ b/scrapy/downloadermiddlewares/retry.py
@@ -9,7 +9,8 @@
 Failed pages are collected on the scraping process and rescheduled at the end,
 once the spider has finished crawling all regular (non failed) pages.
 """
-import logging
+from logging import getLogger, Logger
+from typing import Optional, Union
 
 from twisted.internet import defer
 from twisted.internet.error import (
@@ -23,12 +24,104 @@
 )
 from twisted.web.client import ResponseFailed
 
-from scrapy.exceptions import NotConfigured
-from scrapy.utils.response import response_status_message
 from scrapy.core.downloader.handlers.http11 import TunnelError
+from scrapy.exceptions import NotConfigured
+from scrapy.http.request import Request
+from scrapy.spiders import Spider
 from scrapy.utils.python import global_object_name
+from scrapy.utils.response import response_status_message
+
 
-logger = logging.getLogger(__name__)
+retry_logger = getLogger(__name__)
+
+
+def get_retry_request(
+    request: Request,
+    *,
+    spider: Spider,
+    reason: Union[str, Exception] = 'unspecified',
+    max_retry_times: Optional[int] = None,
+    priority_adjust: Optional[int] = None,
+    logger: Logger = retry_logger,
+    stats_base_key: str = 'retry',
+):
+    """
+    Returns a new :class:`~scrapy.Request` object to retry the specified
+    request, or ``None`` if retries of the specified request have been
+    exhausted.
+
+    For example, in a :class:`~scrapy.Spider` callback, you could use it as
+    follows::
+
+        def parse(self, response):
+            if not response.text:
+                new_request_or_none = get_retry_request(
+                    response.request,
+                    spider=self,
+                    reason='empty',
+                )
+                return new_request_or_none
+
+    *spider* is the :class:`~scrapy.Spider` instance which is asking for the
+    retry request. It is used to access the :ref:`settings <topics-settings>`
+    and :ref:`stats <topics-stats>`, and to provide extra logging context (see
+    :func:`logging.debug`).
+
+    *reason* is a string or an :class:`Exception` object that indicates the
+    reason why the request needs to be retried. It is used to name retry stats.
+
+    *max_retry_times* is a number that determines the maximum number of times
+    that *request* can be retried. If not specified or ``None``, the number is
+    read from the :reqmeta:`max_retry_times` meta key of the request. If the
+    :reqmeta:`max_retry_times` meta key is not defined or ``None``, the number
+    is read from the :setting:`RETRY_TIMES` setting.
+
+    *priority_adjust* is a number that determines how the priority of the new
+    request changes in relation to *request*. If not specified, the number is
+    read from the :setting:`RETRY_PRIORITY_ADJUST` setting.
+
+    *logger* is the logging.Logger object to be used when logging messages
+
+    *stats_base_key* is a string to be used as the base key for the
+    retry-related job stats
+    """
+    settings = spider.crawler.settings
+    stats = spider.crawler.stats
+    retry_times = request.meta.get('retry_times', 0) + 1
+    if max_retry_times is None:
+        max_retry_times = request.meta.get('max_retry_times')
+        if max_retry_times is None:
+            max_retry_times = settings.getint('RETRY_TIMES')
+    if retry_times <= max_retry_times:
+        logger.debug(
+            "Retrying %(request)s (failed %(retry_times)d times): %(reason)s",
+            {'request': request, 'retry_times': retry_times, 'reason': reason},
+            extra={'spider': spider}
+        )
+        new_request = request.copy()
+        new_request.meta['retry_times'] = retry_times
+        new_request.dont_filter = True
+        if priority_adjust is None:
+            priority_adjust = settings.getint('RETRY_PRIORITY_ADJUST')
+        new_request.priority = request.priority + priority_adjust
+
+        if callable(reason):
+            reason = reason()
+        if isinstance(reason, Exception):
+            reason = global_object_name(reason.__class__)
+
+        stats.inc_value(f'{stats_base_key}/count')
+        stats.inc_value(f'{stats_base_key}/reason_count/{reason}')
+        return new_request
+    else:
+        stats.inc_value(f'{stats_base_key}/max_reached')
+        logger.error(
+            "Gave up retrying %(request)s (failed %(retry_times)d times): "
+            "%(reason)s",
+            {'request': request, 'retry_times': retry_times, 'reason': reason},
+            extra={'spider': spider},
+        )
+        return None
 
 
 class RetryMiddleware:
@@ -67,31 +160,12 @@ def process_exception(self, request, exception, spider):
             return self._retry(request, exception, spider)
 
     def _retry(self, request, reason, spider):
-        retries = request.meta.get('retry_times', 0) + 1
-
-        retry_times = self.max_retry_times
-
-        if 'max_retry_times' in request.meta:
-            retry_times = request.meta['max_retry_times']
-
-        stats = spider.crawler.stats
-        if retries <= retry_times:
-            logger.debug("Retrying %(request)s (failed %(retries)d times): %(reason)s",
-                         {'request': request, 'retries': retries, 'reason': reason},
-                         extra={'spider': spider})
-            retryreq = request.copy()
-            retryreq.meta['retry_times'] = retries
-            retryreq.dont_filter = True
-            retryreq.priority = request.priority + self.priority_adjust
-
-            if isinstance(reason, Exception):
-                reason = global_object_name(reason.__class__)
-
-            stats.inc_value('retry/count')
-            stats.inc_value(f'retry/reason_count/{reason}')
-            return retryreq
-        else:
-            stats.inc_value('retry/max_reached')
-            logger.error("Gave up retrying %(request)s (failed %(retries)d times): %(reason)s",
-                         {'request': request, 'retries': retries, 'reason': reason},
-                         extra={'spider': spider})
+        max_retry_times = request.meta.get('max_retry_times', self.max_retry_times)
+        priority_adjust = request.meta.get('priority_adjust', self.priority_adjust)
+        return get_retry_request(
+            request,
+            reason=reason,
+            spider=spider,
+            max_retry_times=max_retry_times,
+            priority_adjust=priority_adjust,
+        )


Test patch:
diff --git a/tests/test_downloadermiddleware_retry.py b/tests/test_downloadermiddleware_retry.py
--- a/tests/test_downloadermiddleware_retry.py
+++ b/tests/test_downloadermiddleware_retry.py
@@ -1,19 +1,21 @@
+import logging
 import unittest
+
+from testfixtures import LogCapture
 from twisted.internet import defer
 from twisted.internet.error import (
     ConnectError,
     ConnectionDone,
     ConnectionLost,
-    ConnectionRefusedError,
     DNSLookupError,
     TCPTimedOutError,
-    TimeoutError,
 )
 from twisted.web.client import ResponseFailed
 
-from scrapy.downloadermiddlewares.retry import RetryMiddleware
-from scrapy.spiders import Spider
+from scrapy.downloadermiddlewares.retry import get_retry_request, RetryMiddleware
+from scrapy.exceptions import IgnoreRequest
 from scrapy.http import Request, Response
+from scrapy.spiders import Spider
 from scrapy.utils.test import get_crawler
 
 
@@ -119,82 +121,514 @@ def _test_retry_exception(self, req, exception):
 
 
 class MaxRetryTimesTest(unittest.TestCase):
-    def setUp(self):
-        self.crawler = get_crawler(Spider)
-        self.spider = self.crawler._create_spider('foo')
-        self.mw = RetryMiddleware.from_crawler(self.crawler)
-        self.mw.max_retry_times = 2
-        self.invalid_url = 'http://www.scrapytest.org/invalid_url'
 
-    def test_with_settings_zero(self):
+    invalid_url = 'http://www.scrapytest.org/invalid_url'
 
-        # SETTINGS: RETRY_TIMES = 0
-        self.mw.max_retry_times = 0
+    def get_spider_and_middleware(self, settings=None):
+        crawler = get_crawler(Spider, settings or {})
+        spider = crawler._create_spider('foo')
+        middleware = RetryMiddleware.from_crawler(crawler)
+        return spider, middleware
 
+    def test_with_settings_zero(self):
+        max_retry_times = 0
+        settings = {'RETRY_TIMES': max_retry_times}
+        spider, middleware = self.get_spider_and_middleware(settings)
         req = Request(self.invalid_url)
-        self._test_retry(req, DNSLookupError('foo'), self.mw.max_retry_times)
+        self._test_retry(
+            req,
+            DNSLookupError('foo'),
+            max_retry_times,
+            spider=spider,
+            middleware=middleware,
+        )
 
     def test_with_metakey_zero(self):
-
-        # SETTINGS: meta(max_retry_times) = 0
-        meta_max_retry_times = 0
-
-        req = Request(self.invalid_url, meta={'max_retry_times': meta_max_retry_times})
-        self._test_retry(req, DNSLookupError('foo'), meta_max_retry_times)
+        max_retry_times = 0
+        spider, middleware = self.get_spider_and_middleware()
+        meta = {'max_retry_times': max_retry_times}
+        req = Request(self.invalid_url, meta=meta)
+        self._test_retry(
+            req,
+            DNSLookupError('foo'),
+            max_retry_times,
+            spider=spider,
+            middleware=middleware,
+        )
 
     def test_without_metakey(self):
-
-        # SETTINGS: RETRY_TIMES is NON-ZERO
-        self.mw.max_retry_times = 5
-
+        max_retry_times = 5
+        settings = {'RETRY_TIMES': max_retry_times}
+        spider, middleware = self.get_spider_and_middleware(settings)
         req = Request(self.invalid_url)
-        self._test_retry(req, DNSLookupError('foo'), self.mw.max_retry_times)
+        self._test_retry(
+            req,
+            DNSLookupError('foo'),
+            max_retry_times,
+            spider=spider,
+            middleware=middleware,
+        )
 
     def test_with_metakey_greater(self):
-
-        # SETINGS: RETRY_TIMES < meta(max_retry_times)
-        self.mw.max_retry_times = 2
         meta_max_retry_times = 3
+        middleware_max_retry_times = 2
 
         req1 = Request(self.invalid_url, meta={'max_retry_times': meta_max_retry_times})
         req2 = Request(self.invalid_url)
 
-        self._test_retry(req1, DNSLookupError('foo'), meta_max_retry_times)
-        self._test_retry(req2, DNSLookupError('foo'), self.mw.max_retry_times)
+        settings = {'RETRY_TIMES': middleware_max_retry_times}
+        spider, middleware = self.get_spider_and_middleware(settings)
+
+        self._test_retry(
+            req1,
+            DNSLookupError('foo'),
+            meta_max_retry_times,
+            spider=spider,
+            middleware=middleware,
+        )
+        self._test_retry(
+            req2,
+            DNSLookupError('foo'),
+            middleware_max_retry_times,
+            spider=spider,
+            middleware=middleware,
+        )
 
     def test_with_metakey_lesser(self):
-
-        # SETINGS: RETRY_TIMES > meta(max_retry_times)
-        self.mw.max_retry_times = 5
         meta_max_retry_times = 4
+        middleware_max_retry_times = 5
 
         req1 = Request(self.invalid_url, meta={'max_retry_times': meta_max_retry_times})
         req2 = Request(self.invalid_url)
 
-        self._test_retry(req1, DNSLookupError('foo'), meta_max_retry_times)
-        self._test_retry(req2, DNSLookupError('foo'), self.mw.max_retry_times)
+        settings = {'RETRY_TIMES': middleware_max_retry_times}
+        spider, middleware = self.get_spider_and_middleware(settings)
+
+        self._test_retry(
+            req1,
+            DNSLookupError('foo'),
+            meta_max_retry_times,
+            spider=spider,
+            middleware=middleware,
+        )
+        self._test_retry(
+            req2,
+            DNSLookupError('foo'),
+            middleware_max_retry_times,
+            spider=spider,
+            middleware=middleware,
+        )
 
     def test_with_dont_retry(self):
-
-        # SETTINGS: meta(max_retry_times) = 4
-        meta_max_retry_times = 4
-
-        req = Request(self.invalid_url, meta={
-            'max_retry_times': meta_max_retry_times, 'dont_retry': True
-        })
-
-        self._test_retry(req, DNSLookupError('foo'), 0)
-
-    def _test_retry(self, req, exception, max_retry_times):
+        max_retry_times = 4
+        spider, middleware = self.get_spider_and_middleware()
+        meta = {
+            'max_retry_times': max_retry_times,
+            'dont_retry': True,
+        }
+        req = Request(self.invalid_url, meta=meta)
+        self._test_retry(
+            req,
+            DNSLookupError('foo'),
+            0,
+            spider=spider,
+            middleware=middleware,
+        )
+
+    def _test_retry(
+        self,
+        req,
+        exception,
+        max_retry_times,
+        spider=None,
+        middleware=None,
+    ):
+        spider = spider or self.spider
+        middleware = middleware or self.mw
 
         for i in range(0, max_retry_times):
-            req = self.mw.process_exception(req, exception, self.spider)
+            req = middleware.process_exception(req, exception, spider)
             assert isinstance(req, Request)
 
         # discard it
-        req = self.mw.process_exception(req, exception, self.spider)
+        req = middleware.process_exception(req, exception, spider)
         self.assertEqual(req, None)
 
 
+class GetRetryRequestTest(unittest.TestCase):
+
+    def get_spider(self, settings=None):
+        crawler = get_crawler(Spider, settings or {})
+        return crawler._create_spider('foo')
+
+    def test_basic_usage(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        with LogCapture() as log:
+            new_request = get_retry_request(
+                request,
+                spider=spider,
+            )
+        self.assertIsInstance(new_request, Request)
+        self.assertNotEqual(new_request, request)
+        self.assertEqual(new_request.dont_filter, True)
+        expected_retry_times = 1
+        self.assertEqual(new_request.meta['retry_times'], expected_retry_times)
+        self.assertEqual(new_request.priority, -1)
+        expected_reason = "unspecified"
+        for stat in ('retry/count', f'retry/reason_count/{expected_reason}'):
+            self.assertEqual(spider.crawler.stats.get_value(stat), 1)
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "DEBUG",
+                f"Retrying {request} (failed {expected_retry_times} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_max_retries_reached(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        max_retry_times = 0
+        with LogCapture() as log:
+            new_request = get_retry_request(
+                request,
+                spider=spider,
+                max_retry_times=max_retry_times,
+            )
+        self.assertEqual(new_request, None)
+        self.assertEqual(
+            spider.crawler.stats.get_value('retry/max_reached'),
+            1
+        )
+        failure_count = max_retry_times + 1
+        expected_reason = "unspecified"
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "ERROR",
+                f"Gave up retrying {request} (failed {failure_count} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_one_retry(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        with LogCapture() as log:
+            new_request = get_retry_request(
+                request,
+                spider=spider,
+                max_retry_times=1,
+            )
+        self.assertIsInstance(new_request, Request)
+        self.assertNotEqual(new_request, request)
+        self.assertEqual(new_request.dont_filter, True)
+        expected_retry_times = 1
+        self.assertEqual(new_request.meta['retry_times'], expected_retry_times)
+        self.assertEqual(new_request.priority, -1)
+        expected_reason = "unspecified"
+        for stat in ('retry/count', f'retry/reason_count/{expected_reason}'):
+            self.assertEqual(spider.crawler.stats.get_value(stat), 1)
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "DEBUG",
+                f"Retrying {request} (failed {expected_retry_times} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_two_retries(self):
+        spider = self.get_spider()
+        request = Request('https://example.com')
+        new_request = request
+        max_retry_times = 2
+        for index in range(max_retry_times):
+            with LogCapture() as log:
+                new_request = get_retry_request(
+                    new_request,
+                    spider=spider,
+                    max_retry_times=max_retry_times,
+                )
+            self.assertIsInstance(new_request, Request)
+            self.assertNotEqual(new_request, request)
+            self.assertEqual(new_request.dont_filter, True)
+            expected_retry_times = index + 1
+            self.assertEqual(new_request.meta['retry_times'], expected_retry_times)
+            self.assertEqual(new_request.priority, -expected_retry_times)
+            expected_reason = "unspecified"
+            for stat in ('retry/count', f'retry/reason_count/{expected_reason}'):
+                value = spider.crawler.stats.get_value(stat)
+                self.assertEqual(value, expected_retry_times)
+            log.check_present(
+                (
+                    "scrapy.downloadermiddlewares.retry",
+                    "DEBUG",
+                    f"Retrying {request} (failed {expected_retry_times} times): "
+                    f"{expected_reason}",
+                )
+            )
+
+        with LogCapture() as log:
+            new_request = get_retry_request(
+                new_request,
+                spider=spider,
+                max_retry_times=max_retry_times,
+            )
+        self.assertEqual(new_request, None)
+        self.assertEqual(
+            spider.crawler.stats.get_value('retry/max_reached'),
+            1
+        )
+        failure_count = max_retry_times + 1
+        expected_reason = "unspecified"
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "ERROR",
+                f"Gave up retrying {request} (failed {failure_count} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_no_spider(self):
+        request = Request('https://example.com')
+        with self.assertRaises(TypeError):
+            get_retry_request(request)  # pylint: disable=missing-kwoa
+
+    def test_max_retry_times_setting(self):
+        max_retry_times = 0
+        spider = self.get_spider({'RETRY_TIMES': max_retry_times})
+        request = Request('https://example.com')
+        new_request = get_retry_request(
+            request,
+            spider=spider,
+        )
+        self.assertEqual(new_request, None)
+
+    def test_max_retry_times_meta(self):
+        max_retry_times = 0
+        spider = self.get_spider({'RETRY_TIMES': max_retry_times + 1})
+        meta = {'max_retry_times': max_retry_times}
+        request = Request('https://example.com', meta=meta)
+        new_request = get_retry_request(
+            request,
+            spider=spider,
+        )
+        self.assertEqual(new_request, None)
+
+    def test_max_retry_times_argument(self):
+        max_retry_times = 0
+        spider = self.get_spider({'RETRY_TIMES': max_retry_times + 1})
+        meta = {'max_retry_times': max_retry_times + 1}
+        request = Request('https://example.com', meta=meta)
+        new_request = get_retry_request(
+            request,
+            spider=spider,
+            max_retry_times=max_retry_times,
+        )
+        self.assertEqual(new_request, None)
+
+    def test_priority_adjust_setting(self):
+        priority_adjust = 1
+        spider = self.get_spider({'RETRY_PRIORITY_ADJUST': priority_adjust})
+        request = Request('https://example.com')
+        new_request = get_retry_request(
+            request,
+            spider=spider,
+        )
+        self.assertEqual(new_request.priority, priority_adjust)
+
+    def test_priority_adjust_argument(self):
+        priority_adjust = 1
+        spider = self.get_spider({'RETRY_PRIORITY_ADJUST': priority_adjust + 1})
+        request = Request('https://example.com')
+        new_request = get_retry_request(
+            request,
+            spider=spider,
+            priority_adjust=priority_adjust,
+        )
+        self.assertEqual(new_request.priority, priority_adjust)
+
+    def test_log_extra_retry_success(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        with LogCapture(attributes=('spider',)) as log:
+            get_retry_request(
+                request,
+                spider=spider,
+            )
+        log.check_present(spider)
+
+    def test_log_extra_retries_exceeded(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        with LogCapture(attributes=('spider',)) as log:
+            get_retry_request(
+                request,
+                spider=spider,
+                max_retry_times=0,
+            )
+        log.check_present(spider)
+
+    def test_reason_string(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        expected_reason = 'because'
+        with LogCapture() as log:
+            get_retry_request(
+                request,
+                spider=spider,
+                reason=expected_reason,
+            )
+        expected_retry_times = 1
+        for stat in ('retry/count', f'retry/reason_count/{expected_reason}'):
+            self.assertEqual(spider.crawler.stats.get_value(stat), 1)
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "DEBUG",
+                f"Retrying {request} (failed {expected_retry_times} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_reason_builtin_exception(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        expected_reason = NotImplementedError()
+        expected_reason_string = 'builtins.NotImplementedError'
+        with LogCapture() as log:
+            get_retry_request(
+                request,
+                spider=spider,
+                reason=expected_reason,
+            )
+        expected_retry_times = 1
+        stat = spider.crawler.stats.get_value(
+            f'retry/reason_count/{expected_reason_string}'
+        )
+        self.assertEqual(stat, 1)
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "DEBUG",
+                f"Retrying {request} (failed {expected_retry_times} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_reason_builtin_exception_class(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        expected_reason = NotImplementedError
+        expected_reason_string = 'builtins.NotImplementedError'
+        with LogCapture() as log:
+            get_retry_request(
+                request,
+                spider=spider,
+                reason=expected_reason,
+            )
+        expected_retry_times = 1
+        stat = spider.crawler.stats.get_value(
+            f'retry/reason_count/{expected_reason_string}'
+        )
+        self.assertEqual(stat, 1)
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "DEBUG",
+                f"Retrying {request} (failed {expected_retry_times} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_reason_custom_exception(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        expected_reason = IgnoreRequest()
+        expected_reason_string = 'scrapy.exceptions.IgnoreRequest'
+        with LogCapture() as log:
+            get_retry_request(
+                request,
+                spider=spider,
+                reason=expected_reason,
+            )
+        expected_retry_times = 1
+        stat = spider.crawler.stats.get_value(
+            f'retry/reason_count/{expected_reason_string}'
+        )
+        self.assertEqual(stat, 1)
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "DEBUG",
+                f"Retrying {request} (failed {expected_retry_times} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_reason_custom_exception_class(self):
+        request = Request('https://example.com')
+        spider = self.get_spider()
+        expected_reason = IgnoreRequest
+        expected_reason_string = 'scrapy.exceptions.IgnoreRequest'
+        with LogCapture() as log:
+            get_retry_request(
+                request,
+                spider=spider,
+                reason=expected_reason,
+            )
+        expected_retry_times = 1
+        stat = spider.crawler.stats.get_value(
+            f'retry/reason_count/{expected_reason_string}'
+        )
+        self.assertEqual(stat, 1)
+        log.check_present(
+            (
+                "scrapy.downloadermiddlewares.retry",
+                "DEBUG",
+                f"Retrying {request} (failed {expected_retry_times} times): "
+                f"{expected_reason}",
+            )
+        )
+
+    def test_custom_logger(self):
+        logger = logging.getLogger("custom-logger")
+        request = Request("https://example.com")
+        spider = self.get_spider()
+        expected_reason = "because"
+        with LogCapture() as log:
+            get_retry_request(
+                request,
+                spider=spider,
+                reason=expected_reason,
+                logger=logger,
+            )
+        log.check_present(
+            (
+                "custom-logger",
+                "DEBUG",
+                f"Retrying {request} (failed 1 times): {expected_reason}",
+            )
+        )
+
+    def test_custom_stats_key(self):
+        request = Request("https://example.com")
+        spider = self.get_spider()
+        expected_reason = "because"
+        stats_key = "custom_retry"
+        get_retry_request(
+            request,
+            spider=spider,
+            reason=expected_reason,
+            stats_base_key=stats_key,
+        )
+        for stat in (f"{stats_key}/count", f"{stats_key}/reason_count/{expected_reason}"):
+            self.assertEqual(spider.crawler.stats.get_value(stat), 1)
+
+
 if __name__ == "__main__":
     unittest.main()