Repository: scrapy/scrapy
Pull request #6274 (2024-03-06 19:01:30)

Improve typing for Spider.parse().
Based on https://stackoverflow.com/questions/77160087/correct-typing-of-a-method-without-a-fixed-signature

Should remove most of the mypy errors for parse() in user spiders (if they use older mypy Spider.parse() will appear untyped so no override error either). Doesn't help with scrapy-poet's callbacks that narrow down `Response` to `DummyResponse`: https://github.com/scrapinghub/scrapy-poet/issues/179

This should probably have typing tests, I'll do that later.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#6274](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (84056f4) into [master](https://app.codecov.io/gh/scrapy/scrapy/commit/861646fbb36c8491a83e46cb749089210e0733a8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (861646f) will **decrease** coverage by `46.17%`.
> The diff coverage is `12.50%`.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           master    #6274       +/-   ##
===========================================
- Coverage   88.82%   42.65%   -46.17%     
===========================================
  Files         161      161               
  Lines       11863    11859        -4     
  Branches     1918     1745      -173     
===========================================
- Hits        10537     5059     -5478     
- Misses        979     6388     +5409     
- Partials      347      412       +65     
```

| [Files](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/commands/bench.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2JlbmNoLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [scrapy/spiders/\_\_init\_\_.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3NwaWRlcnMvX19pbml0X18ucHk=) | `73.52% <14.28%> (-20.23%)` | :arrow_down: |

... and [139 files with indirect coverage changes](https://app.codecov.io/gh/scrapy/scrapy/pull/6274/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy)

</details>

Patch:
diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml
index d6fc0f6c5..ed1629b67 100644
--- a/.github/workflows/checks.yml
+++ b/.github/workflows/checks.yml
@@ -18,6 +18,9 @@ jobs:
         - python-version: 3.8
           env:
             TOXENV: typing
+        - python-version: 3.8
+          env:
+            TOXENV: typing-tests
         - python-version: "3.11"  # Keep in sync with .readthedocs.yml
           env:
             TOXENV: docs
diff --git a/MANIFEST.in b/MANIFEST.in
index 4920dc0c3..06971e39c 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,9 +1,8 @@
-include README.rst
-include AUTHORS
-include INSTALL
-include LICENSE
-include MANIFEST.in
+include CODE_OF_CONDUCT.md
+include CONTRIBUTING.md
+include INSTALL.md
 include NEWS
+include SECURITY.md
 
 include scrapy/VERSION
 include scrapy/mime.types
@@ -12,16 +11,13 @@ include scrapy/py.typed
 include codecov.yml
 include conftest.py
 include pytest.ini
-include requirements-*.txt
 include tox.ini
 
 recursive-include scrapy/templates *
-recursive-include scrapy license.txt
 recursive-include docs *
 prune docs/build
 
 recursive-include extras *
-recursive-include bin *
 recursive-include tests *
 
 global-exclude __pycache__ *.py[cod]
diff --git a/docs/news.rst b/docs/news.rst
index fafea0bf8..7db4e59a1 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -3,6 +3,20 @@
 Release notes
 =============
 
+
+.. _release-VERSION:
+
+Scrapy VERSION (YYYY-MM-DD)
+---------------------------
+
+Deprecations
+~~~~~~~~~~~~
+
+-   :func:`scrapy.core.downloader.Downloader._get_slot_key` is now deprecated.
+    Consider using its corresponding public method get_slot_key() instead.
+    (:issue:`6340`)
+
+
 .. _release-2.11.1:
 
 Scrapy 2.11.1 (2024-02-14)
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 1abbc4968..d4cd062fe 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -884,6 +884,10 @@ Meta tags within these tags are ignored.
    The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
    ``['script', 'noscript']`` to ``[]``.
 
+.. versionchanged:: VERSION
+   The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
+   ``[]`` to ``['noscript']``.
+
 .. setting:: METAREFRESH_MAXDELAY
 
 METAREFRESH_MAXDELAY
diff --git a/scrapy/commands/bench.py b/scrapy/commands/bench.py
index 7523f3cfe..0c4ebcd23 100644
--- a/scrapy/commands/bench.py
+++ b/scrapy/commands/bench.py
@@ -60,7 +60,7 @@ class _BenchSpider(scrapy.Spider):
         url = f"{self.baseurl}?{urlencode(qargs, doseq=True)}"
         return [scrapy.Request(url, dont_filter=True)]
 
-    def parse(self, response: Response) -> Any:  # type: ignore[override]
+    def parse(self, response: Response) -> Any:
         assert isinstance(Response, TextResponse)
         for link in self.link_extractor.extract_links(response):
             yield scrapy.Request(link.url, callback=self.parse)
diff --git a/scrapy/core/downloader/__init__.py b/scrapy/core/downloader/__init__.py
index 98e1af6fb..0ab3bdb77 100644
--- a/scrapy/core/downloader/__init__.py
+++ b/scrapy/core/downloader/__init__.py
@@ -1,4 +1,5 @@
 import random
+import warnings
 from collections import deque
 from datetime import datetime
 from time import time
@@ -10,6 +11,7 @@ from twisted.internet.defer import Deferred
 from scrapy import Request, Spider, signals
 from scrapy.core.downloader.handlers import DownloadHandlers
 from scrapy.core.downloader.middleware import DownloaderMiddlewareManager
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import Response
 from scrapy.resolver import dnscache
 from scrapy.settings import BaseSettings
@@ -125,7 +127,7 @@ class Downloader:
         return len(self.active) >= self.total_concurrency
 
     def _get_slot(self, request: Request, spider: Spider) -> Tuple[str, Slot]:
-        key = self._get_slot_key(request, spider)
+        key = self.get_slot_key(request)
         if key not in self.slots:
             slot_settings = self.per_slot_settings.get(key, {})
             conc = (
@@ -143,7 +145,7 @@ class Downloader:
 
         return key, self.slots[key]
 
-    def _get_slot_key(self, request: Request, spider: Optional[Spider]) -> str:
+    def get_slot_key(self, request: Request) -> str:
         if self.DOWNLOAD_SLOT in request.meta:
             return cast(str, request.meta[self.DOWNLOAD_SLOT])
 
@@ -153,6 +155,14 @@ class Downloader:
 
         return key
 
+    def _get_slot_key(self, request: Request, spider: Optional[Spider]) -> str:
+        warnings.warn(
+            "Use of this protected method is deprecated. Consider using its corresponding public method get_slot_key() instead.",
+            ScrapyDeprecationWarning,
+            stacklevel=2,
+        )
+        return self.get_slot_key(request)
+
     def _enqueue_request(self, request: Request, spider: Spider) -> Deferred:
         key, slot = self._get_slot(request, spider)
         request.meta[self.DOWNLOAD_SLOT] = key
diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py
index 6a82634f1..0e77cd2fe 100644
--- a/scrapy/core/downloader/contextfactory.py
+++ b/scrapy/core/downloader/contextfactory.py
@@ -21,6 +21,7 @@ from scrapy.core.downloader.tls import (
     ScrapyClientTLSOptions,
     openssl_methods,
 )
+from scrapy.crawler import Crawler
 from scrapy.settings import BaseSettings
 from scrapy.utils.misc import build_from_crawler, load_object
 
@@ -102,7 +103,7 @@ class ScrapyClientContextFactory(BrowserLikePolicyForHTTPS):
     # kept for old-style HTTP/1.0 downloader context twisted calls,
     # e.g. connectSSL()
     def getContext(self, hostname: Any = None, port: Any = None) -> SSL.Context:
-        ctx = self.getCertificateOptions().getContext()
+        ctx: SSL.Context = self.getCertificateOptions().getContext()
         ctx.set_options(0x4)  # OP_LEGACY_SERVER_CONNECT
         return ctx
 
@@ -165,7 +166,9 @@ class AcceptableProtocolsContextFactory:
         return options
 
 
-def load_context_factory_from_settings(settings, crawler):
+def load_context_factory_from_settings(
+    settings: BaseSettings, crawler: Crawler
+) -> IPolicyForHTTPS:
     ssl_method = openssl_methods[settings.get("DOWNLOADER_CLIENT_TLS_METHOD")]
     context_factory_cls = load_object(settings["DOWNLOADER_CLIENTCONTEXTFACTORY"])
     # try method-aware context factory
diff --git a/scrapy/core/downloader/handlers/file.py b/scrapy/core/downloader/handlers/file.py
index 4824167da..17dd7483b 100644
--- a/scrapy/core/downloader/handlers/file.py
+++ b/scrapy/core/downloader/handlers/file.py
@@ -2,6 +2,8 @@ from pathlib import Path
 
 from w3lib.url import file_uri_to_path
 
+from scrapy import Request, Spider
+from scrapy.http import Response
 from scrapy.responsetypes import responsetypes
 from scrapy.utils.decorators import defers
 
@@ -10,7 +12,7 @@ class FileDownloadHandler:
     lazy = False
 
     @defers
-    def download_request(self, request, spider):
+    def download_request(self, request: Request, spider: Spider) -> Response:
         filepath = file_uri_to_path(request.url)
         body = Path(filepath).read_bytes()
         respcls = responsetypes.from_args(filename=filepath, body=body)
diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py
index 69add8558..ed94b2221 100644
--- a/scrapy/core/downloader/handlers/ftp.py
+++ b/scrapy/core/downloader/handlers/ftp.py
@@ -32,15 +32,19 @@ from __future__ import annotations
 
 import re
 from io import BytesIO
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any, BinaryIO, Dict, Optional
 from urllib.parse import unquote
 
+from twisted.internet.defer import Deferred
 from twisted.internet.protocol import ClientCreator, Protocol
 from twisted.protocols.ftp import CommandFailed, FTPClient
+from twisted.python.failure import Failure
 
+from scrapy import Request, Spider
 from scrapy.crawler import Crawler
 from scrapy.http import Response
 from scrapy.responsetypes import responsetypes
+from scrapy.settings import BaseSettings
 from scrapy.utils.httpobj import urlparse_cached
 from scrapy.utils.python import to_bytes
 
@@ -50,20 +54,20 @@ if TYPE_CHECKING:
 
 
 class ReceivedDataProtocol(Protocol):
-    def __init__(self, filename=None):
-        self.__filename = filename
-        self.body = open(filename, "wb") if filename else BytesIO()
-        self.size = 0
+    def __init__(self, filename: Optional[str] = None):
+        self.__filename: Optional[str] = filename
+        self.body: BinaryIO = open(filename, "wb") if filename else BytesIO()
+        self.size: int = 0
 
-    def dataReceived(self, data):
+    def dataReceived(self, data: bytes) -> None:
         self.body.write(data)
         self.size += len(data)
 
     @property
-    def filename(self):
+    def filename(self) -> Optional[str]:
         return self.__filename
 
-    def close(self):
+    def close(self) -> None:
         self.body.close() if self.filename else self.body.seek(0)
 
 
@@ -73,12 +77,12 @@ _CODE_RE = re.compile(r"\d+")
 class FTPDownloadHandler:
     lazy = False
 
-    CODE_MAPPING = {
+    CODE_MAPPING: Dict[str, int] = {
         "550": 404,
         "default": 503,
     }
 
-    def __init__(self, settings):
+    def __init__(self, settings: BaseSettings):
         self.default_user = settings["FTP_USER"]
         self.default_password = settings["FTP_PASSWORD"]
         self.passive_mode = settings["FTP_PASSIVE_MODE"]
@@ -87,7 +91,7 @@ class FTPDownloadHandler:
     def from_crawler(cls, crawler: Crawler) -> Self:
         return cls(crawler.settings)
 
-    def download_request(self, request, spider):
+    def download_request(self, request: Request, spider: Spider) -> Deferred:
         from twisted.internet import reactor
 
         parsed_url = urlparse_cached(request)
@@ -99,10 +103,10 @@ class FTPDownloadHandler:
         creator = ClientCreator(
             reactor, FTPClient, user, password, passive=passive_mode
         )
-        dfd = creator.connectTCP(parsed_url.hostname, parsed_url.port or 21)
+        dfd: Deferred = creator.connectTCP(parsed_url.hostname, parsed_url.port or 21)
         return dfd.addCallback(self.gotClient, request, unquote(parsed_url.path))
 
-    def gotClient(self, client, request, filepath):
+    def gotClient(self, client: FTPClient, request: Request, filepath: str) -> Deferred:
         self.client = client
         protocol = ReceivedDataProtocol(request.meta.get("ftp_local_filename"))
         return client.retrieveFile(filepath, protocol).addCallbacks(
@@ -112,15 +116,18 @@ class FTPDownloadHandler:
             errbackArgs=(request,),
         )
 
-    def _build_response(self, result, request, protocol):
+    def _build_response(
+        self, result: Any, request: Request, protocol: ReceivedDataProtocol
+    ) -> Response:
         self.result = result
         protocol.close()
         headers = {"local filename": protocol.filename or "", "size": protocol.size}
         body = to_bytes(protocol.filename or protocol.body.read())
         respcls = responsetypes.from_args(url=request.url, body=body)
-        return respcls(url=request.url, status=200, body=body, headers=headers)
+        # hints for Headers-related types may need to be fixed to not use AnyStr
+        return respcls(url=request.url, status=200, body=body, headers=headers)  # type: ignore[arg-type]
 
-    def _failed(self, result, request):
+    def _failed(self, result: Failure, request: Request) -> Response:
         message = result.getErrorMessage()
         if result.type == CommandFailed:
             m = _CODE_RE.search(message)
@@ -130,4 +137,5 @@ class FTPDownloadHandler:
                 return Response(
                     url=request.url, status=httpcode, body=to_bytes(message)
                 )
+        assert result.type
         raise result.type(result.value)
diff --git a/scrapy/core/downloader/handlers/http10.py b/scrapy/core/downloader/handlers/http10.py
index 256dc36a1..da9559525 100644
--- a/scrapy/core/downloader/handlers/http10.py
+++ b/scrapy/core/downloader/handlers/http10.py
@@ -3,8 +3,13 @@
 
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Type
 
+from twisted.internet.defer import Deferred
+
+from scrapy import Request, Spider
+from scrapy.crawler import Crawler
+from scrapy.settings import BaseSettings
 from scrapy.utils.misc import build_from_crawler, load_object
 from scrapy.utils.python import to_unicode
 
@@ -12,29 +17,34 @@ if TYPE_CHECKING:
     # typing.Self requires Python 3.11
     from typing_extensions import Self
 
+    from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
+    from scrapy.core.downloader.webclient import ScrapyHTTPClientFactory
+
 
 class HTTP10DownloadHandler:
     lazy = False
 
-    def __init__(self, settings, crawler=None):
-        self.HTTPClientFactory = load_object(settings["DOWNLOADER_HTTPCLIENTFACTORY"])
-        self.ClientContextFactory = load_object(
+    def __init__(self, settings: BaseSettings, crawler: Crawler):
+        self.HTTPClientFactory: Type[ScrapyHTTPClientFactory] = load_object(
+            settings["DOWNLOADER_HTTPCLIENTFACTORY"]
+        )
+        self.ClientContextFactory: Type[ScrapyClientContextFactory] = load_object(
             settings["DOWNLOADER_CLIENTCONTEXTFACTORY"]
         )
-        self._settings = settings
-        self._crawler = crawler
+        self._settings: BaseSettings = settings
+        self._crawler: Crawler = crawler
 
     @classmethod
-    def from_crawler(cls, crawler) -> Self:
+    def from_crawler(cls, crawler: Crawler) -> Self:
         return cls(crawler.settings, crawler)
 
-    def download_request(self, request, spider):
+    def download_request(self, request: Request, spider: Spider) -> Deferred:
         """Return a deferred for the HTTP download"""
         factory = self.HTTPClientFactory(request)
         self._connect(factory)
         return factory.deferred
 
-    def _connect(self, factory):
+    def _connect(self, factory: ScrapyHTTPClientFactory) -> Deferred:
         from twisted.internet import reactor
 
         host, port = to_unicode(factory.host), factory.port
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
index 15f8abc64..5e84be6ba 100644
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -8,31 +8,33 @@ import re
 from contextlib import suppress
 from io import BytesIO
 from time import time
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast
 from urllib.parse import urldefrag, urlunparse
 
-from twisted.internet import defer, protocol, ssl
+from twisted.internet import ssl
+from twisted.internet.base import ReactorBase
+from twisted.internet.defer import CancelledError, Deferred, succeed
 from twisted.internet.endpoints import TCP4ClientEndpoint
 from twisted.internet.error import TimeoutError
+from twisted.internet.interfaces import IConsumer
+from twisted.internet.protocol import Factory, Protocol, connectionDone
 from twisted.python.failure import Failure
-from twisted.web.client import (
-    URI,
-    Agent,
-    HTTPConnectionPool,
-    ResponseDone,
-    ResponseFailed,
-)
+from twisted.web.client import URI, Agent, HTTPConnectionPool
+from twisted.web.client import Response as TxResponse
+from twisted.web.client import ResponseDone, ResponseFailed
 from twisted.web.http import PotentialDataLoss, _DataLoss
 from twisted.web.http_headers import Headers as TxHeaders
-from twisted.web.iweb import UNKNOWN_LENGTH, IBodyProducer
+from twisted.web.iweb import UNKNOWN_LENGTH, IBodyProducer, IPolicyForHTTPS
 from zope.interface import implementer
 
-from scrapy import signals
+from scrapy import Request, Spider, signals
 from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
 from scrapy.core.downloader.webclient import _parse
+from scrapy.crawler import Crawler
 from scrapy.exceptions import StopDownload
-from scrapy.http import Headers
+from scrapy.http import Headers, Response
 from scrapy.responsetypes import responsetypes
+from scrapy.settings import BaseSettings
 from scrapy.utils.python import to_bytes, to_unicode
 
 if TYPE_CHECKING:
@@ -46,28 +48,30 @@ logger = logging.getLogger(__name__)
 class HTTP11DownloadHandler:
     lazy = False
 
-    def __init__(self, settings, crawler=None):
+    def __init__(self, settings: BaseSettings, crawler: Crawler):
         self._crawler = crawler
 
         from twisted.internet import reactor
 
-        self._pool = HTTPConnectionPool(reactor, persistent=True)
+        self._pool: HTTPConnectionPool = HTTPConnectionPool(reactor, persistent=True)
         self._pool.maxPersistentPerHost = settings.getint(
             "CONCURRENT_REQUESTS_PER_DOMAIN"
         )
         self._pool._factory.noisy = False
 
-        self._contextFactory = load_context_factory_from_settings(settings, crawler)
-        self._default_maxsize = settings.getint("DOWNLOAD_MAXSIZE")
-        self._default_warnsize = settings.getint("DOWNLOAD_WARNSIZE")
-        self._fail_on_dataloss = settings.getbool("DOWNLOAD_FAIL_ON_DATALOSS")
-        self._disconnect_timeout = 1
+        self._contextFactory: IPolicyForHTTPS = load_context_factory_from_settings(
+            settings, crawler
+        )
+        self._default_maxsize: int = settings.getint("DOWNLOAD_MAXSIZE")
+        self._default_warnsize: int = settings.getint("DOWNLOAD_WARNSIZE")
+        self._fail_on_dataloss: bool = settings.getbool("DOWNLOAD_FAIL_ON_DATALOSS")
+        self._disconnect_timeout: int = 1
 
     @classmethod
-    def from_crawler(cls, crawler) -> Self:
+    def from_crawler(cls, crawler: Crawler) -> Self:
         return cls(crawler.settings, crawler)
 
-    def download_request(self, request, spider):
+    def download_request(self, request: Request, spider: Spider) -> Deferred:
         """Return a deferred for the HTTP download"""
         agent = ScrapyAgent(
             contextFactory=self._contextFactory,
@@ -79,10 +83,10 @@ class HTTP11DownloadHandler:
         )
         return agent.download_request(request)
 
-    def close(self):
+    def close(self) -> Deferred:
         from twisted.internet import reactor
 
-        d = self._pool.closeCachedConnections()
+        d: Deferred = self._pool.closeCachedConnections()
         # closeCachedConnections will hang on network or server issues, so
         # we'll manually timeout the deferred.
         #
@@ -93,7 +97,7 @@ class HTTP11DownloadHandler:
         # issue a callback after `_disconnect_timeout` seconds.
         delayed_call = reactor.callLater(self._disconnect_timeout, d.callback, [])
 
-        def cancel_delayed_call(result):
+        def cancel_delayed_call(result: Any) -> Any:
             if delayed_call.active():
                 delayed_call.cancel()
             return result
@@ -123,39 +127,41 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
 
     def __init__(
         self,
-        reactor,
-        host,
-        port,
-        proxyConf,
-        contextFactory,
-        timeout=30,
-        bindAddress=None,
+        reactor: ReactorBase,
+        host: str,
+        port: int,
+        proxyConf: Tuple[str, int, Optional[bytes]],
+        contextFactory: IPolicyForHTTPS,
+        timeout: float = 30,
+        bindAddress: Optional[Tuple[str, int]] = None,
     ):
         proxyHost, proxyPort, self._proxyAuthHeader = proxyConf
         super().__init__(reactor, proxyHost, proxyPort, timeout, bindAddress)
-        self._tunnelReadyDeferred = defer.Deferred()
-        self._tunneledHost = host
-        self._tunneledPort = port
-        self._contextFactory = contextFactory
-        self._connectBuffer = bytearray()
+        self._tunnelReadyDeferred: Deferred = Deferred()
+        self._tunneledHost: str = host
+        self._tunneledPort: int = port
+        self._contextFactory: IPolicyForHTTPS = contextFactory
+        self._connectBuffer: bytearray = bytearray()
 
-    def requestTunnel(self, protocol):
+    def requestTunnel(self, protocol: Protocol) -> Protocol:
         """Asks the proxy to open a tunnel."""
+        assert protocol.transport
         tunnelReq = tunnel_request_data(
             self._tunneledHost, self._tunneledPort, self._proxyAuthHeader
         )
         protocol.transport.write(tunnelReq)
         self._protocolDataReceived = protocol.dataReceived
-        protocol.dataReceived = self.processProxyResponse
+        protocol.dataReceived = self.processProxyResponse  # type: ignore[method-assign]
         self._protocol = protocol
         return protocol
 
-    def processProxyResponse(self, rcvd_bytes):
+    def processProxyResponse(self, data: bytes) -> None:
         """Processes the response from the proxy. If the tunnel is successfully
         created, notifies the client that we are ready to send requests. If not
         raises a TunnelError.
         """
-        self._connectBuffer += rcvd_bytes
+        assert self._protocol.transport
+        self._connectBuffer += data
         # make sure that enough (all) bytes are consumed
         # and that we've got all HTTP headers (ending with a blank line)
         # from the proxy so that we don't send those bytes to the TLS layer
@@ -163,23 +169,24 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
         # see https://github.com/scrapy/scrapy/issues/2491
         if b"\r\n\r\n" not in self._connectBuffer:
             return
-        self._protocol.dataReceived = self._protocolDataReceived
+        self._protocol.dataReceived = self._protocolDataReceived  # type: ignore[method-assign]
         respm = TunnelingTCP4ClientEndpoint._responseMatcher.match(self._connectBuffer)
         if respm and int(respm.group("status")) == 200:
             # set proper Server Name Indication extension
-            sslOptions = self._contextFactory.creatorForNetloc(
+            sslOptions = self._contextFactory.creatorForNetloc(  # type: ignore[call-arg,misc]
                 self._tunneledHost, self._tunneledPort
             )
             self._protocol.transport.startTLS(sslOptions, self._protocolFactory)
             self._tunnelReadyDeferred.callback(self._protocol)
         else:
+            extra: Any
             if respm:
                 extra = {
                     "status": int(respm.group("status")),
                     "reason": respm.group("reason").strip(),
                 }
             else:
-                extra = rcvd_bytes[: self._truncatedLength]
+                extra = data[: self._truncatedLength]
             self._tunnelReadyDeferred.errback(
                 TunnelError(
                     "Could not open CONNECT tunnel with proxy "
@@ -187,11 +194,11 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
                 )
             )
 
-    def connectFailed(self, reason):
+    def connectFailed(self, reason: Failure) -> None:
         """Propagates the errback to the appropriate deferred."""
         self._tunnelReadyDeferred.errback(reason)
 
-    def connect(self, protocolFactory):
+    def connect(self, protocolFactory: Factory) -> Deferred:
         self._protocolFactory = protocolFactory
         connectDeferred = super().connect(protocolFactory)
         connectDeferred.addCallback(self.requestTunnel)
@@ -199,7 +206,9 @@ class TunnelingTCP4ClientEndpoint(TCP4ClientEndpoint):
         return self._tunnelReadyDeferred
 
 
-def tunnel_request_data(host, port, proxy_auth_header=None):
+def tunnel_request_data(
+    host: str, port: int, proxy_auth_header: Optional[bytes] = None
+) -> bytes:
     r"""
     Return binary content of a CONNECT request.
 
@@ -230,18 +239,19 @@ class TunnelingAgent(Agent):
 
     def __init__(
         self,
-        reactor,
-        proxyConf,
-        contextFactory=None,
-        connectTimeout=None,
-        bindAddress=None,
-        pool=None,
+        *,
+        reactor: ReactorBase,
+        proxyConf: Tuple[str, int, Optional[bytes]],
+        contextFactory: IPolicyForHTTPS,
+        connectTimeout: Optional[float] = None,
+        bindAddress: Optional[bytes] = None,
+        pool: Optional[HTTPConnectionPool] = None,
     ):
         super().__init__(reactor, contextFactory, connectTimeout, bindAddress, pool)
-        self._proxyConf = proxyConf
-        self._contextFactory = contextFactory
+        self._proxyConf: Tuple[str, int, Optional[bytes]] = proxyConf
+        self._contextFactory: IPolicyForHTTPS = contextFactory
 
-    def _getEndpoint(self, uri):
+    def _getEndpoint(self, uri: URI) -> TunnelingTCP4ClientEndpoint:
         return TunnelingTCP4ClientEndpoint(
             reactor=self._reactor,
             host=uri.host,
@@ -253,8 +263,15 @@ class TunnelingAgent(Agent):
         )
 
     def _requestWithEndpoint(
-        self, key, endpoint, method, parsedURI, headers, bodyProducer, requestPath
-    ):
+        self,
+        key: Any,
+        endpoint: TCP4ClientEndpoint,
+        method: bytes,
+        parsedURI: bytes,
+        headers: Optional[TxHeaders],
+        bodyProducer: Optional[IBodyProducer],
+        requestPath: bytes,
+    ) -> Deferred:
         # proxy host and port are required for HTTP pool `key`
         # otherwise, same remote host connection request could reuse
         # a cached tunneled connection to a different proxy
@@ -272,7 +289,12 @@ class TunnelingAgent(Agent):
 
 class ScrapyProxyAgent(Agent):
     def __init__(
-        self, reactor, proxyURI, connectTimeout=None, bindAddress=None, pool=None
+        self,
+        reactor: ReactorBase,
+        proxyURI: bytes,
+        connectTimeout: Optional[float] = None,
+        bindAddress: Optional[bytes] = None,
+        pool: Optional[HTTPConnectionPool] = None,
     ):
         super().__init__(
             reactor=reactor,
@@ -280,9 +302,15 @@ class ScrapyProxyAgent(Agent):
             bindAddress=bindAddress,
             pool=pool,
         )
-        self._proxyURI = URI.fromBytes(proxyURI)
+        self._proxyURI: URI = URI.fromBytes(proxyURI)
 
-    def request(self, method, uri, headers=None, bodyProducer=None):
+    def request(
+        self,
+        method: bytes,
+        uri: bytes,
+        headers: Optional[TxHeaders] = None,
+        bodyProducer: Optional[IBodyProducer] = None,
+    ) -> Deferred:
         """
         Issue a new request via the configured proxy.
         """
@@ -306,26 +334,27 @@ class ScrapyAgent:
 
     def __init__(
         self,
-        contextFactory=None,
-        connectTimeout=10,
-        bindAddress=None,
-        pool=None,
-        maxsize=0,
-        warnsize=0,
-        fail_on_dataloss=True,
-        crawler=None,
+        *,
+        contextFactory: IPolicyForHTTPS,
+        connectTimeout: float = 10,
+        bindAddress: Optional[bytes] = None,
+        pool: Optional[HTTPConnectionPool] = None,
+        maxsize: int = 0,
+        warnsize: int = 0,
+        fail_on_dataloss: bool = True,
+        crawler: Crawler,
     ):
-        self._contextFactory = contextFactory
-        self._connectTimeout = connectTimeout
-        self._bindAddress = bindAddress
-        self._pool = pool
-        self._maxsize = maxsize
-        self._warnsize = warnsize
-        self._fail_on_dataloss = fail_on_dataloss
-        self._txresponse = None
-        self._crawler = crawler
-
-    def _get_agent(self, request, timeout):
+        self._contextFactory: IPolicyForHTTPS = contextFactory
+        self._connectTimeout: float = connectTimeout
+        self._bindAddress: Optional[bytes] = bindAddress
+        self._pool: Optional[HTTPConnectionPool] = pool
+        self._maxsize: int = maxsize
+        self._warnsize: int = warnsize
+        self._fail_on_dataloss: bool = fail_on_dataloss
+        self._txresponse: Optional[TxResponse] = None
+        self._crawler: Crawler = crawler
+
+    def _get_agent(self, request: Request, timeout: float) -> Agent:
         from twisted.internet import reactor
 
         bindaddress = request.meta.get("bindaddress") or self._bindAddress
@@ -333,10 +362,10 @@ class ScrapyAgent:
         if proxy:
             proxyScheme, proxyNetloc, proxyHost, proxyPort, proxyParams = _parse(proxy)
             scheme = _parse(request.url)[0]
-            proxyHost = to_unicode(proxyHost)
+            proxyHost_str = to_unicode(proxyHost)
             if scheme == b"https":
                 proxyAuth = request.headers.get(b"Proxy-Authorization", None)
-                proxyConf = (proxyHost, proxyPort, proxyAuth)
+                proxyConf = (proxyHost_str, proxyPort, proxyAuth)
                 return self._TunnelingAgent(
                     reactor=reactor,
                     proxyConf=proxyConf,
@@ -346,7 +375,9 @@ class ScrapyAgent:
                     pool=self._pool,
                 )
             proxyScheme = proxyScheme or b"http"
-            proxyURI = urlunparse((proxyScheme, proxyNetloc, proxyParams, "", "", ""))
+            proxyURI = urlunparse(
+                (proxyScheme, proxyNetloc, proxyParams, b"", b"", b"")
+            )
             return self._ProxyAgent(
                 reactor=reactor,
                 proxyURI=to_bytes(proxyURI, encoding="ascii"),
@@ -363,7 +394,7 @@ class ScrapyAgent:
             pool=self._pool,
         )
 
-    def download_request(self, request):
+    def download_request(self, request: Request) -> Deferred:
         from twisted.internet import reactor
 
         timeout = request.meta.get("download_timeout") or self._connectTimeout
@@ -380,7 +411,7 @@ class ScrapyAgent:
         else:
             bodyproducer = None
         start_time = time()
-        d = agent.request(
+        d: Deferred = agent.request(
             method, to_bytes(url, encoding="ascii"), headers, bodyproducer
         )
         # set download latency
@@ -393,7 +424,9 @@ class ScrapyAgent:
         d.addBoth(self._cb_timeout, request, url, timeout)
         return d
 
-    def _cb_timeout(self, result, request, url, timeout):
+    def _cb_timeout(
+        self, result: Any, request: Request, url: str, timeout: float
+    ) -> Any:
         if self._timeout_cl.active():
             self._timeout_cl.cancel()
             return result
@@ -404,19 +437,21 @@ class ScrapyAgent:
 
         raise TimeoutError(f"Getting {url} took longer than {timeout} seconds.")
 
-    def _cb_latency(self, result, request, start_time):
+    def _cb_latency(self, result: Any, request: Request, start_time: float) -> Any:
         request.meta["download_latency"] = time() - start_time
         return result
 
     @staticmethod
-    def _headers_from_twisted_response(response):
+    def _headers_from_twisted_response(response: TxResponse) -> Headers:
         headers = Headers()
         if response.length != UNKNOWN_LENGTH:
             headers[b"Content-Length"] = str(response.length).encode()
         headers.update(response.headers.getAllRawHeaders())
         return headers
 
-    def _cb_bodyready(self, txresponse, request):
+    def _cb_bodyready(
+        self, txresponse: TxResponse, request: Request
+    ) -> Union[Dict[str, Any], Deferred]:
         headers_received_result = self._crawler.signals.send_catch_log(
             signal=signals.headers_received,
             headers=self._headers_from_twisted_response(txresponse),
@@ -472,7 +507,7 @@ class ScrapyAgent:
             logger.warning(warning_msg, warning_args)
 
             txresponse._transport.loseConnection()
-            raise defer.CancelledError(warning_msg % warning_args)
+            raise CancelledError(warning_msg % warning_args)
 
         if warnsize and expected_size > warnsize:
             logger.warning(
@@ -481,11 +516,11 @@ class ScrapyAgent:
                 {"size": expected_size, "warnsize": warnsize, "request": request},
             )
 
-        def _cancel(_):
+        def _cancel(_: Any) -> None:
             # Abort connection immediately.
             txresponse._transport._producer.abortConnection()
 
-        d = defer.Deferred(_cancel)
+        d: Deferred = Deferred(_cancel)
         txresponse.deliverBody(
             _ResponseReader(
                 finished=d,
@@ -503,7 +538,9 @@ class ScrapyAgent:
 
         return d
 
-    def _cb_bodydone(self, result, request, url):
+    def _cb_bodydone(
+        self, result: Dict[str, Any], request: Request, url: str
+    ) -> Union[Response, Failure]:
         headers = self._headers_from_twisted_response(result["txresponse"])
         respcls = responsetypes.from_args(headers=headers, url=url, body=result["body"])
         try:
@@ -523,53 +560,57 @@ class ScrapyAgent:
         )
         if result.get("failure"):
             result["failure"].value.response = response
-            return result["failure"]
+            return cast(Failure, result["failure"])
         return response
 
 
 @implementer(IBodyProducer)
 class _RequestBodyProducer:
-    def __init__(self, body):
+    def __init__(self, body: bytes):
         self.body = body
         self.length = len(body)
 
-    def startProducing(self, consumer):
+    def startProducing(self, consumer: IConsumer) -> Deferred:
         consumer.write(self.body)
-        return defer.succeed(None)
+        return succeed(None)
 
-    def pauseProducing(self):
+    def pauseProducing(self) -> None:
         pass
 
-    def stopProducing(self):
+    def stopProducing(self) -> None:
         pass
 
 
-class _ResponseReader(protocol.Protocol):
+class _ResponseReader(Protocol):
     def __init__(
         self,
-        finished,
-        txresponse,
-        request,
-        maxsize,
-        warnsize,
-        fail_on_dataloss,
-        crawler,
+        finished: Deferred,
+        txresponse: TxResponse,
+        request: Request,
+        maxsize: int,
+        warnsize: int,
+        fail_on_dataloss: bool,
+        crawler: Crawler,
     ):
-        self._finished = finished
-        self._txresponse = txresponse
-        self._request = request
-        self._bodybuf = BytesIO()
-        self._maxsize = maxsize
-        self._warnsize = warnsize
-        self._fail_on_dataloss = fail_on_dataloss
-        self._fail_on_dataloss_warned = False
-        self._reached_warnsize = False
-        self._bytes_received = 0
-        self._certificate = None
-        self._ip_address = None
-        self._crawler = crawler
+        self._finished: Deferred = finished
+        self._txresponse: TxResponse = txresponse
+        self._request: Request = request
+        self._bodybuf: BytesIO = BytesIO()
+        self._maxsize: int = maxsize
+        self._warnsize: int = warnsize
+        self._fail_on_dataloss: bool = fail_on_dataloss
+        self._fail_on_dataloss_warned: bool = False
+        self._reached_warnsize: bool = False
+        self._bytes_received: int = 0
+        self._certificate: Optional[ssl.Certificate] = None
+        self._ip_address: Union[ipaddress.IPv4Address, ipaddress.IPv6Address, None] = (
+            None
+        )
+        self._crawler: Crawler = crawler
 
-    def _finish_response(self, flags=None, failure=None):
+    def _finish_response(
+        self, flags: Optional[List[str]] = None, failure: Optional[Failure] = None
+    ) -> None:
         self._finished.callback(
             {
                 "txresponse": self._txresponse,
@@ -581,7 +622,8 @@ class _ResponseReader(protocol.Protocol):
             }
         )
 
-    def connectionMade(self):
+    def connectionMade(self) -> None:
+        assert self.transport
         if self._certificate is None:
             with suppress(AttributeError):
                 self._certificate = ssl.Certificate(
@@ -593,11 +635,12 @@ class _ResponseReader(protocol.Protocol):
                 self.transport._producer.getPeer().host
             )
 
-    def dataReceived(self, bodyBytes):
+    def dataReceived(self, bodyBytes: bytes) -> None:
         # This maybe called several times after cancel was called with buffered data.
         if self._finished.called:
             return
 
+        assert self.transport
         self._bodybuf.write(bodyBytes)
         self._bytes_received += len(bodyBytes)
 
@@ -644,7 +687,7 @@ class _ResponseReader(protocol.Protocol):
                 {"warnsize": self._warnsize, "request": self._request},
             )
 
-    def connectionLost(self, reason):
+    def connectionLost(self, reason: Failure = connectionDone) -> None:
         if self._finished.called:
             return
 
diff --git a/scrapy/core/downloader/handlers/http2.py b/scrapy/core/downloader/handlers/http2.py
index e9a6b6fa3..16fc1e3ae 100644
--- a/scrapy/core/downloader/handlers/http2.py
+++ b/scrapy/core/downloader/handlers/http2.py
@@ -8,6 +8,7 @@ from twisted.internet.base import DelayedCall
 from twisted.internet.defer import Deferred
 from twisted.internet.error import TimeoutError
 from twisted.web.client import URI
+from twisted.web.iweb import IPolicyForHTTPS
 
 from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
 from scrapy.core.downloader.webclient import _parse
@@ -24,7 +25,7 @@ if TYPE_CHECKING:
 
 
 class H2DownloadHandler:
-    def __init__(self, settings: Settings, crawler: Optional[Crawler] = None):
+    def __init__(self, settings: Settings, crawler: Crawler):
         self._crawler = crawler
 
         from twisted.internet import reactor
@@ -54,7 +55,7 @@ class ScrapyH2Agent:
 
     def __init__(
         self,
-        context_factory,
+        context_factory: IPolicyForHTTPS,
         pool: H2ConnectionPool,
         connect_timeout: int = 10,
         bind_address: Optional[bytes] = None,
diff --git a/scrapy/core/downloader/handlers/s3.py b/scrapy/core/downloader/handlers/s3.py
index 99fbb49ce..9a0811a50 100644
--- a/scrapy/core/downloader/handlers/s3.py
+++ b/scrapy/core/downloader/handlers/s3.py
@@ -1,9 +1,14 @@
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any, Optional, Type
 
+from twisted.internet.defer import Deferred
+
+from scrapy import Request, Spider
 from scrapy.core.downloader.handlers.http import HTTPDownloadHandler
+from scrapy.crawler import Crawler
 from scrapy.exceptions import NotConfigured
+from scrapy.settings import BaseSettings
 from scrapy.utils.boto import is_botocore_available
 from scrapy.utils.httpobj import urlparse_cached
 from scrapy.utils.misc import build_from_crawler
@@ -16,14 +21,14 @@ if TYPE_CHECKING:
 class S3DownloadHandler:
     def __init__(
         self,
-        settings,
+        settings: BaseSettings,
         *,
-        crawler=None,
-        aws_access_key_id=None,
-        aws_secret_access_key=None,
-        aws_session_token=None,
-        httpdownloadhandler=HTTPDownloadHandler,
-        **kw,
+        crawler: Crawler,
+        aws_access_key_id: Optional[str] = None,
+        aws_secret_access_key: Optional[str] = None,
+        aws_session_token: Optional[str] = None,
+        httpdownloadhandler: Type[HTTPDownloadHandler] = HTTPDownloadHandler,
+        **kw: Any,
     ):
         if not is_botocore_available():
             raise NotConfigured("missing botocore library")
@@ -51,6 +56,8 @@ class S3DownloadHandler:
         if kw:
             raise TypeError(f"Unexpected keyword arguments: {kw}")
         if not self.anon:
+            assert aws_access_key_id is not None
+            assert aws_secret_access_key is not None
             SignerCls = botocore.auth.AUTH_TYPE_MAPS["s3"]
             self._signer = SignerCls(
                 botocore.credentials.Credentials(
@@ -65,10 +72,10 @@ class S3DownloadHandler:
         self._download_http = _http_handler.download_request
 
     @classmethod
-    def from_crawler(cls, crawler, **kwargs) -> Self:
+    def from_crawler(cls, crawler: Crawler, **kwargs: Any) -> Self:
         return cls(crawler.settings, crawler=crawler, **kwargs)
 
-    def download_request(self, request, spider):
+    def download_request(self, request: Request, spider: Spider) -> Deferred:
         p = urlparse_cached(request)
         scheme = "https" if request.meta.get("is_secure") else "http"
         bucket = p.hostname
@@ -85,6 +92,7 @@ class S3DownloadHandler:
                 headers=request.headers.to_unicode_dict(),
                 data=request.body,
             )
+            assert self._signer
             self._signer.add_auth(awsrequest)
             request = request.replace(url=url, headers=awsrequest.headers.items())
         return self._download_http(request, spider)
diff --git a/scrapy/pqueues.py b/scrapy/pqueues.py
index 773825c5e..58a47ef0f 100644
--- a/scrapy/pqueues.py
+++ b/scrapy/pqueues.py
@@ -180,7 +180,7 @@ class DownloaderInterface:
         return [(self._active_downloads(slot), slot) for slot in possible_slots]
 
     def get_slot_key(self, request: Request) -> str:
-        return self.downloader._get_slot_key(request, None)
+        return self.downloader.get_slot_key(request)
 
     def _active_downloads(self, slot: str) -> int:
         """Return a number of requests in a Downloader for a given slot"""
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 2b3d95a0e..d7ac7ec35 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -239,7 +239,7 @@ MEMUSAGE_NOTIFY_MAIL = []
 MEMUSAGE_WARNING_MB = 0
 
 METAREFRESH_ENABLED = True
-METAREFRESH_IGNORE_TAGS = []
+METAREFRESH_IGNORE_TAGS = ["noscript"]
 METAREFRESH_MAXDELAY = 100
 
 NEWSPIDER_MODULE = ""
diff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.py
index 72c2aaba7..2416d2a4d 100644
--- a/scrapy/spiders/__init__.py
+++ b/scrapy/spiders/__init__.py
@@ -17,12 +17,17 @@ from scrapy.utils.trackref import object_ref
 from scrapy.utils.url import url_is_from_spider
 
 if TYPE_CHECKING:
+    from collections.abc import Callable
+
+    # typing.Concatenate requires Python 3.10
     # typing.Self requires Python 3.11
-    from typing_extensions import Self
+    from typing_extensions import Concatenate, Self
 
     from scrapy.crawler import Crawler
     from scrapy.settings import BaseSettings
 
+    CallbackT = Callable[Concatenate[Response, ...], Any]
+
 
 class Spider(object_ref):
     """Base class for scrapy spiders. All spiders must inherit from this
@@ -79,10 +84,14 @@ class Spider(object_ref):
     def _parse(self, response: Response, **kwargs: Any) -> Any:
         return self.parse(response, **kwargs)
 
-    def parse(self, response: Response, **kwargs: Any) -> Any:
-        raise NotImplementedError(
-            f"{self.__class__.__name__}.parse callback is not defined"
-        )
+    if TYPE_CHECKING:
+        parse: CallbackT
+    else:
+
+        def parse(self, response: Response, **kwargs: Any) -> Any:
+            raise NotImplementedError(
+                f"{self.__class__.__name__}.parse callback is not defined"
+            )
 
     @classmethod
     def update_settings(cls, settings: BaseSettings) -> None:
diff --git a/tox.ini b/tox.ini
index d7527bb04..ede139756 100644
--- a/tox.ini
+++ b/tox.ini
@@ -56,6 +56,15 @@ deps =
 commands =
     mypy {posargs: scrapy tests}
 
+[testenv:typing-tests]
+basepython = python3.8
+deps =
+    {[test-requirements]deps}
+    {[testenv:typing]deps}
+    pytest-mypy-testing==0.1.3
+commands =
+    pytest {posargs: tests_typing}
+
 [testenv:pre-commit]
 basepython = python3
 deps =


Test patch:
diff --git a/tests/test_downloadermiddleware_redirect.py b/tests/test_downloadermiddleware_redirect.py
index 10b8ca9af..83ff25982 100644
--- a/tests/test_downloadermiddleware_redirect.py
+++ b/tests/test_downloadermiddleware_redirect.py
@@ -395,9 +395,8 @@ class MetaRefreshMiddlewareTest(unittest.TestCase):
             """content="0;URL='http://example.org/newpage'"></noscript>"""
         )
         rsp = HtmlResponse(req.url, body=body.encode())
-        req2 = self.mw.process_response(req, rsp, self.spider)
-        assert isinstance(req2, Request)
-        self.assertEqual(req2.url, "http://example.org/newpage")
+        response = self.mw.process_response(req, rsp, self.spider)
+        assert isinstance(response, Response)
 
     def test_ignore_tags_1_x_list(self):
         """Test that Scrapy 1.x behavior remains possible"""
diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py
index 37099dae6..02b50baa3 100644
--- a/tests/test_scheduler.py
+++ b/tests/test_scheduler.py
@@ -25,7 +25,7 @@ class MockDownloader:
     def __init__(self):
         self.slots = {}
 
-    def _get_slot_key(self, request, spider):
+    def get_slot_key(self, request):
         if Downloader.DOWNLOAD_SLOT in request.meta:
             return request.meta[Downloader.DOWNLOAD_SLOT]
 
@@ -273,14 +273,14 @@ class DownloaderAwareSchedulerTestMixin:
         while self.scheduler.has_pending_requests():
             request = self.scheduler.next_request()
             # pylint: disable=protected-access
-            slot = downloader._get_slot_key(request, None)
+            slot = downloader.get_slot_key(request)
             dequeued_slots.append(slot)
             downloader.increment(slot)
             requests.append(request)
 
         for request in requests:
             # pylint: disable=protected-access
-            slot = downloader._get_slot_key(request, None)
+            slot = downloader.get_slot_key(request)
             downloader.decrement(slot)
 
         self.assertTrue(
diff --git a/tests_typing/test_spiders.mypy-testing b/tests_typing/test_spiders.mypy-testing
new file mode 100644
index 000000000..162e31d0c
--- /dev/null
+++ b/tests_typing/test_spiders.mypy-testing
@@ -0,0 +1,68 @@
+from typing import Any
+
+import pytest
+
+from scrapy.http import HtmlResponse, Response
+from scrapy.spiders import Spider
+
+
+class SimpleSpider(Spider):
+    pass
+
+
+class SameOverrideSpider(Spider):
+    def parse(self, response: Response, **kwargs: Any) -> Any:
+        pass
+
+
+class NoKwargsSpider(Spider):
+    def parse(self, response: Response) -> Any:
+        pass
+
+
+class SpecificKwargsSpider(Spider):
+    def parse(self, response: Response, page: int) -> Any:
+        pass
+
+
+class NarrowOverrideSpider(Spider):
+    # without type: ignore this produces several note lines in addition to an error line,
+    # which is unsupported by pytest-mypy-testing
+    def parse(self, response: HtmlResponse, **kwargs: Any) -> Any:  # type: ignore[override]
+        pass
+
+
+@pytest.mark.mypy_testing
+def test_spider_parse() -> None:
+    spider = Spider()
+    reveal_type(spider.parse)  # R: def (scrapy.http.response.Response, *Any, **Any) -> Any
+
+
+@pytest.mark.mypy_testing
+def test_spider_parse_override_simple() -> None:
+    spider = SimpleSpider()
+    reveal_type(spider.parse)  # R: def (scrapy.http.response.Response, *Any, **Any) -> Any
+
+
+@pytest.mark.mypy_testing
+def test_spider_parse_override_same() -> None:
+    spider = SameOverrideSpider()
+    reveal_type(spider.parse)  # R: def (response: scrapy.http.response.Response, **kwargs: Any) -> Any
+
+
+@pytest.mark.mypy_testing
+def test_spider_parse_override_no_kwargs() -> None:
+    spider = NoKwargsSpider()
+    reveal_type(spider.parse)  # R: def (response: scrapy.http.response.Response) -> Any
+
+
+@pytest.mark.mypy_testing
+def test_spider_parse_override_specific_kwargs() -> None:
+    spider = SpecificKwargsSpider()
+    reveal_type(spider.parse)  # R: def (response: scrapy.http.response.Response, page: builtins.int) -> Any
+
+
+@pytest.mark.mypy_testing
+def test_spider_parse_override_narrow() -> None:
+    spider = NarrowOverrideSpider()
+    reveal_type(spider.parse)  # R: def (response: scrapy.http.response.html.HtmlResponse, **kwargs: Any) -> Any