Repository: scrapy/scrapy
Pull request #6568 (2024-12-02 18:20:42)

Fix Default SpiderLoader doesn't take into account addons
Fixes https://github.com/scrapy/scrapy/issues/6544

I had to do it at the crawler level because the settings passed to the spiderloader are frozen.
This achieves the goal of making new spider modules visible, but creates some not elegant behavior because each addon `update_settings` is ran twice. Which I still have to account for in the tests.
I couldn't find a way to make the spider visible without a dummy crawler approach, because ultimately the crawlers need a spider class to be created and we need spider loaders to know which classes are available.
I wonder if this couldn't cause issues if, for example, an Addon counts on the `from_crawler` method to properly run `update_settings`; @VMRuiz what are your thoughts?

kmike: For example, if the crawler is not available yet (and so the addon can't be created yet), the pre-update method could be a classmethod, not a regular method.
GeorgeA92: @guillermo-bondonno 
As previously mentioned on https://github.com/scrapy/scrapy/issues/6544#issuecomment-2512542207
attempt to change settings related to spiderloader is out of scope of addons (on it's current implementation as.. per crawler while spiderloader can be only one per process).
As it is possible to implement this functionality with custom spiderloader class set by `SPIDER_LOADER_CLASS` it is not clear why thi is needed as update for scrapy default spiderloader class.
kmike: @GeorgeA92 we do have a use case - https://github.com/zytedata/zyte-spider-templates provides an addon which should add a few spiders to the project.

I don't think the current implementation is deliberately trying to avoid importing all the modules.
guillermo-bondonno: > Would it make sense to have 2 methods in the addons (update_settings, and e.g. update_settings_early), and call them at different stages of crawler initialization? If we can avoid doing things twice, we should try, as it could lead to all kinds of issues.

@kmike I agree, what about requiring changes in SPIDER_MODULES to go into a specific classmethod of addons (update_spider_modules or something less specific if there's other use case), then running them from a classmethod in the AddonManager that doesn't have access/need a crawler instance?


Patch:
diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml
index ff279e9fd..a064bf5b2 100644
--- a/.github/workflows/checks.yml
+++ b/.github/workflows/checks.yml
@@ -21,7 +21,7 @@ jobs:
         - python-version: "3.9"
           env:
             TOXENV: typing-tests
-        - python-version: "3.12"  # Keep in sync with .readthedocs.yml
+        - python-version: "3.13"  # Keep in sync with .readthedocs.yml
           env:
             TOXENV: docs
         - python-version: "3.13"
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 39b9a33aa..c76c613d9 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1,20 +1,16 @@
 repos:
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.8.1
+  rev: v0.8.4
   hooks:
     - id: ruff
       args: [ --fix ]
 - repo: https://github.com/psf/black.git
-  rev: 24.4.2
+  rev: 24.10.0
   hooks:
   - id: black
-- repo: https://github.com/pycqa/isort
-  rev: 5.13.2
-  hooks:
-  - id: isort
 - repo: https://github.com/adamchainz/blacken-docs
-  rev: 1.18.0
+  rev: 1.19.1
   hooks:
   - id: blacken-docs
     additional_dependencies:
-    - black==24.4.2
+    - black==24.10.0
diff --git a/.readthedocs.yml b/.readthedocs.yml
index 0c544df7e..23e4cabea 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -5,11 +5,11 @@ sphinx:
   fail_on_warning: true
 
 build:
-  os: ubuntu-20.04
+  os: ubuntu-24.04
   tools:
     # For available versions, see:
     # https://docs.readthedocs.io/en/stable/config-file/v2.html#build-tools-python
-    python: "3.12"  # Keep in sync with .github/workflows/checks.yml
+    python: "3.13"  # Keep in sync with .github/workflows/checks.yml
 
 python:
   install:
diff --git a/docs/Makefile b/docs/Makefile
index 48401bac8..ed8809902 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -1,96 +1,20 @@
+# Minimal makefile for Sphinx documentation
 #
-# Makefile for Scrapy documentation [based on Python documentation Makefile]
-# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-#
-
-# You can set these variables from the command line.
-PYTHON       = python
-SPHINXOPTS   =
-PAPER        =
-SOURCES      =
-SHELL        = /usr/bin/env bash
-
-ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees \
-                -D latex_elements.papersize=$(PAPER) \
-                $(SPHINXOPTS) . build/$(BUILDER) $(SOURCES)
 
-.PHONY: help update build html htmlhelp clean
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = build
 
+# Put it first so that "make" without argument is like "make help".
 help:
-	@echo "Please use \`make <target>' where <target> is one of"
-	@echo "  html      to make standalone HTML files"
-	@echo "  htmlhelp  to make HTML files and a HTML help project"
-	@echo "  latex     to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
-	@echo "  text      to make plain text files"
-	@echo "  changes   to make an overview over all changed/added/deprecated items"
-	@echo "  linkcheck to check all external links for integrity"
-	@echo "  watch     build HTML docs, open in browser and watch for changes"
-
-build-dirs:
-	mkdir -p build/$(BUILDER) build/doctrees
-
-build: build-dirs
-	sphinx-build $(ALLSPHINXOPTS)
-	@echo
-
-build-ignore-errors: build-dirs
-	-sphinx-build $(ALLSPHINXOPTS)
-	@echo
-
-
-html: BUILDER = html
-html: build
-	@echo "Build finished. The HTML pages are in build/html."
-
-htmlhelp: BUILDER = htmlhelp
-htmlhelp: build
-	@echo "Build finished; now you can run HTML Help Workshop with the" \
-	      "build/htmlhelp/pydoc.hhp project file."
-
-latex: BUILDER = latex
-latex: build
-	@echo "Build finished; the LaTeX files are in build/latex."
-	@echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \
-	      "run these through (pdf)latex."
-
-text: BUILDER = text
-text: build
-	@echo "Build finished; the text files are in build/text."
-
-changes: BUILDER = changes
-changes: build
-	@echo "The overview file is in build/changes."
-
-linkcheck: BUILDER = linkcheck
-linkcheck: build
-	@echo "Link check complete; look for any errors in the above output " \
-	      "or in build/$(BUILDER)/output.txt"
-
-linkfix: BUILDER = linkcheck
-linkfix: build-ignore-errors
-	$(PYTHON) utils/linkfix.py
-	@echo "Fixing redirecting links in docs has finished; check all " \
-	      "replacements before committing them"
-
-doctest: BUILDER = doctest
-doctest: build
-	@echo "Testing of doctests in the sources finished, look at the " \
-	      "results in build/doctest/output.txt"
-
-pydoc-topics: BUILDER = pydoc-topics
-pydoc-topics: build
-	@echo "Building finished; now copy build/pydoc-topics/pydoc_topics.py " \
-	      "into the Lib/ directory"
-
-coverage: BUILDER = coverage
-coverage: build
-
-htmlview: html
-	 $(PYTHON) -c "import webbrowser; from pathlib import Path; \
-	 webbrowser.open(Path('build/html/index.html').resolve().as_uri())"
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-clean:
-	-rm -rf build/*
+.PHONY: help Makefile
 
-watch: htmlview
-	watchmedo shell-command -p '*.rst' -c 'make html' -R -D
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/_ext/scrapydocs.py b/docs/_ext/scrapydocs.py
index c23a89089..9b63f39f6 100644
--- a/docs/_ext/scrapydocs.py
+++ b/docs/_ext/scrapydocs.py
@@ -1,3 +1,4 @@
+# pylint: disable=import-error
 from operator import itemgetter
 
 from docutils import nodes
diff --git a/docs/_ext/scrapyfixautodoc.py b/docs/_ext/scrapyfixautodoc.py
new file mode 100644
index 000000000..d7a3fb514
--- /dev/null
+++ b/docs/_ext/scrapyfixautodoc.py
@@ -0,0 +1,18 @@
+"""
+Must be included after 'sphinx.ext.autodoc'. Fixes unwanted 'alias of' behavior.
+https://github.com/sphinx-doc/sphinx/issues/4422
+"""
+
+# pylint: disable=import-error
+from sphinx.application import Sphinx
+
+
+def maybe_skip_member(app: Sphinx, what, name: str, obj, skip: bool, options) -> bool:
+    if not skip:
+        # autodocs was generating a text "alias of" for the following members
+        return name in {"default_item_class", "default_selector_class"}
+    return skip
+
+
+def setup(app: Sphinx) -> None:
+    app.connect("autodoc-skip-member", maybe_skip_member)
diff --git a/docs/conf.py b/docs/conf.py
index 7a5166053..be5e07195 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -1,15 +1,12 @@
-# Scrapy documentation build configuration file, created by
-# sphinx-quickstart on Mon Nov 24 12:02:52 2008.
+# Configuration file for the Sphinx documentation builder.
 #
-# This file is execfile()d with the current directory set to its containing dir.
-#
-# The contents of this file are pickled, so don't put values in the namespace
-# that aren't pickleable (module imports are okay, they're removed automatically).
-#
-# All configuration values have a default; values that are commented out
-# serve to show the default.
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# pylint: disable=import-error
 import os
 import sys
+from collections.abc import Sequence
 from pathlib import Path
 
 # If your extensions are in another directory, add it here. If the directory
@@ -18,36 +15,30 @@ sys.path.append(str(Path(__file__).parent / "_ext"))
 sys.path.insert(0, str(Path(__file__).parent.parent))
 
 
-# General configuration
-# ---------------------
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "Scrapy"
+project_copyright = "Scrapy developers"
+author = "Scrapy developers"
+
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
-# Add any Sphinx extension module names here, as strings. They can be extensions
-# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
     "hoverxref.extension",
     "notfound.extension",
     "scrapydocs",
     "sphinx.ext.autodoc",
+    "scrapyfixautodoc",  # Must be after "sphinx.ext.autodoc"
     "sphinx.ext.coverage",
     "sphinx.ext.intersphinx",
     "sphinx.ext.viewcode",
 ]
 
-# Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
-
-# The suffix of source filenames.
-source_suffix = ".rst"
-
-# The encoding of source files.
-# source_encoding = 'utf-8'
-
-# The master toctree document.
-master_doc = "index"
-
-# General information about the project.
-project = "Scrapy"
-copyright = "Scrapy developers"
+exclude_patterns = ["build", "Thumbs.db", ".DS_Store"]
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -63,125 +54,17 @@ except ImportError:
     version = ""
     release = ""
 
-# The language for content autogenerated by Sphinx. Refer to documentation
-# for a list of supported languages.
-language = "en"
-
-# There are two options for replacing |today|: either, you set today to some
-# non-false value, then it is used:
-# today = ''
-# Else, today_fmt is used as the format for a strftime call.
-# today_fmt = '%B %d, %Y'
-
-# List of documents that shouldn't be included in the build.
-# unused_docs = []
-
-exclude_patterns = ["build"]
-
-# List of directories, relative to source directory, that shouldn't be searched
-# for source files.
-exclude_trees = [".build"]
-
-# The reST default role (used for this markup: `text`) to use for all documents.
-# default_role = None
-
-# If true, '()' will be appended to :func: etc. cross-reference text.
-# add_function_parentheses = True
-
-# If true, the current module name will be prepended to all description
-# unit titles (such as .. function::).
-# add_module_names = True
-
-# If true, sectionauthor and moduleauthor directives will be shown in the
-# output. They are ignored by default.
-# show_authors = False
-
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = "sphinx"
-
-# List of Sphinx warnings that will not be raised
 suppress_warnings = ["epub.unknown_project_files"]
 
 
-# Options for HTML output
-# -----------------------
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
 html_theme = "sphinx_rtd_theme"
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-# html_theme_options = {}
-
-# Add any paths that contain custom themes here, relative to this directory.
-# Add path to the RTD explicitly to robustify builds (otherwise might
-# fail in a clean Debian build env)
-import sphinx_rtd_theme
-
-html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
-
-# The style sheet to use for HTML and HTML Help pages. A file of that name
-# must exist either in Sphinx' static/ path, or in one of the custom paths
-# given in html_static_path.
-# html_style = 'scrapydoc.css'
-
-# The name for this set of Sphinx documents.  If None, it defaults to
-# "<project> v<release> documentation".
-# html_title = None
-
-# A shorter title for the navigation bar.  Default is the same as html_title.
-# html_short_title = None
-
-# The name of an image file (relative to this directory) to place at the top
-# of the sidebar.
-# html_logo = None
-
-# The name of an image file (within the static path) to use as favicon of the
-# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
-# pixels large.
-# html_favicon = None
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
 
-# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
-# using the given strftime format.
 html_last_updated_fmt = "%b %d, %Y"
 
-# Custom sidebar templates, maps document names to template names.
-# html_sidebars = {}
-
-# Additional templates that should be rendered to pages, maps page names to
-# template names.
-# html_additional_pages = {}
-
-# If false, no module index is generated.
-# html_use_modindex = True
-
-# If false, no index is generated.
-# html_use_index = True
-
-# If true, the index is split into individual pages for each letter.
-# html_split_index = False
-
-# If true, the reST sources are included in the HTML build as _sources/<name>.
-html_copy_source = True
-
-# If true, an OpenSearch description file will be output, and all pages will
-# contain a <link> tag referring to it.  The value of this option must be the
-# base URL from which the finished HTML is served.
-# html_use_opensearch = ''
-
-# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml").
-# html_file_suffix = ''
-
-# Output file base name for HTML help builder.
-htmlhelp_basename = "Scrapydoc"
-
 html_css_files = [
     "custom.css",
 ]
@@ -189,14 +72,8 @@ html_css_files = [
 # Set canonical URL from the Read the Docs Domain
 html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "")
 
-# Options for LaTeX output
-# ------------------------
-
-# The paper size ('letter' or 'a4').
-# latex_paper_size = 'letter'
-
-# The font size ('10pt', '11pt' or '12pt').
-# latex_font_size = '10pt'
+# -- Options for LaTeX output ------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-latex-output
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title, author, document class [howto/manual]).
@@ -204,39 +81,22 @@ latex_documents = [
     ("index", "Scrapy.tex", "Scrapy Documentation", "Scrapy developers", "manual"),
 ]
 
-# The name of an image file (relative to this directory) to place at the top of
-# the title page.
-# latex_logo = None
-
-# For "manual" documents, if this is true, then toplevel headings are parts,
-# not chapters.
-# latex_use_parts = False
-
-# Additional stuff for the LaTeX preamble.
-# latex_preamble = ''
-
-# Documents to append as an appendix to all manuals.
-# latex_appendices = []
-
-# If false, no module index is generated.
-# latex_use_modindex = True
 
+# -- Options for the linkcheck builder ---------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-the-linkcheck-builder
 
-# Options for the linkcheck builder
-# ---------------------------------
-
-# A list of regular expressions that match URIs that should not be checked when
-# doing a linkcheck build.
 linkcheck_ignore = [
     r"http://localhost:\d+",
     "http://hg.scrapy.org",
-    "http://directory.google.com/",
+    r"https://github.com/scrapy/scrapy/commit/\w+",
     r"https://github.com/scrapy/scrapy/issues/\d+",
 ]
 
+linkcheck_anchors_ignore_for_url = ["https://github.com/pyca/cryptography/issues/2692"]
+
+# -- Options for the Coverage extension --------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/extensions/coverage.html#configuration
 
-# Options for the Coverage extension
-# ----------------------------------
 coverage_ignore_pyobjects = [
     # Contract’s add_pre_hook and add_post_hook are not documented because
     # they should be transparent to contract developers, for whom pre_hook and
@@ -256,6 +116,10 @@ coverage_ignore_pyobjects = [
     # Base classes of downloader middlewares are implementation details that
     # are not meant for users.
     r"^scrapy\.downloadermiddlewares\.\w*?\.Base\w*?Middleware",
+    # The interface methods of duplicate request filtering classes are already
+    # covered in the interface documentation part of the DUPEFILTER_CLASS
+    # setting documentation.
+    r"^scrapy\.dupefilters\.[A-Z]\w*?\.(from_settings|request_seen|open|close|log)$",
     # Private exception used by the command-line interface implementation.
     r"^scrapy\.exceptions\.UsageError",
     # Methods of BaseItemExporter subclasses are only documented in
@@ -273,8 +137,8 @@ coverage_ignore_pyobjects = [
 ]
 
 
-# Options for the InterSphinx extension
-# -------------------------------------
+# -- Options for the InterSphinx extension -----------------------------------
+# https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html#configuration
 
 intersphinx_mapping = {
     "attrs": ("https://www.attrs.org/en/stable/", None),
@@ -282,6 +146,7 @@ intersphinx_mapping = {
     "cryptography": ("https://cryptography.io/en/latest/", None),
     "cssselect": ("https://cssselect.readthedocs.io/en/latest", None),
     "itemloaders": ("https://itemloaders.readthedocs.io/en/latest/", None),
+    "parsel": ("https://parsel.readthedocs.io/en/latest/", None),
     "pytest": ("https://docs.pytest.org/en/latest", None),
     "python": ("https://docs.python.org/3", None),
     "sphinx": ("https://www.sphinx-doc.org/en/master", None),
@@ -290,11 +155,11 @@ intersphinx_mapping = {
     "twistedapi": ("https://docs.twisted.org/en/stable/api/", None),
     "w3lib": ("https://w3lib.readthedocs.io/en/latest", None),
 }
-intersphinx_disabled_reftypes = []
+intersphinx_disabled_reftypes: Sequence[str] = []
 
 
-# Options for sphinx-hoverxref options
-# ------------------------------------
+# -- Options for sphinx-hoverxref extension ----------------------------------
+# https://sphinx-hoverxref.readthedocs.io/en/latest/configuration.html
 
 hoverxref_auto_ref = True
 hoverxref_role_types = {
@@ -309,15 +174,3 @@ hoverxref_role_types = {
     "signal": "tooltip",
 }
 hoverxref_roles = ["command", "reqmeta", "setting", "signal"]
-
-
-def setup(app):
-    app.connect("autodoc-skip-member", maybe_skip_member)
-
-
-def maybe_skip_member(app, what, name, obj, skip, options):
-    if not skip:
-        # autodocs was generating a text "alias of" for the following members
-        # https://github.com/sphinx-doc/sphinx/issues/4422
-        return name in {"default_item_class", "default_selector_class"}
-    return skip
diff --git a/docs/news.rst b/docs/news.rst
index 2bf65272f..8230c3aef 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -635,7 +635,7 @@ Bug fixes
     exception if ``default`` is ``None``.
     (:issue:`6308`, :issue:`6310`)
 
--   :class:`~scrapy.selector.Selector` now uses
+-   :class:`~scrapy.Selector` now uses
     :func:`scrapy.utils.response.get_base_url` to determine the base URL of a
     given :class:`~scrapy.http.Response`. (:issue:`6265`)
 
@@ -653,7 +653,7 @@ Documentation
 -   Add a FAQ entry about :ref:`creating blank requests <faq-blank-request>`.
     (:issue:`6203`, :issue:`6208`)
 
--   Document that :attr:`scrapy.selector.Selector.type` can be ``"json"``.
+-   Document that :attr:`scrapy.Selector.type` can be ``"json"``.
     (:issue:`6328`, :issue:`6334`)
 
 Quality assurance
@@ -734,7 +734,7 @@ Documentation
 -   Improved documentation for :class:`~scrapy.crawler.Crawler` initialization
     changes made in the 2.11.0 release. (:issue:`6057`, :issue:`6147`)
 
--   Extended documentation for :attr:`Request.meta <scrapy.http.Request.meta>`.
+-   Extended documentation for :attr:`.Request.meta`.
     (:issue:`5565`)
 
 -   Fixed the :reqmeta:`dont_merge_cookies` documentation. (:issue:`5936`,
@@ -1095,7 +1095,7 @@ New features
     :setting:`RANDOMIZE_DOWNLOAD_DELAY` can now be set on a per-domain basis
     via the new :setting:`DOWNLOAD_SLOTS` setting. (:issue:`5328`)
 
--   Added :meth:`TextResponse.jmespath`, a shortcut for JMESPath selectors
+-   Added :meth:`.TextResponse.jmespath`, a shortcut for JMESPath selectors
     available since parsel_ 1.8.1. (:issue:`5894`, :issue:`5915`)
 
 -   Added :signal:`feed_slot_closed` and :signal:`feed_exporter_closed`
@@ -1275,7 +1275,7 @@ New features
     avoid confusion.
     (:issue:`5717`, :issue:`5722`, :issue:`5727`)
 
--   The ``callback`` parameter of :class:`~scrapy.http.Request` can now be set
+-   The ``callback`` parameter of :class:`~scrapy.Request` can now be set
     to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from
     ``None``, as the latter indicates that the default spider callback
     (:meth:`~scrapy.Spider.parse`) is to be used.
@@ -1772,17 +1772,17 @@ Highlights:
 Security bug fixes
 ~~~~~~~~~~~~~~~~~~
 
--   When a :class:`~scrapy.http.Request` object with cookies defined gets a
-    redirect response causing a new :class:`~scrapy.http.Request` object to be
+-   When a :class:`~scrapy.Request` object with cookies defined gets a
+    redirect response causing a new :class:`~scrapy.Request` object to be
     scheduled, the cookies defined in the original
-    :class:`~scrapy.http.Request` object are no longer copied into the new
-    :class:`~scrapy.http.Request` object.
+    :class:`~scrapy.Request` object are no longer copied into the new
+    :class:`~scrapy.Request` object.
 
     If you manually set the ``Cookie`` header on a
-    :class:`~scrapy.http.Request` object and the domain name of the redirect
+    :class:`~scrapy.Request` object and the domain name of the redirect
     URL is not an exact match for the domain of the URL of the original
-    :class:`~scrapy.http.Request` object, your ``Cookie`` header is now dropped
-    from the new :class:`~scrapy.http.Request` object.
+    :class:`~scrapy.Request` object, your ``Cookie`` header is now dropped
+    from the new :class:`~scrapy.Request` object.
 
     The old behavior could be exploited by an attacker to gain access to your
     cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more
@@ -1795,10 +1795,10 @@ Security bug fixes
               ``example.com`` and any subdomain) by defining the shared domain
               suffix (e.g. ``example.com``) as the cookie domain when defining
               your cookies. See the documentation of the
-              :class:`~scrapy.http.Request` class for more information.
+              :class:`~scrapy.Request` class for more information.
 
 -   When the domain of a cookie, either received in the ``Set-Cookie`` header
-    of a response or defined in a :class:`~scrapy.http.Request` object, is set
+    of a response or defined in a :class:`~scrapy.Request` object, is set
     to a `public suffix <https://publicsuffix.org/>`_, the cookie is now
     ignored unless the cookie domain is the same as the request domain.
 
@@ -1849,7 +1849,7 @@ Backward-incompatible changes
     meet expectations, :exc:`TypeError` is now raised at startup time. Before,
     other exceptions would be raised at run time. (:issue:`3559`)
 
--   The ``_encoding`` field of serialized :class:`~scrapy.http.Request` objects
+-   The ``_encoding`` field of serialized :class:`~scrapy.Request` objects
     is now named ``encoding``, in line with all other fields (:issue:`5130`)
 
 
@@ -1879,7 +1879,7 @@ Deprecations
 -   :mod:`scrapy.utils.reqser` is deprecated. (:issue:`5130`)
 
     -   Instead of :func:`~scrapy.utils.reqser.request_to_dict`, use the new
-        :meth:`Request.to_dict <scrapy.http.Request.to_dict>` method.
+        :meth:`.Request.to_dict` method.
 
     -   Instead of :func:`~scrapy.utils.reqser.request_from_dict`, use the new
         :func:`scrapy.utils.request.request_from_dict` function.
@@ -1984,9 +1984,9 @@ New features
     using ``queuelib`` 1.6.1 or later), the ``peek`` method raises
     :exc:`NotImplementedError`.
 
--   :class:`~scrapy.http.Request` and :class:`~scrapy.http.Response` now have
+-   :class:`~scrapy.Request` and :class:`~scrapy.http.Response` now have
     an ``attributes`` attribute that makes subclassing easier. For
-    :class:`~scrapy.http.Request`, it also allows subclasses to work with
+    :class:`~scrapy.Request`, it also allows subclasses to work with
     :func:`scrapy.utils.request.request_from_dict`. (:issue:`1877`,
     :issue:`5130`, :issue:`5218`)
 
@@ -2452,14 +2452,13 @@ Backward-incompatible changes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 *   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` once again
-    discards cookies defined in :attr:`Request.headers
-    <scrapy.http.Request.headers>`.
+    discards cookies defined in :attr:`.Request.headers`.
 
     We decided to revert this bug fix, introduced in Scrapy 2.2.0, because it
     was reported that the current implementation could break existing code.
 
     If you need to set cookies for a request, use the :class:`Request.cookies
-    <scrapy.http.Request>` parameter.
+    <scrapy.Request>` parameter.
 
     A future version of Scrapy will include a new, better implementation of the
     reverted bug fix.
@@ -2580,16 +2579,16 @@ New features
     :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`
     or
     :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception`
-    with a custom :class:`~scrapy.http.Request` object assigned to
+    with a custom :class:`~scrapy.Request` object assigned to
     :class:`response.request <scrapy.http.Response.request>`:
 
     -   The response is handled by the callback of that custom
-        :class:`~scrapy.http.Request` object, instead of being handled by the
-        callback of the original :class:`~scrapy.http.Request` object
+        :class:`~scrapy.Request` object, instead of being handled by the
+        callback of the original :class:`~scrapy.Request` object
 
-    -   That custom :class:`~scrapy.http.Request` object is now sent as the
+    -   That custom :class:`~scrapy.Request` object is now sent as the
         ``request`` argument to the :signal:`response_received` signal, instead
-        of the original :class:`~scrapy.http.Request` object
+        of the original :class:`~scrapy.Request` object
 
     (:issue:`4529`, :issue:`4632`)
 
@@ -2760,7 +2759,7 @@ New features
 *   The :command:`parse` command now allows specifying an output file
     (:issue:`4317`, :issue:`4377`)
 
-*   :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
+*   :meth:`.Request.from_curl` and
     :func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support
     ``--data-raw`` (:issue:`4612`)
 
@@ -2776,7 +2775,7 @@ Bug fixes
     :ref:`dataclass items <dataclass-items>` and :ref:`attr.s items
     <attrs-items>` (:issue:`4667`, :issue:`4668`)
 
-*   :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
+*   :meth:`.Request.from_curl` and
     :func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request
     method to ``POST`` when a request body is specified and no request method
     is specified (:issue:`4612`)
@@ -2861,8 +2860,7 @@ Backward-incompatible changes
 Deprecations
 ~~~~~~~~~~~~
 
-*   :meth:`TextResponse.body_as_unicode
-    <scrapy.http.TextResponse.body_as_unicode>` is now deprecated, use
+*   ``TextResponse.body_as_unicode()`` is now deprecated, use
     :attr:`TextResponse.text <scrapy.http.TextResponse.text>` instead
     (:issue:`4546`, :issue:`4555`, :issue:`4579`)
 
@@ -2901,9 +2899,8 @@ New features
 
 *   :ref:`Link extractors <topics-link-extractors>` are now serializable,
     as long as you do not use :ref:`lambdas <lambda>` for parameters; for
-    example, you can now pass link extractors in :attr:`Request.cb_kwargs
-    <scrapy.http.Request.cb_kwargs>` or
-    :attr:`Request.meta <scrapy.http.Request.meta>` when :ref:`persisting
+    example, you can now pass link extractors in :attr:`.Request.cb_kwargs`
+    or :attr:`.Request.meta` when :ref:`persisting
     scheduled requests <topics-jobs>` (:issue:`4554`)
 
 *   Upgraded the :ref:`pickle protocol <pickle-protocols>` that Scrapy uses
@@ -2922,11 +2919,11 @@ Bug fixes
 
 *   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
     discards cookies defined in :attr:`Request.headers
-    <scrapy.http.Request.headers>` (:issue:`1992`, :issue:`2400`)
+    <scrapy.Request.headers>` (:issue:`1992`, :issue:`2400`)
 
 *   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
     re-encodes cookies defined as :class:`bytes` in the ``cookies`` parameter
-    of the ``__init__`` method of :class:`~scrapy.http.Request`
+    of the ``__init__`` method of :class:`~scrapy.Request`
     (:issue:`2400`, :issue:`3575`)
 
 *   When :setting:`FEEDS` defines multiple URIs, :setting:`FEED_STORE_EMPTY` is
@@ -2935,7 +2932,7 @@ Bug fixes
 
 *   :class:`~scrapy.spiders.Spider` callbacks defined using :doc:`coroutine
     syntax <topics/coroutines>` no longer need to return an iterable, and may
-    instead return a :class:`~scrapy.http.Request` object, an
+    instead return a :class:`~scrapy.Request` object, an
     :ref:`item <topics-items>`, or ``None`` (:issue:`4609`)
 
 *   The :command:`startproject` command now ensures that the generated project
@@ -2976,8 +2973,8 @@ Documentation
     :issue:`4587`)
 
 *   The display-on-hover behavior of internal documentation references now also
-    covers links to :ref:`commands <topics-commands>`, :attr:`Request.meta
-    <scrapy.http.Request.meta>` keys, :ref:`settings <topics-settings>` and
+    covers links to :ref:`commands <topics-commands>`, :attr:`.Request.meta`
+    keys, :ref:`settings <topics-settings>` and
     :ref:`signals <topics-signals>` (:issue:`4495`, :issue:`4563`)
 
 *   It is again possible to download the documentation for offline reading
@@ -3262,7 +3259,7 @@ Deprecation removals
 ~~~~~~~~~~~~~~~~~~~~
 
 *   The :ref:`Scrapy shell <topics-shell>` no longer provides a `sel` proxy
-    object, use :meth:`response.selector <scrapy.http.Response.selector>`
+    object, use :meth:`response.selector <scrapy.http.TextResponse.selector>`
     instead (:issue:`4347`)
 
 *   LevelDB support has been removed (:issue:`4112`)
@@ -3332,10 +3329,10 @@ New features
 
 *   The new :attr:`Response.cb_kwargs <scrapy.http.Response.cb_kwargs>`
     attribute serves as a shortcut for :attr:`Response.request.cb_kwargs
-    <scrapy.http.Request.cb_kwargs>` (:issue:`4331`)
+    <scrapy.Request.cb_kwargs>` (:issue:`4331`)
 
 *   :meth:`Response.follow <scrapy.http.Response.follow>` now supports a
-    ``flags`` parameter, for consistency with :class:`~scrapy.http.Request`
+    ``flags`` parameter, for consistency with :class:`~scrapy.Request`
     (:issue:`4277`, :issue:`4279`)
 
 *   :ref:`Item loader processors <topics-loaders-processors>` can now be
@@ -3344,7 +3341,7 @@ New features
 *   :class:`~scrapy.spiders.Rule` now accepts an ``errback`` parameter
     (:issue:`4000`)
 
-*   :class:`~scrapy.http.Request` no longer requires a ``callback`` parameter
+*   :class:`~scrapy.Request` no longer requires a ``callback`` parameter
     when an ``errback`` parameter is specified (:issue:`3586`, :issue:`4008`)
 
 *   :class:`~scrapy.logformatter.LogFormatter` now supports some additional
@@ -3416,7 +3413,7 @@ Bug fixes
 *   Redirects to URLs starting with 3 slashes (``///``) are now supported
     (:issue:`4032`, :issue:`4042`)
 
-*   :class:`~scrapy.http.Request` no longer accepts strings as ``url`` simply
+*   :class:`~scrapy.Request` no longer accepts strings as ``url`` simply
     because they have a colon (:issue:`2552`, :issue:`4094`)
 
 *   The correct encoding is now used for attach names in
@@ -3462,7 +3459,7 @@ Documentation
     using :class:`~scrapy.crawler.CrawlerProcess` (:issue:`2149`,
     :issue:`2352`, :issue:`3146`, :issue:`3960`)
 
-*   Clarified the requirements for :class:`~scrapy.http.Request` objects
+*   Clarified the requirements for :class:`~scrapy.Request` objects
     :ref:`when using persistence <request-serialization>` (:issue:`4124`,
     :issue:`4139`)
 
@@ -3731,17 +3728,17 @@ Scrapy 1.8.2 (2022-03-01)
 
 **Security bug fixes:**
 
--   When a :class:`~scrapy.http.Request` object with cookies defined gets a
-    redirect response causing a new :class:`~scrapy.http.Request` object to be
+-   When a :class:`~scrapy.Request` object with cookies defined gets a
+    redirect response causing a new :class:`~scrapy.Request` object to be
     scheduled, the cookies defined in the original
-    :class:`~scrapy.http.Request` object are no longer copied into the new
-    :class:`~scrapy.http.Request` object.
+    :class:`~scrapy.Request` object are no longer copied into the new
+    :class:`~scrapy.Request` object.
 
     If you manually set the ``Cookie`` header on a
-    :class:`~scrapy.http.Request` object and the domain name of the redirect
+    :class:`~scrapy.Request` object and the domain name of the redirect
     URL is not an exact match for the domain of the URL of the original
-    :class:`~scrapy.http.Request` object, your ``Cookie`` header is now dropped
-    from the new :class:`~scrapy.http.Request` object.
+    :class:`~scrapy.Request` object, your ``Cookie`` header is now dropped
+    from the new :class:`~scrapy.Request` object.
 
     The old behavior could be exploited by an attacker to gain access to your
     cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more
@@ -3754,10 +3751,10 @@ Scrapy 1.8.2 (2022-03-01)
               ``example.com`` and any subdomain) by defining the shared domain
               suffix (e.g. ``example.com``) as the cookie domain when defining
               your cookies. See the documentation of the
-              :class:`~scrapy.http.Request` class for more information.
+              :class:`~scrapy.Request` class for more information.
 
 -   When the domain of a cookie, either received in the ``Set-Cookie`` header
-    of a response or defined in a :class:`~scrapy.http.Request` object, is set
+    of a response or defined in a :class:`~scrapy.Request` object, is set
     to a `public suffix <https://publicsuffix.org/>`_, the cookie is now
     ignored unless the cookie domain is the same as the request domain.
 
@@ -3815,7 +3812,7 @@ Highlights:
 
 * Dropped Python 3.4 support and updated minimum requirements; made Python 3.8
   support official
-* New :meth:`Request.from_curl <scrapy.http.Request.from_curl>` class method
+* New :meth:`.Request.from_curl` class method
 * New :setting:`ROBOTSTXT_PARSER` and :setting:`ROBOTSTXT_USER_AGENT` settings
 * New :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and
   :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` settings
@@ -3869,7 +3866,7 @@ See also :ref:`1.8-deprecation-removals` below.
 New features
 ~~~~~~~~~~~~
 
-*   A new :meth:`Request.from_curl <scrapy.http.Request.from_curl>` class
+*   A new :meth:`Request.from_curl <scrapy.Request.from_curl>` class
     method allows :ref:`creating a request from a cURL command
     <requests-from-curl>` (:issue:`2985`, :issue:`3862`)
 
@@ -3898,9 +3895,8 @@ New features
     ``True`` to enable debug-level messages about TLS connection parameters
     after establishing HTTPS connections (:issue:`2111`, :issue:`3450`)
 
-*   Callbacks that receive keyword arguments
-    (see :attr:`Request.cb_kwargs <scrapy.http.Request.cb_kwargs>`) can now be
-    tested using the new :class:`@cb_kwargs
+*   Callbacks that receive keyword arguments (see :attr:`.Request.cb_kwargs`)
+    can now be tested using the new :class:`@cb_kwargs
     <scrapy.contracts.default.CallbackKeywordArgumentsContract>`
     :ref:`spider contract <topics-contracts>` (:issue:`3985`, :issue:`3988`)
 
@@ -4089,7 +4085,7 @@ Backward-incompatible changes
 
 *   Non-default values for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting
     may stop working. Scheduler priority queue classes now need to handle
-    :class:`~scrapy.http.Request` objects instead of arbitrary Python data
+    :class:`~scrapy.Request` objects instead of arbitrary Python data
     structures.
 
 *   An additional ``crawler`` parameter has been added to the ``__init__``
@@ -4111,7 +4107,7 @@ New features
     scheduling improvement on crawls targeting multiple web domains, at the
     cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)
 
-*   A new :attr:`Request.cb_kwargs <scrapy.http.Request.cb_kwargs>` attribute
+*   A new :attr:`.Request.cb_kwargs` attribute
     provides a cleaner way to pass keyword arguments to callback methods
     (:issue:`1138`, :issue:`3563`)
 
@@ -4192,7 +4188,7 @@ Bug fixes
 *   Requests with private callbacks are now correctly unserialized from disk
     (:issue:`3790`)
 
-*   :meth:`FormRequest.from_response() <scrapy.http.FormRequest.from_response>`
+*   :meth:`.FormRequest.from_response`
     now handles invalid methods like major web browsers (:issue:`3777`,
     :issue:`3794`)
 
@@ -4272,13 +4268,13 @@ The following deprecated APIs have been removed (:issue:`3578`):
 
 *   From both ``scrapy.selector`` and ``scrapy.selector.lxmlsel``:
 
-    *   ``HtmlXPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``HtmlXPathSelector`` (use :class:`~scrapy.Selector`)
 
-    *   ``XmlXPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XmlXPathSelector`` (use :class:`~scrapy.Selector`)
 
-    *   ``XPathSelector`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XPathSelector`` (use :class:`~scrapy.Selector`)
 
-    *   ``XPathSelectorList`` (use :class:`~scrapy.selector.Selector`)
+    *   ``XPathSelectorList`` (use :class:`~scrapy.Selector`)
 
 *   From ``scrapy.selector.csstranslator``:
 
@@ -4288,7 +4284,7 @@ The following deprecated APIs have been removed (:issue:`3578`):
 
     *   ``ScrapyXPathExpr`` (use parsel.csstranslator.XPathExpr_)
 
-*   From :class:`~scrapy.selector.Selector`:
+*   From :class:`~scrapy.Selector`:
 
     *   ``_root`` (both the ``__init__`` method argument and the object property, use
         ``root``)
@@ -4818,7 +4814,7 @@ New Features
   (:issue:`2535`)
 - New :ref:`response.follow <response-follow-example>` shortcut
   for creating requests (:issue:`1940`)
-- Added ``flags`` argument and attribute to :class:`Request <scrapy.http.Request>`
+- Added ``flags`` argument and attribute to :class:`~scrapy.Request`
   objects (:issue:`2047`)
 - Support Anonymous FTP (:issue:`2342`)
 - Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/<reason>``
@@ -4860,7 +4856,7 @@ Bug fixes
 - LinkExtractor now strips leading and trailing whitespaces from attributes
   (:issue:`2547`, fixes :issue:`1614`)
 - Properly handle whitespaces in action attribute in
-  :class:`~scrapy.http.FormRequest` (:issue:`2548`)
+  :class:`~scrapy.FormRequest` (:issue:`2548`)
 - Buffer CONNECT response bytes from proxy until all HTTP headers are received
   (:issue:`2495`, fixes :issue:`2491`)
 - FTP downloader now works on Python 3, provided you use Twisted>=17.1
@@ -4902,8 +4898,7 @@ Documentation
 ~~~~~~~~~~~~~
 
 - Binary mode is required for exporters (:issue:`2564`, fixes :issue:`2553`)
-- Mention issue with :meth:`FormRequest.from_response
-  <scrapy.http.FormRequest.from_response>` due to bug in lxml (:issue:`2572`)
+- Mention issue with :meth:`.FormRequest.from_response` due to bug in lxml (:issue:`2572`)
 - Use single quotes uniformly in templates (:issue:`2596`)
 - Document :reqmeta:`ftp_user` and :reqmeta:`ftp_password` meta keys (:issue:`2587`)
 - Removed section on deprecated ``contrib/`` (:issue:`2636`)
@@ -5442,7 +5437,7 @@ Bugfixes
 - Support empty password for http_proxy config (:issue:`1274`).
 - Interpret ``application/x-json`` as ``TextResponse`` (:issue:`1333`).
 - Support link rel attribute with multiple values (:issue:`1201`).
-- Fixed ``scrapy.http.FormRequest.from_response`` when there is a ``<base>``
+- Fixed ``scrapy.FormRequest.from_response`` when there is a ``<base>``
   tag (:issue:`1564`).
 - Fixed :setting:`TEMPLATES_DIR` handling (:issue:`1575`).
 - Various ``FormRequest`` fixes (:issue:`1595`, :issue:`1596`, :issue:`1597`).
@@ -6369,7 +6364,7 @@ Scrapy 0.18.0 (released 2013-08-09)
 - Moved persistent (on disk) queues to a separate project (queuelib_) which Scrapy now depends on
 - Add Scrapy commands using external libraries (:issue:`260`)
 - Added ``--pdb`` option to ``scrapy`` command line tool
-- Added :meth:`XPathSelector.remove_namespaces <scrapy.selector.Selector.remove_namespaces>` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`.
+- Added :meth:`XPathSelector.remove_namespaces <scrapy.Selector.remove_namespaces>` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`.
 - Several improvements to spider contracts
 - New default middleware named MetaRefreshMiddleware that handles meta-refresh html tag redirections,
 - MetaRefreshMiddleware and RedirectMiddleware have different priorities to address #62
@@ -6867,7 +6862,7 @@ First release of Scrapy.
 .. _parsel.csstranslator.HTMLTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.HTMLTranslator
 .. _parsel.csstranslator.XPathExpr: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.XPathExpr
 .. _PEP 257: https://peps.python.org/pep-0257/
-.. _Pillow: https://python-pillow.org/
+.. _Pillow: https://github.com/python-pillow/Pillow
 .. _pyOpenSSL: https://www.pyopenssl.org/en/stable/
 .. _queuelib: https://github.com/scrapy/queuelib
 .. _registered with IANA: https://www.iana.org/assignments/media-types/media-types.xhtml
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 5f683d34c..7ee897170 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,4 +1,4 @@
 sphinx==6.2.1
-sphinx-hoverxref==1.3.0
-sphinx-notfound-page==1.0.0
-sphinx-rtd-theme==2.0.0
+sphinx-hoverxref==1.4.2
+sphinx-notfound-page==1.0.4
+sphinx-rtd-theme==3.0.2
diff --git a/docs/topics/addons.rst b/docs/topics/addons.rst
index 14b4aa8ba..46cf1edbd 100644
--- a/docs/topics/addons.rst
+++ b/docs/topics/addons.rst
@@ -32,7 +32,7 @@ This is an example where two add-ons are enabled in a project's
 Writing your own add-ons
 ========================
 
-Add-ons are Python classes that include the following method:
+Add-ons are Python classes that include one or both of the following methods:
 
 .. method:: update_settings(settings)
 
@@ -45,6 +45,15 @@ Add-ons are Python classes that include the following method:
     :param settings: The settings object storing Scrapy/component configuration
     :type settings: :class:`~scrapy.settings.Settings`
 
+.. classmethod:: update_pre_crawler_settings(cls, settings)
+
+    Use this class method instead of the :meth:`update_settings` method to
+    update :ref:`pre-crawler settings <pre-crawler-settings>` whose value is
+    used before the :class:`~scrapy.crawler.Crawler` object is created.
+
+    :param settings: The settings object storing Scrapy/component configuration
+    :type settings: :class:`~scrapy.settings.BaseSettings`
+
 They can also have the following method:
 
 .. classmethod:: from_crawler(cls, crawler)
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 11a3fcb94..1ab8f588f 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -80,7 +80,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
       middleware.
 
       :meth:`process_request` should either: return ``None``, return a
-      :class:`~scrapy.Response` object, return a :class:`~scrapy.http.Request`
+      :class:`~scrapy.http.Response` object, return a :class:`~scrapy.Request`
       object, or raise :exc:`~scrapy.exceptions.IgnoreRequest`.
 
       If it returns ``None``, Scrapy will continue processing this request, executing all
@@ -797,9 +797,12 @@ OffsiteMiddleware
    :attr:`~scrapy.Spider.allowed_domains` attribute, or the
    attribute is empty, the offsite middleware will allow all requests.
 
-   If the request has the :attr:`~scrapy.Request.dont_filter` attribute
-   set, the offsite middleware will allow the request even if its domain is not
-   listed in allowed domains.
+   .. reqmeta:: allow_offsite
+
+   If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to
+   ``True`` or :attr:`Request.meta` has ``allow_offsite`` set to ``True``, then
+   the OffsiteMiddleware will allow the request even if its domain is not listed
+   in allowed domains.
 
 RedirectMiddleware
 ------------------
diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst
index 75d980835..801f6d06d 100644
--- a/docs/topics/dynamic-content.rst
+++ b/docs/topics/dynamic-content.rst
@@ -117,7 +117,8 @@ data from it depends on the type of response:
 -   If the response is HTML, XML or JSON, use :ref:`selectors
     <topics-selectors>` as usual.
 
--   If the response is JSON, use :func:`response.json()` to load the desired data:
+-   If the response is JSON, use :func:`response.json()
+    <scrapy.http.TextResponse.json>` to load the desired data:
 
     .. code-block:: python
 
@@ -143,7 +144,7 @@ data from it depends on the type of response:
 
 -   If the response is an image or another format based on images (e.g. PDF),
     read the response as bytes from
-    :attr:`response.body <scrapy.http.TextResponse.body>` and use an OCR
+    :attr:`response.body <scrapy.http.Response.body>` and use an OCR
     solution to extract the desired data as text.
 
     For example, you can use pytesseract_. To read a table from a PDF,
diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst
index ea64edbe6..0b572ff95 100644
--- a/docs/topics/exceptions.rst
+++ b/docs/topics/exceptions.rst
@@ -105,7 +105,7 @@ response:
 In both cases, the response could have its body truncated: the body contains
 all bytes received up until the exception is raised, including the bytes
 received in the signal handler that raises the exception. Also, the response
-object is marked with ``"download_stopped"`` in its :attr:`Response.flags`
+object is marked with ``"download_stopped"`` in its :attr:`~scrapy.http.Response.flags`
 attribute.
 
 .. note:: ``fail`` is a keyword-only parameter, i.e. raising
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 1500011e7..5f6798601 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -246,24 +246,10 @@ Same example but running the spiders sequentially by chaining the deferreds:
     crawl()
     reactor.run()  # the script will block here until the last crawl call is finished
 
-Different spiders can set different values for the same setting, but when they
-run in the same process it may be impossible, by design or because of some
-limitations, to use these different values. What happens in practice is
-different for different settings:
-
-* :setting:`SPIDER_LOADER_CLASS` and the ones used by its value
-  (:setting:`SPIDER_MODULES`, :setting:`SPIDER_LOADER_WARN_ONLY` for the
-  default one) cannot be read from the per-spider settings. These are applied
-  when the :class:`~scrapy.crawler.CrawlerRunner` or
-  :class:`~scrapy.crawler.CrawlerProcess` object is created.
-* For :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` the first
-  available value is used, and if a spider requests a different reactor an
-  exception will be raised. These are applied when the reactor is installed.
-* For :setting:`REACTOR_THREADPOOL_MAXSIZE`, :setting:`DNS_RESOLVER` and the
-  ones used by the resolver (:setting:`DNSCACHE_ENABLED`,
-  :setting:`DNSCACHE_SIZE`, :setting:`DNS_TIMEOUT` for ones included in Scrapy)
-  the first available value is used. These are applied when the reactor is
-  started.
+.. note:: When running multiple spiders in the same process, :ref:`reactor
+    settings <reactor-settings>` should not have a different value per spider.
+    Also, :ref:`pre-crawler settings <pre-crawler-settings>` cannot be defined
+    per spider.
 
 .. seealso:: :ref:`run-from-script`.
 
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 710e2e131..b187f3aaf 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -7,15 +7,15 @@ Requests and Responses
 .. module:: scrapy.http
    :synopsis: Request and Response classes
 
-Scrapy uses :class:`Request` and :class:`Response` objects for crawling web
+Scrapy uses :class:`~scrapy.Request` and :class:`Response` objects for crawling web
 sites.
 
-Typically, :class:`Request` objects are generated in the spiders and pass
+Typically, :class:`~scrapy.Request` objects are generated in the spiders and pass
 across the system until they reach the Downloader, which executes the request
 and returns a :class:`Response` object which travels back to the spider that
 issued the request.
 
-Both :class:`Request` and :class:`Response` classes have subclasses which add
+Both :class:`~scrapy.Request` and :class:`Response` classes have subclasses which add
 functionality not required in the base classes. These are described
 below in :ref:`topics-request-response-ref-request-subclasses` and
 :ref:`topics-request-response-ref-response-subclasses`.
@@ -24,7 +24,7 @@ below in :ref:`topics-request-response-ref-request-subclasses` and
 Request objects
 ===============
 
-.. autoclass:: Request
+.. autoclass:: scrapy.Request
 
     :param url: the URL of this request
 
@@ -52,7 +52,7 @@ Request objects
     :param method: the HTTP method of this request. Defaults to ``'GET'``.
     :type method: str
 
-    :param meta: the initial values for the :attr:`Request.meta` attribute. If
+    :param meta: the initial values for the :attr:`.Request.meta` attribute. If
        given, the dict passed in this parameter will be shallow copied.
     :type meta: dict
 
@@ -67,10 +67,10 @@ Request objects
        (for single valued headers) or lists (for multi-valued headers). If
        ``None`` is passed as value, the HTTP header will not be sent at all.
 
-        .. caution:: Cookies set via the ``Cookie`` header are not considered by the
-            :ref:`cookies-mw`. If you need to set cookies for a request, use the
-            :class:`Request.cookies <scrapy.Request>` parameter. This is a known
-            current limitation that is being worked on.
+       .. caution:: Cookies set via the ``Cookie`` header are not considered by the
+           :ref:`cookies-mw`. If you need to set cookies for a request, use the
+           ``cookies`` argument. This is a known current limitation that is being
+           worked on.
 
     :type headers: dict
 
@@ -124,7 +124,7 @@ Request objects
 
         .. caution:: Cookies set via the ``Cookie`` header are not considered by the
             :ref:`cookies-mw`. If you need to set cookies for a request, use the
-            :class:`Request.cookies <scrapy.Request>` parameter. This is a known
+            :class:`scrapy.Request.cookies <scrapy.Request>` parameter. This is a known
             current limitation that is being worked on.
 
         .. versionadded:: 2.6.0
@@ -145,9 +145,9 @@ Request objects
     :type priority: int
 
     :param dont_filter: indicates that this request should not be filtered by
-       the scheduler. This is used when you want to perform an identical
-       request multiple times, to ignore the duplicates filter. Use it with
-       care, or you will get into crawling loops. Default to ``False``.
+       the scheduler or some middlewares. This is used when you want to perform
+       an identical request multiple times, to ignore the duplicates filter.
+       Use it with care, or you will get into crawling loops. Default to ``False``.
     :type dont_filter: bool
 
     :param errback: a function that will be called if any exception was
@@ -172,7 +172,7 @@ Request objects
 
         A string containing the URL of this request. Keep in mind that this
         attribute contains the escaped URL, so it can differ from the URL passed in
-        the ``__init__`` method.
+        the ``__init__()`` method.
 
         This attribute is read-only. To change the URL of a Request use
         :meth:`replace`.
@@ -184,7 +184,8 @@ Request objects
 
     .. attribute:: Request.headers
 
-        A dictionary-like object which contains the request headers.
+        A dictionary-like (:class:`scrapy.http.headers.Headers`) object which contains
+        the request headers.
 
     .. attribute:: Request.body
 
@@ -240,8 +241,8 @@ Request objects
 
         A dictionary that contains arbitrary metadata for this request. Its contents
         will be passed to the Request's callback as keyword arguments. It is empty
-        for new Requests, which means by default callbacks only get a :class:`Response`
-        object as argument.
+        for new Requests, which means by default callbacks only get a
+        :class:`~scrapy.http.Response` object as argument.
 
         This dict is :doc:`shallow copied <library/copy>` when the request is
         cloned using the ``copy()`` or ``replace()`` methods, and can also be
@@ -262,7 +263,7 @@ Request objects
 
        Return a Request object with the same members, except for those members
        given new values by whichever keyword arguments are specified. The
-       :attr:`Request.cb_kwargs` and :attr:`Request.meta` attributes are shallow
+       :attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta` attributes are shallow
        copied by default (unless new values are given as arguments). See also
        :ref:`topics-request-response-ref-request-callback-arguments`.
 
@@ -305,7 +306,7 @@ Example:
 In some cases you may be interested in passing arguments to those callback
 functions so you can receive the arguments later, in the second callback.
 The following example shows how to achieve this by using the
-:attr:`Request.cb_kwargs` attribute:
+:attr:`.Request.cb_kwargs` attribute:
 
 .. code-block:: python
 
@@ -326,10 +327,10 @@ The following example shows how to achieve this by using the
             foo=foo,
         )
 
-.. caution:: :attr:`Request.cb_kwargs` was introduced in version ``1.7``.
-   Prior to that, using :attr:`Request.meta` was recommended for passing
-   information around callbacks. After ``1.7``, :attr:`Request.cb_kwargs`
-   became the preferred way for handling user information, leaving :attr:`Request.meta`
+.. caution:: :attr:`.Request.cb_kwargs` was introduced in version ``1.7``.
+   Prior to that, using :attr:`.Request.meta` was recommended for passing
+   information around callbacks. After ``1.7``, :attr:`.Request.cb_kwargs`
+   became the preferred way for handling user information, leaving :attr:`.Request.meta`
    for communication with components like middlewares and extensions.
 
 .. _topics-request-response-ref-errbacks:
@@ -441,7 +442,7 @@ Request fingerprints
 There are some aspects of scraping, such as filtering out duplicate requests
 (see :setting:`DUPEFILTER_CLASS`) or caching responses (see
 :setting:`HTTPCACHE_POLICY`), where you need the ability to generate a short,
-unique identifier from a :class:`~scrapy.http.Request` object: a request
+unique identifier from a :class:`~scrapy.Request` object: a request
 fingerprint.
 
 You often do not need to worry about request fingerprints, the default request
@@ -486,7 +487,7 @@ A request fingerprinter is a class that must implement the following method:
    See also :ref:`request-fingerprint-restrictions`.
 
    :param request: request to fingerprint
-   :type request: scrapy.http.Request
+   :type request: scrapy.Request
 
 Additionally, it may also implement the following method:
 
@@ -566,7 +567,7 @@ URL canonicalization or taking the request method or body into account:
 
 If you need to be able to override the request fingerprinting for arbitrary
 requests from your spider callbacks, you may implement a request fingerprinter
-that reads fingerprints from :attr:`request.meta <scrapy.http.Request.meta>`
+that reads fingerprints from :attr:`request.meta <scrapy.Request.meta>`
 when available, and then falls back to
 :func:`scrapy.utils.request.fingerprint`. For example:
 
@@ -581,10 +582,8 @@ when available, and then falls back to
                 return request.meta["fingerprint"]
             return fingerprint(request)
 
-If you need to reproduce the same fingerprinting algorithm as Scrapy 2.6
-without using the deprecated ``'2.6'`` value of the
-:setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION` setting, use the following
-request fingerprinter:
+If you need to reproduce the same fingerprinting algorithm as Scrapy 2.6, use
+the following request fingerprinter:
 
 .. code-block:: python
 
@@ -628,7 +627,7 @@ The following built-in Scrapy components have such restrictions:
     :setting:`HTTPCACHE_DIR` also apply. Inside :setting:`HTTPCACHE_DIR`,
     the following directory structure is created:
 
-    -   :attr:`Spider.name <scrapy.spiders.Spider.name>`
+    -   :attr:`.Spider.name`
 
         -   first byte of a request fingerprint as hexadecimal
 
@@ -656,11 +655,12 @@ The following built-in Scrapy components have such restrictions:
 Request.meta special keys
 =========================
 
-The :attr:`Request.meta` attribute can contain any arbitrary data, but there
+The :attr:`.Request.meta` attribute can contain any arbitrary data, but there
 are some special keys recognized by Scrapy and its built-in extensions.
 
 Those are:
 
+* :reqmeta:`allow_offsite`
 * :reqmeta:`autothrottle_dont_adjust_delay`
 * :reqmeta:`bindaddress`
 * :reqmeta:`cookiejar`
@@ -780,24 +780,25 @@ call their callback instead, like in this example, pass ``fail=False`` to the
 Request subclasses
 ==================
 
-Here is the list of built-in :class:`Request` subclasses. You can also subclass
+Here is the list of built-in :class:`~scrapy.Request` subclasses. You can also subclass
 it to implement your own custom functionality.
 
 FormRequest objects
 -------------------
 
-The FormRequest class extends the base :class:`Request` with functionality for
+The FormRequest class extends the base :class:`~scrapy.Request` with functionality for
 dealing with HTML forms. It uses `lxml.html forms`_  to pre-populate form
 fields with form data from :class:`Response` objects.
 
 .. _lxml.html forms: https://lxml.de/lxmlhtml.html#forms
 
-.. class:: scrapy.http.request.form.FormRequest
-.. class:: scrapy.http.FormRequest
+.. currentmodule:: None
+
 .. class:: scrapy.FormRequest(url, [formdata, ...])
+    :canonical: scrapy.http.request.form.FormRequest
 
-    The :class:`FormRequest` class adds a new keyword parameter to the ``__init__`` method. The
-    remaining arguments are the same as for the :class:`Request` class and are
+    The :class:`~scrapy.FormRequest` class adds a new keyword parameter to the ``__init__()`` method. The
+    remaining arguments are the same as for the :class:`~scrapy.Request` class and are
     not documented here.
 
     :param formdata: is a dictionary (or iterable of (key, value) tuples)
@@ -805,12 +806,12 @@ fields with form data from :class:`Response` objects.
        body of the request.
     :type formdata: dict or collections.abc.Iterable
 
-    The :class:`FormRequest` objects support the following class method in
-    addition to the standard :class:`Request` methods:
+    The :class:`~scrapy.FormRequest` objects support the following class method in
+    addition to the standard :class:`~scrapy.Request` methods:
 
-    .. classmethod:: FormRequest.from_response(response, [formname=None, formid=None, formnumber=0, formdata=None, formxpath=None, formcss=None, clickdata=None, dont_click=False, ...])
+    .. classmethod:: from_response(response, [formname=None, formid=None, formnumber=0, formdata=None, formxpath=None, formcss=None, clickdata=None, dont_click=False, ...])
 
-       Returns a new :class:`FormRequest` object with its form field values
+       Returns a new :class:`~scrapy.FormRequest` object with its form field values
        pre-populated with those found in the HTML ``<form>`` element contained
        in the given response. For an example see
        :ref:`topics-request-response-ref-request-userlogin`.
@@ -832,7 +833,7 @@ fields with form data from :class:`Response` objects.
 
        :param response: the response containing a HTML form which will be used
           to pre-populate the form fields
-       :type response: :class:`Response` object
+       :type response: :class:`~scrapy.http.Response` object
 
        :param formname: if given, the form with name attribute set to this value will be used.
        :type formname: str
@@ -869,7 +870,9 @@ fields with form data from :class:`Response` objects.
        :type dont_click: bool
 
        The other parameters of this class method are passed directly to the
-       :class:`FormRequest` ``__init__`` method.
+       :class:`~scrapy.FormRequest` ``__init__()`` method.
+
+.. currentmodule:: scrapy.http
 
 Request usage examples
 ----------------------
@@ -878,7 +881,7 @@ Using FormRequest to send data via HTTP POST
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you want to simulate a HTML Form POST in your spider and send a couple of
-key-value fields, you can return a :class:`FormRequest` object (from your
+key-value fields, you can return a :class:`~scrapy.FormRequest` object (from your
 spider) like this:
 
 .. skip: next
@@ -901,7 +904,7 @@ It is usual for web sites to provide pre-populated form fields through ``<input
 type="hidden">`` elements, such as session related data or authentication
 tokens (for login pages). When scraping, you'll want these fields to be
 automatically pre-populated and only override a couple of them, such as the
-user name and password. You can use the :meth:`FormRequest.from_response`
+user name and password. You can use the :meth:`.FormRequest.from_response()`
 method for this job. Here's an example spider which uses it:
 
 .. code-block:: python
@@ -936,21 +939,22 @@ method for this job. Here's an example spider which uses it:
 JsonRequest
 -----------
 
-The JsonRequest class extends the base :class:`Request` class with functionality for
+The JsonRequest class extends the base :class:`~scrapy.Request` class with functionality for
 dealing with JSON requests.
 
 .. class:: JsonRequest(url, [... data, dumps_kwargs])
 
-   The :class:`JsonRequest` class adds two new keyword parameters to the ``__init__`` method. The
-   remaining arguments are the same as for the :class:`Request` class and are
+   The :class:`JsonRequest` class adds two new keyword parameters to the ``__init__()`` method. The
+   remaining arguments are the same as for the :class:`~scrapy.Request` class and are
    not documented here.
 
    Using the :class:`JsonRequest` will set the ``Content-Type`` header to ``application/json``
    and ``Accept`` header to ``application/json, text/javascript, */*; q=0.01``
 
    :param data: is any JSON serializable object that needs to be JSON encoded and assigned to body.
-      if :attr:`Request.body` argument is provided this parameter will be ignored.
-      if :attr:`Request.body` argument is not provided and data argument is provided :attr:`Request.method` will be
+      If the :attr:`~scrapy.Request.body` argument is provided this parameter will be ignored.
+      If the :attr:`~scrapy.Request.body` argument is not provided and the
+      ``data`` argument is provided the :attr:`~scrapy.Request.method` will be
       set to ``'POST'`` automatically.
    :type data: object
 
@@ -1002,7 +1006,7 @@ Response objects
     :type flags: list
 
     :param request: the initial value of the :attr:`Response.request` attribute.
-        This represents the :class:`Request` that generated this response.
+        This represents the :class:`~scrapy.Request` that generated this response.
     :type request: scrapy.Request
 
     :param certificate: an object representing the server's SSL certificate.
@@ -1038,11 +1042,12 @@ Response objects
 
     .. attribute:: Response.headers
 
-        A dictionary-like object which contains the response headers. Values can
-        be accessed using :meth:`get` to return the first header value with the
-        specified name or :meth:`getlist` to return all header values with the
-        specified name. For example, this call will give you all cookies in the
-        headers::
+        A dictionary-like (:class:`scrapy.http.headers.Headers`) object which contains
+        the response headers. Values can be accessed using
+        :meth:`~scrapy.http.headers.Headers.get` to return the first header value with
+        the specified name or :meth:`~scrapy.http.headers.Headers.getlist` to return
+        all header values with the specified name. For example, this call will give you
+        all cookies in the headers::
 
             response.headers.getlist('Set-Cookie')
 
@@ -1058,7 +1063,7 @@ Response objects
 
     .. attribute:: Response.request
 
-        The :class:`Request` object that generated this response. This attribute is
+        The :class:`~scrapy.Request` object that generated this response. This attribute is
         assigned in the Scrapy engine, after the response and the request have passed
         through all :ref:`Downloader Middlewares <topics-downloader-middleware>`.
         In particular, this means that:
@@ -1077,34 +1082,33 @@ Response objects
 
     .. attribute:: Response.meta
 
-        A shortcut to the :attr:`Request.meta` attribute of the
+        A shortcut to the :attr:`~scrapy.Request.meta` attribute of the
         :attr:`Response.request` object (i.e. ``self.request.meta``).
 
         Unlike the :attr:`Response.request` attribute, the :attr:`Response.meta`
         attribute is propagated along redirects and retries, so you will get
-        the original :attr:`Request.meta` sent from your spider.
+        the original :attr:`.Request.meta` sent from your spider.
 
-        .. seealso:: :attr:`Request.meta` attribute
+        .. seealso:: :attr:`.Request.meta` attribute
 
     .. attribute:: Response.cb_kwargs
 
         .. versionadded:: 2.0
 
-        A shortcut to the :attr:`Request.cb_kwargs` attribute of the
+        A shortcut to the :attr:`~scrapy.Request.cb_kwargs` attribute of the
         :attr:`Response.request` object (i.e. ``self.request.cb_kwargs``).
 
         Unlike the :attr:`Response.request` attribute, the
         :attr:`Response.cb_kwargs` attribute is propagated along redirects and
-        retries, so you will get the original :attr:`Request.cb_kwargs` sent
-        from your spider.
+        retries, so you will get the original :attr:`.Request.cb_kwargs` sent from your spider.
 
-        .. seealso:: :attr:`Request.cb_kwargs` attribute
+        .. seealso:: :attr:`.Request.cb_kwargs` attribute
 
     .. attribute:: Response.flags
 
         A list that contains flags for this response. Flags are labels used for
         tagging Responses. For example: ``'cached'``, ``'redirected``', etc. And
-        they're shown on the string representation of the Response (`__str__`
+        they're shown on the string representation of the Response (``__str__()``
         method) which is used by the engine for logging.
 
     .. attribute:: Response.certificate
@@ -1181,7 +1185,7 @@ TextResponse objects
     :class:`Response` class, which is meant to be used only for binary data,
     such as images, sounds or any media file.
 
-    :class:`TextResponse` objects support a new ``__init__`` method argument, in
+    :class:`TextResponse` objects support a new ``__init__()`` method argument, in
     addition to the base :class:`Response` objects. The remaining functionality
     is the same as for the :class:`Response` class and is not documented here.
 
@@ -1219,7 +1223,7 @@ TextResponse objects
        A string with the encoding of this response. The encoding is resolved by
        trying the following mechanisms, in order:
 
-       1. the encoding passed in the ``__init__`` method ``encoding`` argument
+       1. the encoding passed in the ``__init__()`` method ``encoding`` argument
 
        2. the encoding declared in the Content-Type HTTP header. If this
           encoding is not valid (i.e. unknown), it is ignored and the next
@@ -1273,7 +1277,7 @@ TextResponse objects
 
         Constructs an absolute url by combining the Response's base url with
         a possible relative url. The base url shall be extracted from the
-        ``<base>`` tag, or just the Response's :attr:`url` if there is no such
+        ``<base>`` tag, or just :attr:`Response.url` if there is no such
         tag.
 
 
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index 202b0823a..b95e6eab3 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -777,7 +777,7 @@ Removing namespaces
 When dealing with scraping projects, it is often quite convenient to get rid of
 namespaces altogether and just work with element names, to write more
 simple/convenient XPaths. You can use the
-:meth:`Selector.remove_namespaces` method for that.
+:meth:`.Selector.remove_namespaces` method for that.
 
 Let's show an example that illustrates this with the Python Insider blog atom feed.
 
@@ -814,7 +814,7 @@ doesn't work (because the Atom XML namespace is obfuscating those nodes):
     >>> response.xpath("//link")
     []
 
-But once we call the :meth:`Selector.remove_namespaces` method, all
+But once we call the :meth:`.Selector.remove_namespaces` method, all
 nodes can be accessed directly by their names:
 
 .. code-block:: pycon
@@ -1046,7 +1046,7 @@ Built-in Selectors reference
 Selector objects
 ----------------
 
-.. autoclass:: Selector
+.. autoclass:: scrapy.Selector
 
   .. automethod:: xpath
 
@@ -1126,8 +1126,8 @@ Examples
 Selector examples on HTML response
 ----------------------------------
 
-Here are some :class:`Selector` examples to illustrate several concepts.
-In all cases, we assume there is already a :class:`Selector` instantiated with
+Here are some :class:`~scrapy.Selector` examples to illustrate several concepts.
+In all cases, we assume there is already a :class:`~scrapy.Selector` instantiated with
 a :class:`~scrapy.http.HtmlResponse` object like this:
 
 .. code-block:: python
@@ -1135,7 +1135,7 @@ a :class:`~scrapy.http.HtmlResponse` object like this:
       sel = Selector(html_response)
 
 1. Select all ``<h1>`` elements from an HTML response body, returning a list of
-   :class:`Selector` objects (i.e. a :class:`SelectorList` object):
+   :class:`~scrapy.Selector` objects (i.e. a :class:`SelectorList` object):
 
    .. code-block:: python
 
@@ -1165,7 +1165,7 @@ Selector examples on XML response
 
 .. skip: start
 
-Here are some examples to illustrate concepts for :class:`Selector` objects
+Here are some examples to illustrate concepts for :class:`~scrapy.Selector` objects
 instantiated with an :class:`~scrapy.http.XmlResponse` object:
 
 .. code-block:: python
@@ -1173,7 +1173,7 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object:
       sel = Selector(xml_response)
 
 1. Select all ``<product>`` elements from an XML response body, returning a list
-   of :class:`Selector` objects (i.e. a :class:`SelectorList` object):
+   of :class:`~scrapy.Selector` objects (i.e. a :class:`SelectorList` object):
 
    .. code-block:: python
 
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 76904a26e..a53e0806d 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -33,42 +33,48 @@ Python :ref:`import search path <tut-searchpath>`.
 Populating the settings
 =======================
 
-Settings can be populated using different mechanisms, each of which having a
-different precedence. Here is the list of them in decreasing order of
-precedence:
+Settings can be populated using different mechanisms, each of which has a
+different precedence:
 
- 1. Command line options (most precedence)
- 2. Settings per-spider
- 3. Project settings module
- 4. Settings set by add-ons
- 5. Default settings per-command
- 6. Default global settings (less precedence)
+ 1. :ref:`Command-line settings <cli-settings>` (highest precedence)
+ 2. :ref:`Spider settings <spider-settings>`
+ 3. :ref:`Project settings <project-settings>`
+ 4. :ref:`Add-on settings <addon-settings>`
+ 5. :ref:`Command-specific default settings <cmd-default-settings>`
+ 6. :ref:`Global default settings <default-settings>` (lowest precedence)
 
-The population of these settings sources is taken care of internally, but a
-manual handling is possible using API calls. See the
-:ref:`topics-api-settings` topic for reference.
+.. _cli-settings:
 
-These mechanisms are described in more detail below.
+1. Command-line settings
+------------------------
 
-1. Command line options
------------------------
+Settings set in the command line have the highest precedence, overriding any
+other settings.
 
-Arguments provided by the command line are the ones that take most precedence,
-overriding any other options. You can explicitly override one (or more)
-settings using the ``-s`` (or ``--set``) command line option.
+You can explicitly override one or more settings using the ``-s`` (or
+``--set``) command-line option.
 
 .. highlight:: sh
 
 Example::
 
-    scrapy crawl myspider -s LOG_FILE=scrapy.log
+    scrapy crawl myspider -s LOG_LEVEL=INFO -s LOG_FILE=scrapy.log
 
-2. Settings per-spider
-----------------------
+.. _spider-settings:
+
+2. Spider settings
+------------------
 
-Spiders (See the :ref:`topics-spiders` chapter for reference) can define their
-own settings that will take precedence and override the project ones. One way
-to do so is by setting their :attr:`~scrapy.Spider.custom_settings` attribute:
+:ref:`Spiders <topics-spiders>` can define their own settings that will take
+precedence and override the project ones.
+
+.. note:: :ref:`Pre-crawler settings <pre-crawler-settings>` cannot be defined
+    per spider, and :ref:`reactor settings <reactor-settings>` should not have
+    a different value per spider when :ref:`running multiple spiders in the
+    same process <run-multiple-spiders>`.
+
+One way to do so is by setting their :attr:`~scrapy.Spider.custom_settings`
+attribute:
 
 .. code-block:: python
 
@@ -83,7 +89,7 @@ to do so is by setting their :attr:`~scrapy.Spider.custom_settings` attribute:
         }
 
 It's often better to implement :meth:`~scrapy.Spider.update_settings` instead,
-and settings set there should use the "spider" priority explicitly:
+and settings set there should use the ``"spider"`` priority explicitly:
 
 .. code-block:: python
 
@@ -121,27 +127,37 @@ arguments <spiderargs>` or other logic:
                 )
             return spider
 
-3. Project settings module
---------------------------
+.. _project-settings:
 
-The project settings module is the standard configuration file for your Scrapy
-project, it's where most of your custom settings will be populated. For a
-standard Scrapy project, this means you'll be adding or changing the settings
-in the ``settings.py`` file created for your project.
+3. Project settings
+-------------------
 
-4. Settings set by add-ons
---------------------------
+Scrapy projects include a settings module, usually a file called
+``settings.py``, where you should populate most settings that apply to all your
+spiders.
+
+.. seealso:: :ref:`topics-settings-module-envvar`
+
+.. _addon-settings:
+
+4. Add-on settings
+------------------
 
 :ref:`Add-ons <topics-addons>` can modify settings. They should do this with
-this priority, though this is not enforced.
+``"addon"`` priority where possible.
 
-5. Default settings per-command
--------------------------------
+.. _cmd-default-settings:
 
-Each :doc:`Scrapy tool </topics/commands>` command can have its own default
-settings, which override the global default settings. Those custom command
-settings are specified in the ``default_settings`` attribute of the command
-class.
+5. Command-specific default settings
+------------------------------------
+
+Each :ref:`Scrapy command <topics-commands>` can have its own default settings,
+which override the :ref:`global default settings <default-settings>`.
+
+Those command-specific default settings are specified in the
+``default_settings`` attribute of each command class.
+
+.. _default-settings:
 
 6. Default global settings
 --------------------------
@@ -234,6 +250,61 @@ example, proper setting names for a fictional robots.txt extension would be
 ``ROBOTSTXT_ENABLED``, ``ROBOTSTXT_OBEY``, ``ROBOTSTXT_CACHEDIR``, etc.
 
 
+Special settings
+================
+
+The following settings work slightly differently than all other settings.
+
+.. _pre-crawler-settings:
+
+Pre-crawler settings
+--------------------
+
+**Pre-crawler settings** are settings used before the
+:class:`~scrapy.crawler.Crawler` object is created.
+
+These settings cannot be :ref:`set from a spider <spider-settings>`.
+
+These settings are :setting:`SPIDER_LOADER_CLASS` and settings used by the
+corresponding :ref:`component <topics-components>`, e.g.
+:setting:`SPIDER_MODULES` and :setting:`SPIDER_LOADER_WARN_ONLY` for the
+default component.
+
+
+.. _reactor-settings:
+
+Reactor settings
+----------------
+
+**Reactor settings** are settings tied to the :doc:`Twisted reactor
+<twisted:core/howto/reactor-basics>`.
+
+These settings can be defined from a spider. However, because only 1 reactor
+can be used per process, these settings cannot use a different value per spider
+when :ref:`running multiple spiders in the same process
+<run-multiple-spiders>`.
+
+In general, if different spiders define different values, the first defined
+value is used. However, if two spiders request a different reactor, an
+exception is raised.
+
+These settings are:
+
+-   :setting:`ASYNCIO_EVENT_LOOP`
+
+-   :setting:`DNS_RESOLVER` and settings used by the corresponding
+    component, e.g. :setting:`DNSCACHE_ENABLED`, :setting:`DNSCACHE_SIZE`
+    and :setting:`DNS_TIMEOUT` for the default one.
+
+-   :setting:`REACTOR_THREADPOOL_MAXSIZE`
+
+-   :setting:`TWISTED_REACTOR`
+
+:setting:`ASYNCIO_EVENT_LOOP` and :setting:`TWISTED_REACTOR` are used upon
+installing the reactor. The rest of the settings are applied when starting
+the reactor.
+
+
 .. _topics-settings-ref:
 
 Built-in settings reference
@@ -418,6 +489,38 @@ This setting also affects :setting:`DOWNLOAD_DELAY` and
 :ref:`topics-autothrottle`: if :setting:`CONCURRENT_REQUESTS_PER_IP`
 is non-zero, download delay is enforced per IP, not per domain.
 
+.. setting:: DEFAULT_DROPITEM_LOG_LEVEL
+
+DEFAULT_DROPITEM_LOG_LEVEL
+--------------------------
+
+Default: ``"WARNING"``
+
+Default :ref:`log level <levels>` of messages about dropped items.
+
+When an item is dropped by raising :exc:`scrapy.exceptions.DropItem` from the
+:func:`process_item` method of an :ref:`item pipeline <topics-item-pipeline>`,
+a message is logged, and by default its log level is the one configured in this
+setting.
+
+You may specify this log level as an integer (e.g. ``20``), as a log level
+constant (e.g. ``logging.INFO``) or as a string with the name of a log level
+constant (e.g. ``"INFO"``).
+
+When writing an item pipeline, you can force a different log level by setting
+:attr:`scrapy.exceptions.DropItem.log_level` in your
+:exc:`scrapy.exceptions.DropItem` exception. For example:
+
+.. code-block:: python
+
+   from scrapy.exceptions import DropItem
+
+
+   class MyPipeline:
+       def process_item(self, item, spider):
+           if not item.get("price"):
+               raise DropItem("Missing price data", log_level="INFO")
+           return item
 
 .. setting:: DEFAULT_ITEM_CLASS
 
@@ -955,15 +1058,79 @@ Default: ``'scrapy.dupefilters.RFPDupeFilter'``
 
 The class used to detect and filter duplicate requests.
 
-The default (``RFPDupeFilter``) filters based on the
+The default, :class:`~scrapy.dupefilters.RFPDupeFilter`, filters based on the
 :setting:`REQUEST_FINGERPRINTER_CLASS` setting.
 
-You can disable filtering of duplicate requests by setting
-:setting:`DUPEFILTER_CLASS` to ``'scrapy.dupefilters.BaseDupeFilter'``.
-Be very careful about this however, because you can get into crawling loops.
-It's usually a better idea to set the ``dont_filter`` parameter to
-``True`` on the specific :class:`~scrapy.Request` that should not be
-filtered.
+To change how duplicates are checked, you can point :setting:`DUPEFILTER_CLASS`
+to a custom subclass of :class:`~scrapy.dupefilters.RFPDupeFilter` that
+overrides its ``__init__`` method to use a :ref:`different request
+fingerprinting class <custom-request-fingerprinter>`. For example:
+
+.. code-block:: python
+
+    from scrapy.dupefilters import RFPDupeFilter
+    from scrapy.utils.request import fingerprint
+
+
+    class CustomRequestFingerprinter:
+        def fingerprint(self, request):
+            return fingerprint(request, include_headers=["X-ID"])
+
+
+    class CustomDupeFilter(RFPDupeFilter):
+
+        def __init__(self, path=None, debug=False, *, fingerprinter=None):
+            super().__init__(
+                path=path, debug=debug, fingerprinter=CustomRequestFingerprinter()
+            )
+
+To disable duplicate request filtering set :setting:`DUPEFILTER_CLASS` to
+``'scrapy.dupefilters.BaseDupeFilter'``. Note that not filtering out duplicate
+requests may cause crawling loops. It is usually better to set
+the ``dont_filter`` parameter to ``True`` on the ``__init__`` method of a
+specific :class:`~scrapy.Request` object that should not be filtered out.
+
+A class assigned to :setting:`DUPEFILTER_CLASS` must implement the following
+interface::
+
+    class MyDupeFilter:
+
+        @classmethod
+        def from_settings(cls, settings):
+            """Returns an instance of this duplicate request filtering class
+            based on the current crawl settings."""
+            return cls()
+
+        def request_seen(self, request):
+            """Returns ``True`` if *request* is a duplicate of another request
+            seen in a previous call to :meth:`request_seen`, or ``False``
+            otherwise."""
+            return False
+
+        def open(self):
+            """Called before the spider opens. It may return a deferred."""
+            pass
+
+        def close(self, reason):
+            """Called before the spider closes. It may return a deferred."""
+            pass
+
+        def log(self, request, spider):
+            """Logs that a request has been filtered out.
+
+            It is called right after a call to :meth:`request_seen` that
+            returns ``True``.
+
+            If :meth:`request_seen` always returns ``False``, such as in the
+            case of :class:`~scrapy.dupefilters.BaseDupeFilter`, this method
+            may be omitted.
+            """
+            pass
+
+.. autoclass:: scrapy.dupefilters.BaseDupeFilter
+
+.. autoclass:: scrapy.dupefilters.RFPDupeFilter
+
 
 .. setting:: DUPEFILTER_DEBUG
 
diff --git a/pyproject.toml b/pyproject.toml
index 7dc1f6ec3..29e26399f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -72,9 +72,9 @@ version = {file = "./scrapy/VERSION"}
 
 [tool.mypy]
 ignore_missing_imports = true
+implicit_reexport = false
 
 # Interface classes are hard to support
-
 [[tool.mypy.overrides]]
 module = "twisted.internet.interfaces"
 follow_imports = "skip"
@@ -92,6 +92,14 @@ follow_imports = "skip"
 module = "scrapy.settings.default_settings"
 ignore_errors = true
 
+[[tool.mypy.overrides]]
+module = "itemadapter"
+implicit_reexport = true
+
+[[tool.mypy.overrides]]
+module = "twisted"
+implicit_reexport = true
+
 [tool.bumpversion]
 current_version = "2.12.0"
 commit = true
@@ -112,13 +120,16 @@ include = ["scrapy/*"]
 omit = ["tests/*"]
 disable_warnings = ["include-ignored"]
 
+[tool.coverage.paths]
+source = [
+    "scrapy",
+    ".tox/**/site-packages/scrapy"
+]
+
 [tool.coverage.report]
 # https://github.com/nedbat/coveragepy/issues/831#issuecomment-517778185
 exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]
 
-[tool.isort]
-profile = "black"
-
 [tool.pylint.MASTER]
 persistent = "no"
 jobs = 1          # >1 hides results
@@ -127,44 +138,34 @@ extension-pkg-allow-list=[
 ]
 
 [tool.pylint."MESSAGES CONTROL"]
+enable = [
+    "useless-suppression",
+]
 disable = [
-    "abstract-method",
-    "arguments-differ",
-    "arguments-renamed",
+    # Ones we want to ignore
     "attribute-defined-outside-init",
     "broad-exception-caught",
     "consider-using-with",
     "cyclic-import",
-    "dangerous-default-value",
     "disallowed-name",
-    "duplicate-code",                            # https://github.com/PyCQA/pylint/issues/214
-    "eval-used",
+    "duplicate-code",  # https://github.com/pylint-dev/pylint/issues/214
     "fixme",
-    "import-error",
     "import-outside-toplevel",
-    "inherit-non-class",
+    "inherit-non-class",  # false positives with create_deprecated_class()
     "invalid-name",
     "invalid-overridden-method",
-    "isinstance-second-argument-not-valid-type",
-    "keyword-arg-before-vararg",
+    "isinstance-second-argument-not-valid-type",  # false positives with create_deprecated_class()
     "line-too-long",
     "logging-format-interpolation",
     "logging-fstring-interpolation",
     "logging-not-lazy",
     "missing-docstring",
     "no-member",
-    "no-method-argument",
-    "no-name-in-module",
-    "no-self-argument",
-    "no-value-for-parameter",                    # https://github.com/pylint-dev/pylint/issues/3268
+    "no-value-for-parameter",  # https://github.com/pylint-dev/pylint/issues/3268
     "not-callable",
-    "pointless-statement",
-    "pointless-string-statement",
     "protected-access",
-    "raise-missing-from",
     "redefined-builtin",
     "redefined-outer-name",
-    "signature-differs",
     "too-few-public-methods",
     "too-many-ancestors",
     "too-many-arguments",
@@ -173,16 +174,26 @@ disable = [
     "too-many-instance-attributes",
     "too-many-lines",
     "too-many-locals",
+    "too-many-positional-arguments",
     "too-many-public-methods",
     "too-many-return-statements",
-    "unbalanced-tuple-unpacking",
-    "unnecessary-dunder-call",
     "unused-argument",
     "unused-import",
     "unused-variable",
-    "used-before-assignment",
-    "useless-return",
+    "useless-return",  # https://github.com/pylint-dev/pylint/issues/6530
     "wrong-import-position",
+
+    # Ones that we may want to address (fix, ignore per-line or move to "don't want to fix")
+    "abstract-method",
+    "arguments-differ",
+    "arguments-renamed",
+    "dangerous-default-value",
+    "keyword-arg-before-vararg",
+    "pointless-statement",
+    "raise-missing-from",
+    "unbalanced-tuple-unpacking",
+    "unnecessary-dunder-call",
+    "used-before-assignment",
 ]
 
 [tool.pytest.ini_options]
@@ -224,8 +235,12 @@ extend-select = [
     "D",
     # flake8-future-annotations
     "FA",
+    # flynt
+    "FLY",
     # refurb
     "FURB",
+    # isort
+    "I",
     # flake8-implicit-str-concat
     "ISC",
     # flake8-logging
@@ -238,6 +253,8 @@ extend-select = [
     "PIE",
     # pylint
     "PL",
+    # flake8-use-pathlib
+    "PTH",
     # flake8-pyi
     "PYI",
     # flake8-quotes
@@ -246,8 +263,12 @@ extend-select = [
     "RET",
     # flake8-raise
     "RSE",
+    # Ruff-specific rules
+    "RUF",
     # flake8-bandit
     "S",
+    # flake8-simplify
+    "SIM",
     # flake8-slots
     "SLOT",
     # flake8-debugger
@@ -262,22 +283,8 @@ extend-select = [
     "YTT",
 ]
 ignore = [
-    # Assigning to `os.environ` doesn't clear the environment.
-    "B003",
-    # Do not use mutable data structures for argument defaults.
-    "B006",
-    # Loop control variable not used within the loop body.
-    "B007",
-    # Do not perform function calls in argument defaults.
-    "B008",
-    # Star-arg unpacking after a keyword argument is strongly discouraged.
-    "B026",
-    # Found useless expression.
-    "B018",
-    # No explicit stacklevel argument found.
-    "B028",
-    # Within an `except` clause, raise exceptions with `raise ... from`
-    "B904",
+    # Ones we want to ignore
+
     # Missing docstring in public module
     "D100",
     # Missing docstring in public class
@@ -324,22 +331,51 @@ ignore = [
     "PLR2004",
     # `for` loop variable overwritten by assignment target
     "PLW2901",
+    # String contains ambiguous {}.
+    "RUF001",
+    # Docstring contains ambiguous {}.
+    "RUF002",
+    # Comment contains ambiguous {}.
+    "RUF003",
+    # Mutable class attributes should be annotated with `typing.ClassVar`
+    "RUF012",
     # Use of `assert` detected; needed for mypy
     "S101",
     # FTP-related functions are being called; https://github.com/scrapy/scrapy/issues/4180
     "S321",
     # Argument default set to insecure SSL protocol
     "S503",
+    # Use a context manager for opening files
+    "SIM115",
+    # Yoda condition detected
+    "SIM300",
+
+    # Ones that we may want to address (fix, ignore per-line or move to "don't want to fix")
+
+    # Assigning to `os.environ` doesn't clear the environment.
+    "B003",
+    # Do not use mutable data structures for argument defaults.
+    "B006",
+    # Loop control variable not used within the loop body.
+    "B007",
+    # Do not perform function calls in argument defaults.
+    "B008",
+    # Found useless expression.
+    "B018",
+    # Star-arg unpacking after a keyword argument is strongly discouraged.
+    "B026",
+    # No explicit stacklevel argument found.
+    "B028",
+    # Within an `except` clause, raise exceptions with `raise ... from`
+    "B904",
+    # Use capitalized environment variable
+    "SIM112",
 ]
 
 [tool.ruff.lint.per-file-ignores]
-# Exclude files that are meant to provide top-level imports
-"scrapy/__init__.py" = ["E402"]
-"scrapy/core/downloader/handlers/http.py" = ["F401"]
-"scrapy/http/__init__.py" = ["F401"]
-"scrapy/linkextractors/__init__.py" = ["E402", "F401"]
-"scrapy/selector/__init__.py" = ["F401"]
-"scrapy/spiders/__init__.py" = ["E402", "F401"]
+# Circular import workarounds
+"scrapy/linkextractors/__init__.py" = ["E402"]
+"scrapy/spiders/__init__.py" = ["E402"]
 
 # Skip bandit in tests
 "tests/**" = ["S"]
diff --git a/scrapy/__init__.py b/scrapy/__init__.py
index c19710a6a..256504c9c 100644
--- a/scrapy/__init__.py
+++ b/scrapy/__init__.py
@@ -13,14 +13,14 @@ from scrapy.selector import Selector
 from scrapy.spiders import Spider
 
 __all__ = [
-    "__version__",
-    "version_info",
-    "Spider",
-    "Request",
+    "Field",
     "FormRequest",
-    "Selector",
     "Item",
-    "Field",
+    "Request",
+    "Selector",
+    "Spider",
+    "__version__",
+    "version_info",
 ]
 
 
diff --git a/scrapy/addons.py b/scrapy/addons.py
index 7a1da3afc..1024d2dcd 100644
--- a/scrapy/addons.py
+++ b/scrapy/addons.py
@@ -9,7 +9,7 @@ from scrapy.utils.misc import build_from_crawler, load_object
 
 if TYPE_CHECKING:
     from scrapy.crawler import Crawler
-    from scrapy.settings import Settings
+    from scrapy.settings import BaseSettings, Settings
 
 
 logger = logging.getLogger(__name__)
@@ -36,7 +36,8 @@ class AddonManager:
             try:
                 addoncls = load_object(clspath)
                 addon = build_from_crawler(addoncls, self.crawler)
-                addon.update_settings(settings)
+                if hasattr(addon, "update_settings"):
+                    addon.update_settings(settings)
                 self.addons.append(addon)
             except NotConfigured as e:
                 if e.args:
@@ -52,3 +53,20 @@ class AddonManager:
             },
             extra={"crawler": self.crawler},
         )
+
+    @classmethod
+    def load_pre_crawler_settings(cls, settings: BaseSettings):
+        """Update early settings that do not require a crawler instance, such as SPIDER_MODULES.
+
+        Similar to the load_settings method, this loads each add-on configured in the
+        ``ADDONS`` setting and calls their 'update_pre_crawler_settings' class method if present.
+        This method doesn't have access to the crawler instance or the addons list.
+
+        :param settings: The :class:`~scrapy.settings.BaseSettings` object from \
+            which to read the early add-on configuration
+        :type settings: :class:`~scrapy.settings.Settings`
+        """
+        for clspath in build_component_list(settings["ADDONS"]):
+            addoncls = load_object(clspath)
+            if hasattr(addoncls, "update_pre_crawler_settings"):
+                addoncls.update_pre_crawler_settings(settings)
diff --git a/scrapy/cmdline.py b/scrapy/cmdline.py
index 9a24871de..b08fd3409 100644
--- a/scrapy/cmdline.py
+++ b/scrapy/cmdline.py
@@ -89,13 +89,16 @@ def _get_commands_dict(
     return cmds
 
 
+def _get_project_only_cmds(settings: BaseSettings) -> set[str]:
+    return set(_get_commands_dict(settings, inproject=True)) - set(
+        _get_commands_dict(settings, inproject=False)
+    )
+
+
 def _pop_command_name(argv: list[str]) -> str | None:
-    i = 0
-    for arg in argv[1:]:
-        if not arg.startswith("-"):
-            del argv[i]
-            return arg
-        i += 1
+    for i in range(1, len(argv)):
+        if not argv[i].startswith("-"):
+            return argv.pop(i)
     return None
 
 
@@ -123,11 +126,25 @@ def _print_commands(settings: BaseSettings, inproject: bool) -> None:
     print('Use "scrapy <command> -h" to see more info about a command')
 
 
+def _print_unknown_command_msg(
+    settings: BaseSettings, cmdname: str, inproject: bool
+) -> None:
+    proj_only_cmds = _get_project_only_cmds(settings)
+    if cmdname in proj_only_cmds and not inproject:
+        cmd_list = ", ".join(sorted(proj_only_cmds))
+        print(
+            f"The {cmdname} command is not available from this location.\n"
+            f"These commands are only available from within a project: {cmd_list}.\n"
+        )
+    else:
+        print(f"Unknown command: {cmdname}\n")
+
+
 def _print_unknown_command(
     settings: BaseSettings, cmdname: str, inproject: bool
 ) -> None:
     _print_header(settings, inproject)
-    print(f"Unknown command: {cmdname}\n")
+    _print_unknown_command_msg(settings, cmdname, inproject)
     print('Use "scrapy" to see available commands')
 
 
diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py
index 86d4cc41c..184bd5ca4 100644
--- a/scrapy/commands/crawl.py
+++ b/scrapy/commands/crawl.py
@@ -39,9 +39,8 @@ class Command(BaseRunSpiderCommand):
         else:
             self.crawler_process.start()
 
-            if (
-                self.crawler_process.bootstrap_failed
-                or hasattr(self.crawler_process, "has_exception")
+            if self.crawler_process.bootstrap_failed or (
+                hasattr(self.crawler_process, "has_exception")
                 and self.crawler_process.has_exception
             ):
                 self.exitcode = 1
diff --git a/scrapy/commands/genspider.py b/scrapy/commands/genspider.py
index d7dc104c2..2a1dea997 100644
--- a/scrapy/commands/genspider.py
+++ b/scrapy/commands/genspider.py
@@ -154,7 +154,7 @@ class Command(ScrapyCommand):
             spiders_dir = Path(spiders_module.__file__).parent.resolve()
         else:
             spiders_module = None
-            spiders_dir = Path(".")
+            spiders_dir = Path()
         spider_file = f"{spiders_dir / module}.py"
         shutil.copyfile(template_file, spider_file)
         render_templatefile(spider_file, **tvars)
diff --git a/scrapy/commands/parse.py b/scrapy/commands/parse.py
index cc5c1350b..61aea3ee4 100644
--- a/scrapy/commands/parse.py
+++ b/scrapy/commands/parse.py
@@ -174,13 +174,12 @@ class Command(BaseRunSpiderCommand):
         display.pprint([ItemAdapter(x).asdict() for x in items], colorize=colour)
 
     def print_requests(self, lvl: int | None = None, colour: bool = True) -> None:
-        if lvl is None:
-            if self.requests:
-                requests = self.requests[max(self.requests)]
-            else:
-                requests = []
-        else:
+        if lvl is not None:
             requests = self.requests.get(lvl, [])
+        elif self.requests:
+            requests = self.requests[max(self.requests)]
+        else:
+            requests = []
 
         print("# Requests ", "-" * 65)
         display.pprint(requests, colorize=colour)
@@ -269,7 +268,7 @@ class Command(BaseRunSpiderCommand):
         assert self.crawler_process
         assert self.spidercls
         self.crawler_process.crawl(self.spidercls, **opts.spargs)
-        self.pcrawler = list(self.crawler_process.crawlers)[0]
+        self.pcrawler = next(iter(self.crawler_process.crawlers))
         self.crawler_process.start()
 
         if not self.first_response:
diff --git a/scrapy/commands/runspider.py b/scrapy/commands/runspider.py
index bf8e41020..357ca8b37 100644
--- a/scrapy/commands/runspider.py
+++ b/scrapy/commands/runspider.py
@@ -20,7 +20,7 @@ def _import_file(filepath: str | PathLike[str]) -> ModuleType:
     if abspath.suffix not in (".py", ".pyw"):
         raise ValueError(f"Not a Python source file: {abspath}")
     dirname = str(abspath.parent)
-    sys.path = [dirname] + sys.path
+    sys.path = [dirname, *sys.path]
     try:
         module = import_module(abspath.stem)
     finally:
diff --git a/scrapy/commands/startproject.py b/scrapy/commands/startproject.py
index 5cb73f0d2..1adc1530f 100644
--- a/scrapy/commands/startproject.py
+++ b/scrapy/commands/startproject.py
@@ -1,6 +1,5 @@
 from __future__ import annotations
 
-import os
 import re
 import string
 from importlib.util import find_spec
@@ -28,9 +27,9 @@ TEMPLATES_TO_RENDER: tuple[tuple[str, ...], ...] = (
 IGNORE = ignore_patterns("*.pyc", "__pycache__", ".svn")
 
 
-def _make_writable(path: str | os.PathLike) -> None:
-    current_permissions = os.stat(path).st_mode
-    os.chmod(path, current_permissions | OWNER_WRITE_PERMISSION)
+def _make_writable(path: Path) -> None:
+    current_permissions = path.stat().st_mode
+    path.chmod(current_permissions | OWNER_WRITE_PERMISSION)
 
 
 class Command(ScrapyCommand):
@@ -96,10 +95,7 @@ class Command(ScrapyCommand):
 
         project_name = args[0]
 
-        if len(args) == 2:
-            project_dir = Path(args[1])
-        else:
-            project_dir = Path(args[0])
+        project_dir = Path(args[-1])
 
         if (project_dir / "scrapy.cfg").exists():
             self.exitcode = 1
diff --git a/scrapy/contracts/__init__.py b/scrapy/contracts/__init__.py
index 3b4f932a0..bdb68c4ad 100644
--- a/scrapy/contracts/__init__.py
+++ b/scrapy/contracts/__init__.py
@@ -199,7 +199,7 @@ def _create_testcase(method: Callable, desc: str) -> TestCase:
     spider = method.__self__.name  # type: ignore[attr-defined]
 
     class ContractTestCase(TestCase):
-        def __str__(_self) -> str:
+        def __str__(_self) -> str:  # pylint: disable=no-self-argument
             return f"[{spider}] {method.__name__} ({desc})"
 
     name = f"{spider}_{method.__name__}"
diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py
index 598659b4d..0ad10baff 100644
--- a/scrapy/core/downloader/handlers/ftp.py
+++ b/scrapy/core/downloader/handlers/ftp.py
@@ -32,6 +32,7 @@ from __future__ import annotations
 
 import re
 from io import BytesIO
+from pathlib import Path
 from typing import TYPE_CHECKING, Any, BinaryIO
 from urllib.parse import unquote
 
@@ -56,9 +57,11 @@ if TYPE_CHECKING:
 
 
 class ReceivedDataProtocol(Protocol):
-    def __init__(self, filename: str | None = None):
-        self.__filename: str | None = filename
-        self.body: BinaryIO = open(filename, "wb") if filename else BytesIO()
+    def __init__(self, filename: bytes | None = None):
+        self.__filename: bytes | None = filename
+        self.body: BinaryIO = (
+            Path(filename.decode()).open("wb") if filename else BytesIO()
+        )
         self.size: int = 0
 
     def dataReceived(self, data: bytes) -> None:
@@ -66,7 +69,7 @@ class ReceivedDataProtocol(Protocol):
         self.size += len(data)
 
     @property
-    def filename(self) -> str | None:
+    def filename(self) -> bytes | None:
         return self.__filename
 
     def close(self) -> None:
@@ -128,8 +131,8 @@ class FTPDownloadHandler:
     ) -> Response:
         self.result = result
         protocol.close()
-        headers = {"local filename": protocol.filename or "", "size": protocol.size}
-        body = to_bytes(protocol.filename or protocol.body.read())
+        headers = {"local filename": protocol.filename or b"", "size": protocol.size}
+        body = protocol.filename or protocol.body.read()
         respcls = responsetypes.from_args(url=request.url, body=body)
         # hints for Headers-related types may need to be fixed to not use AnyStr
         return respcls(url=request.url, status=200, body=body, headers=headers)  # type: ignore[arg-type]
diff --git a/scrapy/core/downloader/handlers/http.py b/scrapy/core/downloader/handlers/http.py
index 52535bd8b..93b96c779 100644
--- a/scrapy/core/downloader/handlers/http.py
+++ b/scrapy/core/downloader/handlers/http.py
@@ -2,3 +2,8 @@ from scrapy.core.downloader.handlers.http10 import HTTP10DownloadHandler
 from scrapy.core.downloader.handlers.http11 import (
     HTTP11DownloadHandler as HTTPDownloadHandler,
 )
+
+__all__ = [
+    "HTTP10DownloadHandler",
+    "HTTPDownloadHandler",
+]
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
index bd3200e9f..aa8a1a2a4 100644
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -17,9 +17,14 @@ from twisted.internet.endpoints import TCP4ClientEndpoint
 from twisted.internet.error import TimeoutError
 from twisted.internet.protocol import Factory, Protocol, connectionDone
 from twisted.python.failure import Failure
-from twisted.web.client import URI, Agent, HTTPConnectionPool
+from twisted.web.client import (
+    URI,
+    Agent,
+    HTTPConnectionPool,
+    ResponseDone,
+    ResponseFailed,
+)
 from twisted.web.client import Response as TxResponse
-from twisted.web.client import ResponseDone, ResponseFailed
 from twisted.web.http import PotentialDataLoss, _DataLoss
 from twisted.web.http_headers import Headers as TxHeaders
 from twisted.web.iweb import UNKNOWN_LENGTH, IBodyProducer, IPolicyForHTTPS
@@ -419,10 +424,7 @@ class ScrapyAgent:
         headers = TxHeaders(request.headers)
         if isinstance(agent, self._TunnelingAgent):
             headers.removeHeader(b"Proxy-Authorization")
-        if request.body:
-            bodyproducer = _RequestBodyProducer(request.body)
-        else:
-            bodyproducer = None
+        bodyproducer = _RequestBodyProducer(request.body) if request.body else None
         start_time = time()
         d: Deferred[TxResponse] = agent.request(
             method, to_bytes(url, encoding="ascii"), headers, bodyproducer
diff --git a/scrapy/core/engine.py b/scrapy/core/engine.py
index 5480df72c..61f444e31 100644
--- a/scrapy/core/engine.py
+++ b/scrapy/core/engine.py
@@ -291,9 +291,7 @@ class ExecutionEngine:
             return False
         if self.slot.start_requests is not None:  # not all start requests are handled
             return False
-        if self.slot.scheduler.has_pending_requests():
-            return False
-        return True
+        return not self.slot.scheduler.has_pending_requests()
 
     def crawl(self, request: Request) -> None:
         """Inject the request into the spider <-> downloader pipeline"""
@@ -388,9 +386,8 @@ class ExecutionEngine:
         )
         self.slot = Slot(start_requests, close_if_idle, nextcall, scheduler)
         self.spider = spider
-        if hasattr(scheduler, "open"):
-            if d := scheduler.open(spider):
-                yield d
+        if hasattr(scheduler, "open") and (d := scheduler.open(spider)):
+            yield d
         yield self.scraper.open_spider(spider)
         assert self.crawler.stats
         self.crawler.stats.open_spider(spider)
diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py
index fcc94879a..4bb143dfd 100644
--- a/scrapy/core/scheduler.py
+++ b/scrapy/core/scheduler.py
@@ -115,7 +115,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
     @abstractmethod
     def next_request(self) -> Request | None:
         """
-        Return the next :class:`~scrapy.http.Request` to be processed, or ``None``
+        Return the next :class:`~scrapy.Request` to be processed, or ``None``
         to indicate that there are no requests to be considered ready at the moment.
 
         Returning ``None`` implies that no request from the scheduler will be sent
@@ -263,7 +263,7 @@ class Scheduler(BaseScheduler):
 
     def next_request(self) -> Request | None:
         """
-        Return a :class:`~scrapy.http.Request` object from the memory queue,
+        Return a :class:`~scrapy.Request` object from the memory queue,
         falling back to the disk queue if the memory queue is empty.
         Return ``None`` if there are no more enqueued requests.
 
diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py
index a63ee40bf..4b2520aa1 100644
--- a/scrapy/core/spidermw.py
+++ b/scrapy/core/spidermw.py
@@ -198,10 +198,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
         # chain, they went through it already from the process_spider_exception method
         recovered: MutableChain[_T] | MutableAsyncChain[_T]
         last_result_is_async = isinstance(result, AsyncIterable)
-        if last_result_is_async:
-            recovered = MutableAsyncChain()
-        else:
-            recovered = MutableChain()
+        recovered = MutableAsyncChain() if last_result_is_async else MutableChain()
 
         # There are three cases for the middleware: def foo, async def foo, def foo + async def foo_async.
         # 1. def foo. Sync iterables are passed as is, async ones are downgraded.
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index 05af1bf8a..1aa68cb00 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import contextlib
 import logging
 import pprint
 import signal
@@ -43,7 +44,7 @@ if TYPE_CHECKING:
     from scrapy.logformatter import LogFormatter
     from scrapy.spiderloader import SpiderLoader
     from scrapy.statscollectors import StatsCollector
-    from scrapy.utils.request import RequestFingerprinter
+    from scrapy.utils.request import RequestFingerprinterProtocol
 
 
 logger = logging.getLogger(__name__)
@@ -79,7 +80,7 @@ class Crawler:
         self.extensions: ExtensionManager | None = None
         self.stats: StatsCollector | None = None
         self.logformatter: LogFormatter | None = None
-        self.request_fingerprinter: RequestFingerprinter | None = None
+        self.request_fingerprinter: RequestFingerprinterProtocol | None = None
         self.spider: Spider | None = None
         self.engine: ExecutionEngine | None = None
 
@@ -291,6 +292,7 @@ class CrawlerRunner:
     def __init__(self, settings: dict[str, Any] | Settings | None = None):
         if isinstance(settings, dict) or settings is None:
             settings = Settings(settings)
+        AddonManager.load_pre_crawler_settings(settings)
         self.settings: Settings = settings
         self.spider_loader: SpiderLoader = self._get_spider_loader(settings)
         self._crawlers: set[Crawler] = set()
@@ -503,7 +505,6 @@ class CrawlerProcess(CrawlerRunner):
     def _stop_reactor(self, _: Any = None) -> None:
         from twisted.internet import reactor
 
-        try:
+        # raised if already stopped or in shutdown stage
+        with contextlib.suppress(RuntimeError):
             reactor.stop()
-        except RuntimeError:  # raised if already stopped or in shutdown stage
-            pass
diff --git a/scrapy/downloadermiddlewares/cookies.py b/scrapy/downloadermiddlewares/cookies.py
index 545dcaac9..43348f632 100644
--- a/scrapy/downloadermiddlewares/cookies.py
+++ b/scrapy/downloadermiddlewares/cookies.py
@@ -131,25 +131,26 @@ class CookiesMiddleware:
         decoded = {}
         flags = set()
         for key in ("name", "value", "path", "domain"):
-            if cookie.get(key) is None:
+            value = cookie.get(key)
+            if value is None:
                 if key in ("name", "value"):
                     msg = f"Invalid cookie found in request {request}: {cookie} ('{key}' is missing)"
                     logger.warning(msg)
                     return None
                 continue
-            # https://github.com/python/mypy/issues/7178, https://github.com/python/mypy/issues/9168
-            if isinstance(cookie[key], (bool, float, int, str)):  # type: ignore[literal-required]
-                decoded[key] = str(cookie[key])  # type: ignore[literal-required]
+            if isinstance(value, (bool, float, int, str)):
+                decoded[key] = str(value)
             else:
+                assert isinstance(value, bytes)
                 try:
-                    decoded[key] = cookie[key].decode("utf8")  # type: ignore[literal-required]
+                    decoded[key] = value.decode("utf8")
                 except UnicodeDecodeError:
                     logger.warning(
                         "Non UTF-8 encoded cookie found in request %s: %s",
                         request,
                         cookie,
                     )
-                    decoded[key] = cookie[key].decode("latin1", errors="replace")  # type: ignore[literal-required]
+                    decoded[key] = value.decode("latin1", errors="replace")
         for flag in ("secure",):
             value = cookie.get(flag, _UNSET)
             if value is _UNSET or not value:
diff --git a/scrapy/downloadermiddlewares/httpauth.py b/scrapy/downloadermiddlewares/httpauth.py
index b74140ee1..80107261b 100644
--- a/scrapy/downloadermiddlewares/httpauth.py
+++ b/scrapy/downloadermiddlewares/httpauth.py
@@ -42,7 +42,10 @@ class HttpAuthMiddleware:
         self, request: Request, spider: Spider
     ) -> Request | Response | None:
         auth = getattr(self, "auth", None)
-        if auth and b"Authorization" not in request.headers:
-            if not self.domain or url_is_from_any_domain(request.url, [self.domain]):
-                request.headers[b"Authorization"] = auth
+        if (
+            auth
+            and b"Authorization" not in request.headers
+            and (not self.domain or url_is_from_any_domain(request.url, [self.domain]))
+        ):
+            request.headers[b"Authorization"] = auth
         return None
diff --git a/scrapy/downloadermiddlewares/httpproxy.py b/scrapy/downloadermiddlewares/httpproxy.py
index 2f3f2db47..cb7fa8c90 100644
--- a/scrapy/downloadermiddlewares/httpproxy.py
+++ b/scrapy/downloadermiddlewares/httpproxy.py
@@ -51,10 +51,7 @@ class HttpProxyMiddleware:
         proxy_type, user, password, hostport = _parse_proxy(url)
         proxy_url = urlunparse((proxy_type or orig_type, hostport, "", "", "", ""))
 
-        if user:
-            creds = self._basic_auth_header(user, password)
-        else:
-            creds = None
+        creds = self._basic_auth_header(user, password) if user else None
 
         return creds, proxy_url
 
diff --git a/scrapy/downloadermiddlewares/offsite.py b/scrapy/downloadermiddlewares/offsite.py
index a69f531a7..a2cff65e7 100644
--- a/scrapy/downloadermiddlewares/offsite.py
+++ b/scrapy/downloadermiddlewares/offsite.py
@@ -40,7 +40,11 @@ class OffsiteMiddleware:
         self.process_request(request, spider)
 
     def process_request(self, request: Request, spider: Spider) -> None:
-        if request.dont_filter or self.should_follow(request, spider):
+        if (
+            request.dont_filter
+            or request.meta.get("allow_offsite")
+            or self.should_follow(request, spider)
+        ):
             return
         domain = urlparse_cached(request).hostname
         if domain and domain not in self.domains_seen:
diff --git a/scrapy/downloadermiddlewares/redirect.py b/scrapy/downloadermiddlewares/redirect.py
index 0b883b43a..612426371 100644
--- a/scrapy/downloadermiddlewares/redirect.py
+++ b/scrapy/downloadermiddlewares/redirect.py
@@ -101,12 +101,14 @@ class BaseRedirectMiddleware:
         if ttl and redirects <= self.max_redirect_times:
             redirected.meta["redirect_times"] = redirects
             redirected.meta["redirect_ttl"] = ttl - 1
-            redirected.meta["redirect_urls"] = request.meta.get("redirect_urls", []) + [
-                request.url
+            redirected.meta["redirect_urls"] = [
+                *request.meta.get("redirect_urls", []),
+                request.url,
+            ]
+            redirected.meta["redirect_reasons"] = [
+                *request.meta.get("redirect_reasons", []),
+                reason,
             ]
-            redirected.meta["redirect_reasons"] = request.meta.get(
-                "redirect_reasons", []
-            ) + [reason]
             redirected.dont_filter = request.dont_filter
             redirected.priority = request.priority + self.priority_adjust
             logger.debug(
diff --git a/scrapy/dupefilters.py b/scrapy/dupefilters.py
index caf69daf4..a3e2c5eb4 100644
--- a/scrapy/dupefilters.py
+++ b/scrapy/dupefilters.py
@@ -4,6 +4,7 @@ import logging
 import warnings
 from pathlib import Path
 from typing import TYPE_CHECKING
+from warnings import warn
 
 from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.utils.job import job_dir
@@ -26,6 +27,9 @@ if TYPE_CHECKING:
 
 
 class BaseDupeFilter:
+    """Dummy duplicate request filtering class (:setting:`DUPEFILTER_CLASS`)
+    that does not filter out any request."""
+
     @classmethod
     def from_settings(cls, settings: BaseSettings) -> Self:
         warnings.warn(
@@ -50,10 +54,19 @@ class BaseDupeFilter:
 
     def log(self, request: Request, spider: Spider) -> None:
         """Log that a request has been filtered"""
+        warn(
+            "Calling BaseDupeFilter.log() is deprecated.",
+            ScrapyDeprecationWarning,
+            stacklevel=2,
+        )
 
 
 class RFPDupeFilter(BaseDupeFilter):
-    """Request Fingerprint duplicates filter"""
+    """Duplicate request filtering class (:setting:`DUPEFILTER_CLASS`) that
+    filters out requests with the canonical
+    (:func:`w3lib.url.canonicalize_url`) :attr:`~scrapy.http.Request.url`,
+    :attr:`~scrapy.http.Request.method` and :attr:`~scrapy.http.Request.body`.
+    """
 
     def __init__(
         self,
@@ -117,6 +130,7 @@ class RFPDupeFilter(BaseDupeFilter):
         return False
 
     def request_fingerprint(self, request: Request) -> str:
+        """Returns a string that uniquely identifies the specified request."""
         return self.fingerprinter.fingerprint(request).hex()
 
     def close(self, reason: str) -> None:
diff --git a/scrapy/exceptions.py b/scrapy/exceptions.py
index 96566ba86..f37f881a7 100644
--- a/scrapy/exceptions.py
+++ b/scrapy/exceptions.py
@@ -5,6 +5,8 @@ These exceptions are documented in docs/topics/exceptions.rst. Please don't add
 new exceptions here without documenting them there.
 """
 
+from __future__ import annotations
+
 from typing import Any
 
 # Internal
@@ -58,6 +60,10 @@ class StopDownload(Exception):
 class DropItem(Exception):
     """Drop item from the item pipeline"""
 
+    def __init__(self, message: str, log_level: str | None = None):
+        super().__init__(message)
+        self.log_level = log_level
+
 
 class NotSupported(Exception):
     """Indicates a feature or method is not supported"""
diff --git a/scrapy/exporters.py b/scrapy/exporters.py
index cdb7ac159..46c6aa3fa 100644
--- a/scrapy/exporters.py
+++ b/scrapy/exporters.py
@@ -25,13 +25,13 @@ if TYPE_CHECKING:
 
 __all__ = [
     "BaseItemExporter",
-    "PprintItemExporter",
-    "PickleItemExporter",
     "CsvItemExporter",
-    "XmlItemExporter",
-    "JsonLinesItemExporter",
     "JsonItemExporter",
+    "JsonLinesItemExporter",
     "MarshalItemExporter",
+    "PickleItemExporter",
+    "PprintItemExporter",
+    "XmlItemExporter",
 ]
 
 
@@ -81,10 +81,7 @@ class BaseItemExporter:
             include_empty = self.export_empty_fields
 
         if self.fields_to_export is None:
-            if include_empty:
-                field_iter = item.field_names()
-            else:
-                field_iter = item.keys()
+            field_iter = item.field_names() if include_empty else item.keys()
         elif isinstance(self.fields_to_export, Mapping):
             if include_empty:
                 field_iter = self.fields_to_export.items()
diff --git a/scrapy/extensions/debug.py b/scrapy/extensions/debug.py
index 6948c394c..afaf81928 100644
--- a/scrapy/extensions/debug.py
+++ b/scrapy/extensions/debug.py
@@ -6,6 +6,7 @@ See documentation in docs/topics/extensions.rst
 
 from __future__ import annotations
 
+import contextlib
 import logging
 import signal
 import sys
@@ -69,12 +70,10 @@ class StackTraceDump:
 
 class Debugger:
     def __init__(self) -> None:
-        try:
+        # win32 platforms don't support SIGUSR signals
+        with contextlib.suppress(AttributeError):
             signal.signal(signal.SIGUSR2, self._enter_debugger)  # type: ignore[attr-defined]
-        except AttributeError:
-            # win32 platforms don't support SIGUSR signals
-            pass
 
     def _enter_debugger(self, signum: int, frame: FrameType | None) -> None:
         assert frame
-        Pdb().set_trace(frame.f_back)  # noqa: T100
+        Pdb().set_trace(frame.f_back)
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
index b6e6f55a6..c6e2aa0dd 100644
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -6,6 +6,7 @@ See documentation in docs/topics/feed-exports.rst
 
 from __future__ import annotations
 
+import contextlib
 import logging
 import re
 import sys
@@ -109,6 +110,8 @@ class ItemFilter:
 class IFeedStorage(Interface):
     """Interface that all Feed Storages must implement"""
 
+    # pylint: disable=no-self-argument
+
     def __init__(uri, *, feed_options=None):  # pylint: disable=super-init-not-called
         """Initialize the storage with the parameters given in the URI and the
         feed-specific options (see :setting:`FEEDS`)"""
@@ -642,10 +645,8 @@ class FeedExporter:
         )
         d = {}
         for k, v in conf.items():
-            try:
+            with contextlib.suppress(NotConfigured):
                 d[k] = load_object(v)
-            except NotConfigured:
-                pass
         return d
 
     def _exporter_supported(self, format: str) -> bool:
diff --git a/scrapy/extensions/httpcache.py b/scrapy/extensions/httpcache.py
index 929807de8..0cd16d737 100644
--- a/scrapy/extensions/httpcache.py
+++ b/scrapy/extensions/httpcache.py
@@ -29,7 +29,7 @@ if TYPE_CHECKING:
     from scrapy.http.request import Request
     from scrapy.settings import BaseSettings
     from scrapy.spiders import Spider
-    from scrapy.utils.request import RequestFingerprinter
+    from scrapy.utils.request import RequestFingerprinterProtocol
 
 
 logger = logging.getLogger(__name__)
@@ -89,10 +89,7 @@ class RFC2616Policy:
             return False
         cc = self._parse_cachecontrol(request)
         # obey user-agent directive "Cache-Control: no-store"
-        if b"no-store" in cc:
-            return False
-        # Any other is eligible for caching
-        return True
+        return b"no-store" not in cc
 
     def should_cache_response(self, response: Response, request: Request) -> bool:
         # What is cacheable - https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1
@@ -268,7 +265,9 @@ class DbmCacheStorage:
         )
 
         assert spider.crawler.request_fingerprinter
-        self._fingerprinter: RequestFingerprinter = spider.crawler.request_fingerprinter
+        self._fingerprinter: RequestFingerprinterProtocol = (
+            spider.crawler.request_fingerprinter
+        )
 
     def close_spider(self, spider: Spider) -> None:
         self.db.close()
diff --git a/scrapy/extensions/logstats.py b/scrapy/extensions/logstats.py
index e829d8b92..f2e1f57b8 100644
--- a/scrapy/extensions/logstats.py
+++ b/scrapy/extensions/logstats.py
@@ -83,12 +83,15 @@ class LogStats:
         self, spider: Spider
     ) -> tuple[None, None] | tuple[float, float]:
         start_time = self.stats.get_value("start_time")
-        finished_time = self.stats.get_value("finished_time")
+        finish_time = self.stats.get_value("finish_time")
 
-        if not start_time or not finished_time:
+        if not start_time or not finish_time:
             return None, None
 
-        mins_elapsed = (finished_time - start_time).seconds / 60
+        mins_elapsed = (finish_time - start_time).seconds / 60
+
+        if mins_elapsed == 0:
+            return None, None
 
         items = self.stats.get_value("item_scraped_count", 0)
         pages = self.stats.get_value("response_received_count", 0)
diff --git a/scrapy/extensions/periodic_log.py b/scrapy/extensions/periodic_log.py
index 7cf08a1bb..f97577442 100644
--- a/scrapy/extensions/periodic_log.py
+++ b/scrapy/extensions/periodic_log.py
@@ -151,10 +151,7 @@ class PeriodicLog:
                 return False
         if exclude and not include:
             return True
-        for p in include:
-            if p in stat_name:
-                return True
-        return False
+        return any(p in stat_name for p in include)
 
     def spider_closed(self, spider: Spider, reason: str) -> None:
         self.log()
diff --git a/scrapy/extensions/telnet.py b/scrapy/extensions/telnet.py
index 89c83d20d..189b1953b 100644
--- a/scrapy/extensions/telnet.py
+++ b/scrapy/extensions/telnet.py
@@ -75,7 +75,7 @@ class TelnetConsole(protocol.ServerFactory):
     def stop_listening(self) -> None:
         self.port.stopListening()
 
-    def protocol(self) -> telnet.TelnetTransport:  # type: ignore[override]
+    def protocol(self) -> telnet.TelnetTransport:
         # these import twisted.internet.reactor
         from twisted.conch import manhole, telnet
         from twisted.conch.insults import insults
@@ -84,7 +84,9 @@ class TelnetConsole(protocol.ServerFactory):
             """An implementation of IPortal"""
 
             @defers
-            def login(self_, credentials, mind, *interfaces):
+            def login(
+                self_, credentials, mind, *interfaces
+            ):  # pylint: disable=no-self-argument
                 if not (
                     credentials.username == self.username.encode("utf8")
                     and credentials.checkPassword(self.password.encode("utf8"))
diff --git a/scrapy/http/__init__.py b/scrapy/http/__init__.py
index d0b726bad..0e5c2b53b 100644
--- a/scrapy/http/__init__.py
+++ b/scrapy/http/__init__.py
@@ -15,3 +15,16 @@ from scrapy.http.response.html import HtmlResponse
 from scrapy.http.response.json import JsonResponse
 from scrapy.http.response.text import TextResponse
 from scrapy.http.response.xml import XmlResponse
+
+__all__ = [
+    "FormRequest",
+    "Headers",
+    "HtmlResponse",
+    "JsonRequest",
+    "JsonResponse",
+    "Request",
+    "Response",
+    "TextResponse",
+    "XmlResponse",
+    "XmlRpcRequest",
+]
diff --git a/scrapy/http/cookies.py b/scrapy/http/cookies.py
index 56941ad51..b7c3b9d37 100644
--- a/scrapy/http/cookies.py
+++ b/scrapy/http/cookies.py
@@ -2,9 +2,8 @@ from __future__ import annotations
 
 import re
 import time
-from http.cookiejar import Cookie
+from http.cookiejar import Cookie, CookiePolicy, DefaultCookiePolicy
 from http.cookiejar import CookieJar as _CookieJar
-from http.cookiejar import CookiePolicy, DefaultCookiePolicy
 from typing import TYPE_CHECKING, Any, cast
 
 from scrapy.utils.httpobj import urlparse_cached
@@ -65,9 +64,8 @@ class CookieJar:
                 cookies += self.jar._cookies_for_domain(host, wreq)  # type: ignore[attr-defined]
 
         attrs = self.jar._cookie_attrs(cookies)  # type: ignore[attr-defined]
-        if attrs:
-            if not wreq.has_header("Cookie"):
-                wreq.add_unredirected_header("Cookie", "; ".join(attrs))
+        if attrs and not wreq.has_header("Cookie"):
+            wreq.add_unredirected_header("Cookie", "; ".join(attrs))
 
         self.processed += 1
         if self.processed % self.check_expired_frequency == 0:
diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py
index a96a215f4..4eee5ffbb 100644
--- a/scrapy/http/request/__init__.py
+++ b/scrapy/http/request/__init__.py
@@ -44,10 +44,10 @@ if TYPE_CHECKING:
 
 
 class VerboseCookie(TypedDict):
-    name: str
-    value: str
-    domain: NotRequired[str]
-    path: NotRequired[str]
+    name: str | bytes
+    value: str | bytes | bool | float | int
+    domain: NotRequired[str | bytes]
+    path: NotRequired[str | bytes]
     secure: NotRequired[bool]
 
 
@@ -59,7 +59,7 @@ RequestTypeVar = TypeVar("RequestTypeVar", bound="Request")
 
 def NO_CALLBACK(*args: Any, **kwargs: Any) -> NoReturn:
     """When assigned to the ``callback`` parameter of
-    :class:`~scrapy.http.Request`, it indicates that the request is not meant
+    :class:`~scrapy.Request`, it indicates that the request is not meant
     to have a spider callback at all.
 
     For example:
@@ -83,7 +83,7 @@ def NO_CALLBACK(*args: Any, **kwargs: Any) -> NoReturn:
 
 class Request(object_ref):
     """Represents an HTTP request, which is usually generated in a Spider and
-    executed by the Downloader, thus generating a :class:`Response`.
+    executed by the Downloader, thus generating a :class:`~scrapy.http.Response`.
     """
 
     attributes: tuple[str, ...] = (
@@ -103,9 +103,9 @@ class Request(object_ref):
     )
     """A tuple of :class:`str` objects containing the name of all public
     attributes of the class that are also keyword parameters of the
-    ``__init__`` method.
+    ``__init__()`` method.
 
-    Currently used by :meth:`Request.replace`, :meth:`Request.to_dict` and
+    Currently used by :meth:`.Request.replace`, :meth:`.Request.to_dict` and
     :func:`~scrapy.utils.request.request_from_dict`.
     """
 
@@ -233,7 +233,7 @@ class Request(object_ref):
         finding unknown options call this method by passing
         ``ignore_unknown_options=False``.
 
-        .. caution:: Using :meth:`from_curl` from :class:`~scrapy.http.Request`
+        .. caution:: Using :meth:`from_curl` from :class:`~scrapy.Request`
                      subclasses, such as :class:`~scrapy.http.JsonRequest`, or
                      :class:`~scrapy.http.XmlRpcRequest`, as well as having
                      :ref:`downloader middlewares <topics-downloader-middleware>`
@@ -244,7 +244,7 @@ class Request(object_ref):
                      :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`,
                      or
                      :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
-                     may modify the :class:`~scrapy.http.Request` object.
+                     may modify the :class:`~scrapy.Request` object.
 
         To translate a cURL command into a Scrapy request,
         you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
diff --git a/scrapy/http/request/json_request.py b/scrapy/http/request/json_request.py
index 289c60591..e26cbe05b 100644
--- a/scrapy/http/request/json_request.py
+++ b/scrapy/http/request/json_request.py
@@ -20,7 +20,7 @@ if TYPE_CHECKING:
 
 
 class JsonRequest(Request):
-    attributes: tuple[str, ...] = Request.attributes + ("dumps_kwargs",)
+    attributes: tuple[str, ...] = (*Request.attributes, "dumps_kwargs")
 
     def __init__(
         self, *args: Any, dumps_kwargs: dict[str, Any] | None = None, **kwargs: Any
@@ -29,7 +29,7 @@ class JsonRequest(Request):
         dumps_kwargs.setdefault("sort_keys", True)
         self._dumps_kwargs: dict[str, Any] = dumps_kwargs
 
-        body_passed = kwargs.get("body", None) is not None
+        body_passed = kwargs.get("body") is not None
         data: Any = kwargs.pop("data", None)
         data_passed: bool = data is not None
 
@@ -61,7 +61,7 @@ class JsonRequest(Request):
     def replace(
         self, *args: Any, cls: type[Request] | None = None, **kwargs: Any
     ) -> Request:
-        body_passed = kwargs.get("body", None) is not None
+        body_passed = kwargs.get("body") is not None
         data: Any = kwargs.pop("data", None)
         data_passed: bool = data is not None
 
diff --git a/scrapy/http/response/__init__.py b/scrapy/http/response/__init__.py
index 387805f57..b84110b29 100644
--- a/scrapy/http/response/__init__.py
+++ b/scrapy/http/response/__init__.py
@@ -51,7 +51,7 @@ class Response(object_ref):
     )
     """A tuple of :class:`str` objects containing the name of all public
     attributes of the class that are also keyword parameters of the
-    ``__init__`` method.
+    ``__init__()`` method.
 
     Currently used by :meth:`Response.replace`.
     """
@@ -199,8 +199,8 @@ class Response(object_ref):
     ) -> Request:
         """
         Return a :class:`~.Request` instance to follow a link ``url``.
-        It accepts the same arguments as ``Request.__init__`` method,
-        but ``url`` can be a relative URL or a ``scrapy.link.Link`` object,
+        It accepts the same arguments as ``Request.__init__()`` method,
+        but ``url`` can be a relative URL or a :class:`~scrapy.link.Link` object,
         not only an absolute URL.
 
         :class:`~.TextResponse` provides a :meth:`~.TextResponse.follow`
@@ -254,7 +254,7 @@ class Response(object_ref):
         .. versionadded:: 2.0
 
         Return an iterable of :class:`~.Request` instances to follow all links
-        in ``urls``. It accepts the same arguments as ``Request.__init__`` method,
+        in ``urls``. It accepts the same arguments as ``Request.__init__()`` method,
         but elements of ``urls`` can be relative URLs or :class:`~scrapy.link.Link` objects,
         not only absolute URLs.
 
diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py
index f954b5e9e..081223882 100644
--- a/scrapy/http/response/text.py
+++ b/scrapy/http/response/text.py
@@ -43,7 +43,7 @@ class TextResponse(Response):
     _DEFAULT_ENCODING = "ascii"
     _cached_decoded_json = _NONE
 
-    attributes: tuple[str, ...] = Response.attributes + ("encoding",)
+    attributes: tuple[str, ...] = (*Response.attributes, "encoding")
 
     def __init__(self, *args: Any, **kwargs: Any):
         self._encoding: str | None = kwargs.pop("encoding", None)
@@ -185,15 +185,15 @@ class TextResponse(Response):
     ) -> Request:
         """
         Return a :class:`~.Request` instance to follow a link ``url``.
-        It accepts the same arguments as ``Request.__init__`` method,
+        It accepts the same arguments as ``Request.__init__()`` method,
         but ``url`` can be not only an absolute URL, but also
 
         * a relative URL
         * a :class:`~scrapy.link.Link` object, e.g. the result of
           :ref:`topics-link-extractors`
-        * a :class:`~scrapy.selector.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
+        * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
           ``response.css('a.my_link')[0]``
-        * an attribute :class:`~scrapy.selector.Selector` (not SelectorList), e.g.
+        * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
           ``response.css('a::attr(href)')[0]`` or
           ``response.xpath('//img/@src')[0]``
 
@@ -241,20 +241,20 @@ class TextResponse(Response):
         """
         A generator that produces :class:`~.Request` instances to follow all
         links in ``urls``. It accepts the same arguments as the :class:`~.Request`'s
-        ``__init__`` method, except that each ``urls`` element does not need to be
+        ``__init__()`` method, except that each ``urls`` element does not need to be
         an absolute URL, it can be any of the following:
 
         * a relative URL
         * a :class:`~scrapy.link.Link` object, e.g. the result of
           :ref:`topics-link-extractors`
-        * a :class:`~scrapy.selector.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
+        * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
           ``response.css('a.my_link')[0]``
-        * an attribute :class:`~scrapy.selector.Selector` (not SelectorList), e.g.
+        * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
           ``response.css('a::attr(href)')[0]`` or
           ``response.xpath('//img/@src')[0]``
 
         In addition, ``css`` and ``xpath`` arguments are accepted to perform the link extraction
-        within the ``follow_all`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
+        within the ``follow_all()`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
 
         Note that when passing a ``SelectorList`` as argument for the ``urls`` parameter or
         using the ``css`` or ``xpath`` parameters, this method will not produce requests for
diff --git a/scrapy/interfaces.py b/scrapy/interfaces.py
index 9a2c5f170..b4f1d9394 100644
--- a/scrapy/interfaces.py
+++ b/scrapy/interfaces.py
@@ -1,3 +1,5 @@
+# pylint: disable=no-method-argument,no-self-argument
+
 from zope.interface import Interface
 
 
diff --git a/scrapy/link.py b/scrapy/link.py
index 1a569f892..9c272ab2f 100644
--- a/scrapy/link.py
+++ b/scrapy/link.py
@@ -24,7 +24,7 @@ class Link:
                     of the anchor tag.
     """
 
-    __slots__ = ["url", "text", "fragment", "nofollow"]
+    __slots__ = ["fragment", "nofollow", "text", "url"]
 
     def __init__(
         self, url: str, text: str = "", fragment: str = "", nofollow: bool = False
diff --git a/scrapy/linkextractors/__init__.py b/scrapy/linkextractors/__init__.py
index 1c7e96ae0..b39859f7b 100644
--- a/scrapy/linkextractors/__init__.py
+++ b/scrapy/linkextractors/__init__.py
@@ -126,3 +126,8 @@ def _is_valid_url(url: str) -> bool:
 
 # Top-level imports
 from scrapy.linkextractors.lxmlhtml import LxmlLinkExtractor as LinkExtractor
+
+__all__ = [
+    "IGNORED_EXTENSIONS",
+    "LinkExtractor",
+]
diff --git a/scrapy/linkextractors/lxmlhtml.py b/scrapy/linkextractors/lxmlhtml.py
index f195dbdd7..4fd932b88 100644
--- a/scrapy/linkextractors/lxmlhtml.py
+++ b/scrapy/linkextractors/lxmlhtml.py
@@ -41,9 +41,12 @@ _collect_string_content = etree.XPath("string()")
 
 
 def _nons(tag: Any) -> Any:
-    if isinstance(tag, str):
-        if tag[0] == "{" and tag[1 : len(XHTML_NAMESPACE) + 1] == XHTML_NAMESPACE:
-            return tag.split("}")[-1]
+    if (
+        isinstance(tag, str)
+        and tag[0] == "{"
+        and tag[1 : len(XHTML_NAMESPACE) + 1] == XHTML_NAMESPACE
+    ):
+        return tag.split("}")[-1]
     return tag
 
 
@@ -230,9 +233,7 @@ class LxmlLinkExtractor:
             parsed_url, self.deny_extensions
         ):
             return False
-        if self.restrict_text and not _matches(link.text, self.restrict_text):
-            return False
-        return True
+        return not self.restrict_text or _matches(link.text, self.restrict_text)
 
     def matches(self, url: str) -> bool:
         if self.allow_domains and not url_is_from_any_domain(url, self.allow_domains):
diff --git a/scrapy/loader/__init__.py b/scrapy/loader/__init__.py
index d35720a45..2f5c0343b 100644
--- a/scrapy/loader/__init__.py
+++ b/scrapy/loader/__init__.py
@@ -32,7 +32,7 @@ class ItemLoader(itemloaders.ItemLoader):
     :param selector: The selector to extract data from, when using the
         :meth:`add_xpath`, :meth:`add_css`, :meth:`replace_xpath`, or
         :meth:`replace_css` method.
-    :type selector: :class:`~scrapy.selector.Selector` object
+    :type selector: :class:`~scrapy.Selector` object
 
     :param response: The response used to construct the selector using the
         :attr:`default_selector_class`, unless the selector argument is given,
@@ -79,7 +79,7 @@ class ItemLoader(itemloaders.ItemLoader):
 
     .. attribute:: selector
 
-        The :class:`~scrapy.selector.Selector` object to extract data from.
+        The :class:`~scrapy.Selector` object to extract data from.
         It's either the selector given in the ``__init__`` method or one created from
         the response given in the ``__init__`` method using the
         :attr:`default_selector_class`. This attribute is meant to be
diff --git a/scrapy/logformatter.py b/scrapy/logformatter.py
index 544f4adfe..f10e91beb 100644
--- a/scrapy/logformatter.py
+++ b/scrapy/logformatter.py
@@ -76,8 +76,8 @@ class LogFormatter:
         self, request: Request, response: Response, spider: Spider
     ) -> LogFormatterResult:
         """Logs a message when the crawler finds a webpage."""
-        request_flags = f" {str(request.flags)}" if request.flags else ""
-        response_flags = f" {str(response.flags)}" if response.flags else ""
+        request_flags = f" {request.flags!s}" if request.flags else ""
+        response_flags = f" {response.flags!s}" if response.flags else ""
         return {
             "level": logging.DEBUG,
             "msg": CRAWLEDMSG,
@@ -120,8 +120,12 @@ class LogFormatter:
         spider: Spider,
     ) -> LogFormatterResult:
         """Logs a message when an item is dropped while it is passing through the item pipeline."""
+        if (level := getattr(exception, "log_level", None)) is None:
+            level = spider.crawler.settings["DEFAULT_DROPITEM_LOG_LEVEL"]
+        if isinstance(level, str):
+            level = getattr(logging, level)
         return {
-            "level": logging.WARNING,
+            "level": level,
             "msg": DROPPEDMSG,
             "args": {
                 "exception": exception,
diff --git a/scrapy/mail.py b/scrapy/mail.py
index a3c642401..be2423965 100644
--- a/scrapy/mail.py
+++ b/scrapy/mail.py
@@ -111,11 +111,9 @@ class MailSender:
     ) -> Deferred[None] | None:
         from twisted.internet import reactor
 
-        msg: MIMEBase
-        if attachs:
-            msg = MIMEMultipart()
-        else:
-            msg = MIMENonMultipart(*mimetype.split("/", 1))
+        msg: MIMEBase = (
+            MIMEMultipart() if attachs else MIMENonMultipart(*mimetype.split("/", 1))
+        )
 
         to = list(arg_to_iter(to))
         cc = list(arg_to_iter(cc))
diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py
index 16bd45c00..a10117590 100644
--- a/scrapy/pipelines/files.py
+++ b/scrapy/pipelines/files.py
@@ -553,10 +553,8 @@ class FilesPipeline(MediaPipeline):
         ftp_store.USE_ACTIVE_MODE = settings.getbool("FEED_STORAGE_FTP_ACTIVE")
 
     def _get_store(self, uri: str) -> FilesStoreProtocol:
-        if Path(uri).is_absolute():  # to support win32 paths like: C:\\some\dir
-            scheme = "file"
-        else:
-            scheme = urlparse(uri).scheme
+        # to support win32 paths like: C:\\some\dir
+        scheme = "file" if Path(uri).is_absolute() else urlparse(uri).scheme
         store_cls = self.STORE_SCHEMES[scheme]
         return store_cls(uri)
 
diff --git a/scrapy/pipelines/media.py b/scrapy/pipelines/media.py
index 5438b8522..e66b86ce6 100644
--- a/scrapy/pipelines/media.py
+++ b/scrapy/pipelines/media.py
@@ -30,7 +30,7 @@ if TYPE_CHECKING:
     from scrapy import Spider
     from scrapy.crawler import Crawler
     from scrapy.http import Response
-    from scrapy.utils.request import RequestFingerprinter
+    from scrapy.utils.request import RequestFingerprinterProtocol
 
 
 class FileInfo(TypedDict):
@@ -47,7 +47,7 @@ logger = logging.getLogger(__name__)
 
 class MediaPipeline(ABC):
     crawler: Crawler
-    _fingerprinter: RequestFingerprinter
+    _fingerprinter: RequestFingerprinterProtocol
     _modern_init = False
 
     LOG_FAILED_RESULTS: bool = True
@@ -127,8 +127,7 @@ class MediaPipeline(ABC):
         if (
             not base_class_name
             or class_name == base_class_name
-            or settings
-            and not settings.get(formatted_key)
+            or (settings and not settings.get(formatted_key))
         ):
             return key
         return formatted_key
diff --git a/scrapy/pqueues.py b/scrapy/pqueues.py
index 5b2f81335..a04e0107b 100644
--- a/scrapy/pqueues.py
+++ b/scrapy/pqueues.py
@@ -34,7 +34,7 @@ def _path_safe(text: str) -> str:
     # as we replace some letters we can get collision for different slots
     # add we add unique part
     unique_slot = hashlib.md5(text.encode("utf8")).hexdigest()  # noqa: S324
-    return "-".join([pathable_slot, unique_slot])
+    return f"{pathable_slot}-{unique_slot}"
 
 
 class QueueProtocol(Protocol):
diff --git a/scrapy/selector/__init__.py b/scrapy/selector/__init__.py
index 85c500d66..7cfa3c364 100644
--- a/scrapy/selector/__init__.py
+++ b/scrapy/selector/__init__.py
@@ -4,3 +4,8 @@ Selectors
 
 # top-level imports
 from scrapy.selector.unified import Selector, SelectorList
+
+__all__ = [
+    "Selector",
+    "SelectorList",
+]
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 0bbde118e..7ef365f68 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -49,6 +49,8 @@ CONCURRENT_REQUESTS_PER_IP = 0
 COOKIES_ENABLED = True
 COOKIES_DEBUG = False
 
+DEFAULT_DROPITEM_LOG_LEVEL = "WARNING"
+
 DEFAULT_ITEM_CLASS = "scrapy.item.Item"
 
 DEFAULT_REQUEST_HEADERS = {
diff --git a/scrapy/shell.py b/scrapy/shell.py
index 5d0ab1e4d..5e5e57a9a 100644
--- a/scrapy/shell.py
+++ b/scrapy/shell.py
@@ -6,6 +6,7 @@ See documentation in docs/topics/shell.rst
 
 from __future__ import annotations
 
+import contextlib
 import os
 import signal
 from typing import TYPE_CHECKING, Any
@@ -70,17 +71,16 @@ class Shell:
         else:
             self.populate_vars()
         if self.code:
+            # pylint: disable-next=eval-used
             print(eval(self.code, globals(), self.vars))  # noqa: S307
         else:
-            """
-            Detect interactive shell setting in scrapy.cfg
-            e.g.: ~/.config/scrapy.cfg or ~/.scrapy.cfg
-            [settings]
-            # shell can be one of ipython, bpython or python;
-            # to be used as the interactive python console, if available.
-            # (default is ipython, fallbacks in the order listed above)
-            shell = python
-            """
+            # Detect interactive shell setting in scrapy.cfg
+            # e.g.: ~/.config/scrapy.cfg or ~/.scrapy.cfg
+            # [settings]
+            # # shell can be one of ipython, bpython or python;
+            # # to be used as the interactive python console, if available.
+            # # (default is ipython, fallbacks in the order listed above)
+            # shell = python
             cfg = get_config()
             section, option = "settings", "shell"
             env = os.environ.get("SCRAPY_PYTHON_SHELL")
@@ -143,12 +143,10 @@ class Shell:
             else:
                 request.meta["handle_httpstatus_all"] = True
         response = None
-        try:
+        with contextlib.suppress(IgnoreRequest):
             response, spider = threads.blockingCallFromThread(
                 reactor, self._schedule, request, spider
             )
-        except IgnoreRequest:
-            pass
         self.populate_vars(response, request, spider)
 
     def populate_vars(
diff --git a/scrapy/spidermiddlewares/offsite.py b/scrapy/spidermiddlewares/offsite.py
index d3ed64ef5..95e753830 100644
--- a/scrapy/spidermiddlewares/offsite.py
+++ b/scrapy/spidermiddlewares/offsite.py
@@ -61,7 +61,11 @@ class OffsiteMiddleware:
     def _filter(self, request: Any, spider: Spider) -> bool:
         if not isinstance(request, Request):
             return True
-        if request.dont_filter or self.should_follow(request, spider):
+        if (
+            request.dont_filter
+            or request.meta.get("allow_offsite")
+            or self.should_follow(request, spider)
+        ):
             return True
         domain = urlparse_cached(request).hostname
         if domain and domain not in self.domains_seen:
diff --git a/scrapy/spidermiddlewares/referer.py b/scrapy/spidermiddlewares/referer.py
index 93b7fcf17..a3a1e5b92 100644
--- a/scrapy/spidermiddlewares/referer.py
+++ b/scrapy/spidermiddlewares/referer.py
@@ -195,8 +195,7 @@ class StrictOriginPolicy(ReferrerPolicy):
         if (
             self.tls_protected(response_url)
             and self.potentially_trustworthy(request_url)
-            or not self.tls_protected(response_url)
-        ):
+        ) or not self.tls_protected(response_url):
             return self.origin_referrer(response_url)
         return None
 
@@ -249,8 +248,7 @@ class StrictOriginWhenCrossOriginPolicy(ReferrerPolicy):
         if (
             self.tls_protected(response_url)
             and self.potentially_trustworthy(request_url)
-            or not self.tls_protected(response_url)
-        ):
+        ) or not self.tls_protected(response_url):
             return self.origin_referrer(response_url)
         return None
 
@@ -282,7 +280,7 @@ class DefaultReferrerPolicy(NoReferrerWhenDowngradePolicy):
     using ``file://`` or ``s3://`` scheme.
     """
 
-    NOREFERRER_SCHEMES: tuple[str, ...] = LOCAL_SCHEMES + ("file", "s3")
+    NOREFERRER_SCHEMES: tuple[str, ...] = (*LOCAL_SCHEMES, "file", "s3")
     name: str = POLICY_SCRAPY_DEFAULT
 
 
@@ -362,11 +360,10 @@ class RefererMiddleware:
         - otherwise, the policy from settings is used.
         """
         policy_name = request.meta.get("referrer_policy")
-        if policy_name is None:
-            if isinstance(resp_or_url, Response):
-                policy_header = resp_or_url.headers.get("Referrer-Policy")
-                if policy_header is not None:
-                    policy_name = to_unicode(policy_header.decode("latin1"))
+        if policy_name is None and isinstance(resp_or_url, Response):
+            policy_header = resp_or_url.headers.get("Referrer-Policy")
+            if policy_header is not None:
+                policy_name = to_unicode(policy_header.decode("latin1"))
         if policy_name is None:
             return self.default_policy()
 
diff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.py
index 6136dabc7..e255e91cc 100644
--- a/scrapy/spiders/__init__.py
+++ b/scrapy/spiders/__init__.py
@@ -117,3 +117,12 @@ class Spider(object_ref):
 from scrapy.spiders.crawl import CrawlSpider, Rule
 from scrapy.spiders.feed import CSVFeedSpider, XMLFeedSpider
 from scrapy.spiders.sitemap import SitemapSpider
+
+__all__ = [
+    "CSVFeedSpider",
+    "CrawlSpider",
+    "Rule",
+    "SitemapSpider",
+    "Spider",
+    "XMLFeedSpider",
+]
diff --git a/scrapy/utils/_compression.py b/scrapy/utils/_compression.py
index 591737b8e..6b09f36ff 100644
--- a/scrapy/utils/_compression.py
+++ b/scrapy/utils/_compression.py
@@ -1,3 +1,4 @@
+import contextlib
 import zlib
 from io import BytesIO
 from warnings import warn
@@ -37,10 +38,8 @@ else:
             return decompressor.process(data)
 
 
-try:
+with contextlib.suppress(ImportError):
     import zstandard
-except ImportError:
-    pass
 
 
 _CHUNK_SIZE = 65536  # 64 KiB
diff --git a/scrapy/utils/console.py b/scrapy/utils/console.py
index 95844a48c..7425543ff 100644
--- a/scrapy/utils/console.py
+++ b/scrapy/utils/console.py
@@ -59,7 +59,7 @@ def _embed_ptpython_shell(
     namespace: dict[str, Any] = {}, banner: str = ""
 ) -> EmbedFuncT:
     """Start a ptpython shell"""
-    import ptpython.repl
+    import ptpython.repl  # pylint: disable=import-error
 
     @wraps(_embed_ptpython_shell)
     def wrapper(namespace: dict[str, Any] = namespace, banner: str = "") -> None:
diff --git a/scrapy/utils/datatypes.py b/scrapy/utils/datatypes.py
index 98ecb2f02..3d0e0d3c7 100644
--- a/scrapy/utils/datatypes.py
+++ b/scrapy/utils/datatypes.py
@@ -8,6 +8,7 @@ This module must not depend on any module outside the Standard Library.
 from __future__ import annotations
 
 import collections
+import contextlib
 import warnings
 import weakref
 from collections import OrderedDict
@@ -173,10 +174,9 @@ class LocalWeakReferencedCache(weakref.WeakKeyDictionary):
         self.data: LocalCache = LocalCache(limit=limit)
 
     def __setitem__(self, key: _KT, value: _VT) -> None:
-        try:
+        # if raised, key is not weak-referenceable, skip caching
+        with contextlib.suppress(TypeError):
             super().__setitem__(key, value)
-        except TypeError:
-            pass  # key is not weak-referenceable, skip caching
 
     def __getitem__(self, key: _KT) -> _VT | None:  # type: ignore[override]
         try:
diff --git a/scrapy/utils/deprecate.py b/scrapy/utils/deprecate.py
index 0a0acc742..20d03cae6 100644
--- a/scrapy/utils/deprecate.py
+++ b/scrapy/utils/deprecate.py
@@ -57,6 +57,7 @@ def create_deprecated_class(
 
     # https://github.com/python/mypy/issues/4177
     class DeprecatedClass(new_class.__class__):  # type: ignore[misc, name-defined]
+        # pylint: disable=no-self-argument
         deprecated_class: type | None = None
         warned_on_subclass: bool = False
 
diff --git a/scrapy/utils/display.py b/scrapy/utils/display.py
index 39f46270b..20744a604 100644
--- a/scrapy/utils/display.py
+++ b/scrapy/utils/display.py
@@ -30,6 +30,7 @@ def _tty_supports_color() -> bool:
 
 
 def _colorize(text: str, colorize: bool = True) -> str:
+    # pylint: disable=no-name-in-module
     if not colorize or not sys.stdout.isatty() or not _tty_supports_color():
         return text
     try:
diff --git a/scrapy/utils/engine.py b/scrapy/utils/engine.py
index 1948009e8..52f29e22c 100644
--- a/scrapy/utils/engine.py
+++ b/scrapy/utils/engine.py
@@ -32,7 +32,7 @@ def get_engine_status(engine: ExecutionEngine) -> list[tuple[str, Any]]:
     checks: list[tuple[str, Any]] = []
     for test in tests:
         try:
-            checks += [(test, eval(test))]  # noqa: S307
+            checks += [(test, eval(test))]  # noqa: S307  # pylint: disable=eval-used
         except Exception as e:
             checks += [(test, f"{type(e).__name__} (exception)")]
 
diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py
index 5ce4863f6..d319e7950 100644
--- a/scrapy/utils/misc.py
+++ b/scrapy/utils/misc.py
@@ -252,7 +252,9 @@ def is_generator_with_return_value(callable: Callable[..., Any]) -> bool:
 
     def returns_none(return_node: ast.Return) -> bool:
         value = return_node.value
-        return value is None or isinstance(value, ast.Constant) and value.value is None
+        return value is None or (
+            isinstance(value, ast.Constant) and value.value is None
+        )
 
     if inspect.isgeneratorfunction(callable):
         func = callable
diff --git a/scrapy/utils/reactor.py b/scrapy/utils/reactor.py
index 66a06a9f0..679e38206 100644
--- a/scrapy/utils/reactor.py
+++ b/scrapy/utils/reactor.py
@@ -100,7 +100,7 @@ def install_reactor(reactor_path: str, event_loop_path: str | None = None) -> No
             asyncioreactor.install(eventloop=event_loop)
     else:
         *module, _ = reactor_path.split(".")
-        installer_path = module + ["install"]
+        installer_path = [*module, "install"]
         installer = load_object(".".join(installer_path))
         with suppress(error.ReactorAlreadyInstalledError):
             installer()
diff --git a/scrapy/utils/request.py b/scrapy/utils/request.py
index ad811e804..9c1161968 100644
--- a/scrapy/utils/request.py
+++ b/scrapy/utils/request.py
@@ -1,6 +1,6 @@
 """
 This module provides some useful functions for working with
-scrapy.http.Request objects
+scrapy.Request objects
 """
 
 from __future__ import annotations
@@ -109,12 +109,10 @@ class RequestFingerprinter:
 
     It takes into account a canonical version
     (:func:`w3lib.url.canonicalize_url`) of :attr:`request.url
-    <scrapy.http.Request.url>` and the values of :attr:`request.method
-    <scrapy.http.Request.method>` and :attr:`request.body
-    <scrapy.http.Request.body>`. It then generates an `SHA1
+    <scrapy.Request.url>` and the values of :attr:`request.method
+    <scrapy.Request.method>` and :attr:`request.body
+    <scrapy.Request.body>`. It then generates an `SHA1
     <https://en.wikipedia.org/wiki/SHA-1>`_ hash.
-
-    .. seealso:: :setting:`REQUEST_FINGERPRINTER_IMPLEMENTATION`.
     """
 
     @classmethod
@@ -229,7 +227,8 @@ def request_to_curl(request: Request) -> str:
             cookies = f"--cookie '{cookie}'"
         elif isinstance(request.cookies, list):
             cookie = "; ".join(
-                f"{list(c.keys())[0]}={list(c.values())[0]}" for c in request.cookies
+                f"{next(iter(c.keys()))}={next(iter(c.values()))}"
+                for c in request.cookies
             )
             cookies = f"--cookie '{cookie}'"
 
diff --git a/scrapy/utils/signal.py b/scrapy/utils/signal.py
index c1d3bfffb..5fd176a3f 100644
--- a/scrapy/utils/signal.py
+++ b/scrapy/utils/signal.py
@@ -36,7 +36,7 @@ def send_catch_log(
     dont_log = named.pop("dont_log", ())
     dont_log = tuple(dont_log) if isinstance(dont_log, Sequence) else (dont_log,)
     dont_log += (StopDownload,)
-    spider = named.get("spider", None)
+    spider = named.get("spider")
     responses: list[tuple[TypingAny, TypingAny]] = []
     for receiver in liveReceivers(getAllReceivers(sender, signal)):
         result: TypingAny
@@ -88,7 +88,7 @@ def send_catch_log_deferred(
         return failure
 
     dont_log = named.pop("dont_log", None)
-    spider = named.get("spider", None)
+    spider = named.get("spider")
     dfds: list[Deferred[tuple[TypingAny, TypingAny]]] = []
     for receiver in liveReceivers(getAllReceivers(sender, signal)):
         d: Deferred[TypingAny] = maybeDeferred_coro(
diff --git a/scrapy/utils/sitemap.py b/scrapy/utils/sitemap.py
index b60fe929e..e0d9f4595 100644
--- a/scrapy/utils/sitemap.py
+++ b/scrapy/utils/sitemap.py
@@ -26,13 +26,15 @@ class Sitemap:
         )
         self._root = lxml.etree.fromstring(xmltext, parser=xmlp)  # noqa: S320
         rt = self._root.tag
-        self.type = self._root.tag.split("}", 1)[1] if "}" in rt else rt
+        assert isinstance(rt, str)
+        self.type = rt.split("}", 1)[1] if "}" in rt else rt
 
     def __iter__(self) -> Iterator[dict[str, Any]]:
         for elem in self._root.getchildren():
             d: dict[str, Any] = {}
             for el in elem.getchildren():
                 tag = el.tag
+                assert isinstance(tag, str)
                 name = tag.split("}", 1)[1] if "}" in tag else tag
 
                 if name == "link":
diff --git a/scrapy/utils/url.py b/scrapy/utils/url.py
index 3bf831c26..db2749d79 100644
--- a/scrapy/utils/url.py
+++ b/scrapy/utils/url.py
@@ -51,7 +51,7 @@ def url_is_from_any_domain(url: UrlT, domains: Iterable[str]) -> bool:
 def url_is_from_spider(url: UrlT, spider: type[Spider]) -> bool:
     """Return True if the url belongs to the given spider"""
     return url_is_from_any_domain(
-        url, [spider.name] + list(getattr(spider, "allowed_domains", []))
+        url, [spider.name, *getattr(spider, "allowed_domains", [])]
     )
 
 
@@ -173,13 +173,19 @@ def strip_url(
         parsed_url.username or parsed_url.password
     ):
         netloc = netloc.split("@")[-1]
-    if strip_default_port and parsed_url.port:
-        if (parsed_url.scheme, parsed_url.port) in (
+
+    if (
+        strip_default_port
+        and parsed_url.port
+        and (parsed_url.scheme, parsed_url.port)
+        in (
             ("http", 80),
             ("https", 443),
             ("ftp", 21),
-        ):
-            netloc = netloc.replace(f":{parsed_url.port}", "")
+        )
+    ):
+        netloc = netloc.replace(f":{parsed_url.port}", "")
+
     return urlunparse(
         (
             parsed_url.scheme,
diff --git a/scrapy/utils/versions.py b/scrapy/utils/versions.py
index ff1f9b346..052321ae3 100644
--- a/scrapy/utils/versions.py
+++ b/scrapy/utils/versions.py
@@ -11,7 +11,7 @@ from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.settings.default_settings import LOG_VERSIONS
 from scrapy.utils.ssl import get_openssl_version
 
-_DEFAULT_SOFTWARE = ["Scrapy"] + LOG_VERSIONS
+_DEFAULT_SOFTWARE = ["Scrapy", *LOG_VERSIONS]
 
 
 def _version(item):
diff --git a/tox.ini b/tox.ini
index 4e1a99473..cf5e19a61 100644
--- a/tox.ini
+++ b/tox.ini
@@ -10,11 +10,12 @@ minversion = 1.7.0
 [test-requirements]
 deps =
     attrs
+    coverage >= 7.4.0
     pexpect >= 4.8.0
     pyftpdlib >= 2.0.1
     pygments
     pytest
-    pytest-cov==4.0.0
+    pytest-cov >= 4.0.0
     pytest-xdist
     sybil >= 1.3.0  # https://github.com/cjw296/sybil/issues/20#issuecomment-605433422
     testfixtures
@@ -26,6 +27,8 @@ deps =
 
     # mitmproxy does not support PyPy
     mitmproxy; implementation_name != 'pypy'
+setenv =
+    COVERAGE_CORE=sysmon
 passenv =
     S3_TEST_FILE_URI
     AWS_ACCESS_KEY_ID
@@ -36,19 +39,19 @@ passenv =
 #allow tox virtualenv to upgrade pip/wheel/setuptools
 download = true
 commands =
-    pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests} --doctest-modules
+    pytest --cov-config=pyproject.toml --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 docs scrapy tests} --doctest-modules
 install_command =
     python -I -m pip install -ctests/upper-constraints.txt {opts} {packages}
 
 [testenv:typing]
 basepython = python3
 deps =
-    mypy==1.12.0
+    mypy==1.14.0
     typing-extensions==4.12.2
-    types-lxml==2024.9.16
+    types-lxml==2024.12.13
     types-Pygments==2.18.0.20240506
-    botocore-stubs==1.35.39
-    boto3-stubs[s3]==1.35.39
+    botocore-stubs==1.35.90
+    boto3-stubs[s3]==1.35.90
     attrs >= 18.2.0
     Pillow >= 10.3.0
     pyOpenSSL >= 24.2.1
@@ -77,15 +80,15 @@ commands =
 basepython = python3
 deps =
     {[testenv:extra-deps]deps}
-    pylint==3.2.5
+    pylint==3.3.3
 commands =
     pylint conftest.py docs extras scrapy tests
 
 [testenv:twinecheck]
 basepython = python3
 deps =
-    twine==5.1.1
-    build==1.2.1
+    twine==6.0.1
+    build==1.2.2.post1
 commands =
     python -m build --sdist
     twine check dist/*
@@ -115,7 +118,7 @@ setenv =
 install_command =
     python -I -m pip install {opts} {packages}
 commands =
-    pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 scrapy tests}
+    pytest --cov-config=pyproject.toml --cov=scrapy --cov-report=xml --cov-report= {posargs:--durations=10 scrapy tests}
 
 [testenv:pinned]
 basepython = {[pinned]basepython}
@@ -241,7 +244,7 @@ deps =
     {[testenv]deps}
     botocore>=1.4.87
 commands =
-    pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:tests -m requires_botocore}
+    pytest --cov-config=pyproject.toml --cov=scrapy --cov-report=xml --cov-report= {posargs:tests -m requires_botocore}
 
 [testenv:botocore-pinned]
 basepython = {[pinned]basepython}
@@ -252,4 +255,4 @@ install_command = {[pinned]install_command}
 setenv =
     {[pinned]setenv}
 commands =
-    pytest --cov=scrapy --cov-report=xml --cov-report= {posargs:tests -m requires_botocore}
+    pytest --cov-config=pyproject.toml --cov=scrapy --cov-report=xml --cov-report= {posargs:tests -m requires_botocore}


Test patch:
diff --git a/.github/workflows/tests-macos.yml b/.github/workflows/tests-macos.yml
index 9e78e26e3..c28a99982 100644
--- a/.github/workflows/tests-macos.yml
+++ b/.github/workflows/tests-macos.yml
@@ -27,4 +27,4 @@ jobs:
         tox -e py
 
     - name: Upload coverage report
-      run: bash <(curl -s https://codecov.io/bash)
+      uses: codecov/codecov-action@v5
diff --git a/.github/workflows/tests-ubuntu.yml b/.github/workflows/tests-ubuntu.yml
index b2a5681df..89d1e70ac 100644
--- a/.github/workflows/tests-ubuntu.yml
+++ b/.github/workflows/tests-ubuntu.yml
@@ -73,7 +73,7 @@ jobs:
       if: contains(matrix.python-version, 'pypy') || contains(matrix.env.TOXENV, 'pinned')
       run: |
         sudo apt-get update
-        sudo apt-get install libxml2-dev libxslt-dev
+        sudo apt-get install libxml2-dev libxslt-dev libjpeg-dev
 
     - name: Run tests
       env: ${{ matrix.env }}
@@ -82,4 +82,4 @@ jobs:
         tox
 
     - name: Upload coverage report
-      run: bash <(curl -s https://codecov.io/bash)
+      uses: codecov/codecov-action@v5
diff --git a/.github/workflows/tests-windows.yml b/.github/workflows/tests-windows.yml
index 67a32aac6..45e4ca157 100644
--- a/.github/workflows/tests-windows.yml
+++ b/.github/workflows/tests-windows.yml
@@ -44,3 +44,6 @@ jobs:
       run: |
         pip install -U tox
         tox
+
+    - name: Upload coverage report
+      uses: codecov/codecov-action@v5
diff --git a/scrapy/utils/testproc.py b/scrapy/utils/testproc.py
index 05e04e2d1..85d7c940f 100644
--- a/scrapy/utils/testproc.py
+++ b/scrapy/utils/testproc.py
@@ -17,7 +17,7 @@ if TYPE_CHECKING:
 class ProcessTest:
     command: str | None = None
     prefix = [sys.executable, "-m", "scrapy.cmdline"]
-    cwd = os.getcwd()  # trial chdirs to temp dir
+    cwd = os.getcwd()  # trial chdirs to temp dir  # noqa: PTH109
 
     def execute(
         self,
@@ -31,7 +31,7 @@ class ProcessTest:
         if settings is not None:
             env["SCRAPY_SETTINGS_MODULE"] = settings
         assert self.command
-        cmd = self.prefix + [self.command] + list(args)
+        cmd = [*self.prefix, self.command, *args]
         pp = TestProcessProtocol()
         pp.deferred.addCallback(self._process_finished, cmd, check_code)
         reactor.spawnProcess(pp, cmd[0], cmd, env=env, path=self.cwd)
diff --git a/tests/CrawlerRunner/ip_address.py b/tests/CrawlerRunner/ip_address.py
index 5bf7512bc..2f1bb7713 100644
--- a/tests/CrawlerRunner/ip_address.py
+++ b/tests/CrawlerRunner/ip_address.py
@@ -1,9 +1,8 @@
 from urllib.parse import urlparse
 
 from twisted.internet import reactor
-from twisted.names import cache
+from twisted.names import cache, resolve
 from twisted.names import hosts as hostsModule
-from twisted.names import resolve
 from twisted.names.client import Resolver
 from twisted.python.runtime import platform
 
diff --git a/tests/__init__.py b/tests/__init__.py
index 5f0c0f7ad..cd52ade58 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -15,13 +15,6 @@ os.environ["http_proxy"] = ""
 os.environ["https_proxy"] = ""
 os.environ["ftp_proxy"] = ""
 
-# Absolutize paths to coverage config and output file because tests that
-# spawn subprocesses also changes current working directory.
-_sourceroot = Path(__file__).resolve().parent.parent
-if "COV_CORE_CONFIG" in os.environ:
-    os.environ["COVERAGE_FILE"] = str(_sourceroot / ".coverage")
-    os.environ["COV_CORE_CONFIG"] = str(_sourceroot / os.environ["COV_CORE_CONFIG"])
-
 tests_datadir = str(Path(__file__).parent.resolve() / "sample_data")
 
 
diff --git a/tests/test_addons.py b/tests/test_addons.py
index 17949997c..a0caa3511 100644
--- a/tests/test_addons.py
+++ b/tests/test_addons.py
@@ -166,19 +166,21 @@ class AddonManagerTest(unittest.TestCase):
             def update_settings(self, settings):
                 pass
 
-        with patch("scrapy.addons.logger") as logger_mock:
-            with patch("scrapy.addons.build_from_crawler") as build_from_crawler_mock:
-                settings_dict = {
-                    "ADDONS": {LoggedAddon: 1},
-                }
-                addon = LoggedAddon()
-                build_from_crawler_mock.return_value = addon
-                crawler = get_crawler(settings_dict=settings_dict)
-                logger_mock.info.assert_called_once_with(
-                    "Enabled addons:\n%(addons)s",
-                    {"addons": [addon]},
-                    extra={"crawler": crawler},
-                )
+        with (
+            patch("scrapy.addons.logger") as logger_mock,
+            patch("scrapy.addons.build_from_crawler") as build_from_crawler_mock,
+        ):
+            settings_dict = {
+                "ADDONS": {LoggedAddon: 1},
+            }
+            addon = LoggedAddon()
+            build_from_crawler_mock.return_value = addon
+            crawler = get_crawler(settings_dict=settings_dict)
+            logger_mock.info.assert_called_once_with(
+                "Enabled addons:\n%(addons)s",
+                {"addons": [addon]},
+                extra={"crawler": crawler},
+            )
 
     @inlineCallbacks
     def test_enable_addon_in_spider(self):
diff --git a/tests/test_cmdline/__init__.py b/tests/test_cmdline/__init__.py
index 4835e936b..acd524ea4 100644
--- a/tests/test_cmdline/__init__.py
+++ b/tests/test_cmdline/__init__.py
@@ -21,7 +21,7 @@ class CmdlineTest(unittest.TestCase):
 
     def _execute(self, *new_args, **kwargs):
         encoding = sys.stdout.encoding or "utf-8"
-        args = (sys.executable, "-m", "scrapy.cmdline") + new_args
+        args = (sys.executable, "-m", "scrapy.cmdline", *new_args)
         proc = Popen(args, stdout=PIPE, stderr=PIPE, env=self.env, **kwargs)
         comm = proc.communicate()[0].strip()
         return comm.decode(encoding)
diff --git a/tests/test_commands.py b/tests/test_commands.py
index 32b69de8a..50f093043 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -9,6 +9,7 @@ import re
 import subprocess
 import sys
 from contextlib import contextmanager
+from io import StringIO
 from itertools import chain
 from pathlib import Path
 from shutil import copytree, rmtree
@@ -16,12 +17,13 @@ from stat import S_IWRITE as ANYONE_WRITE_PERMISSION
 from tempfile import TemporaryFile, mkdtemp
 from threading import Timer
 from typing import TYPE_CHECKING
-from unittest import skipIf
+from unittest import mock, skipIf
 
 from pytest import mark
 from twisted.trial import unittest
 
 import scrapy
+from scrapy.cmdline import _pop_command_name, _print_unknown_command_msg
 from scrapy.commands import ScrapyCommand, ScrapyHelpFormatter, view
 from scrapy.commands.startproject import IGNORE
 from scrapy.settings import Settings
@@ -87,13 +89,13 @@ class ProjectTest(unittest.TestCase):
 
     def call(self, *new_args, **kwargs):
         with TemporaryFile() as out:
-            args = (sys.executable, "-m", "scrapy.cmdline") + new_args
+            args = (sys.executable, "-m", "scrapy.cmdline", *new_args)
             return subprocess.call(
                 args, stdout=out, stderr=out, cwd=self.cwd, env=self.env, **kwargs
             )
 
     def proc(self, *new_args, **popen_kwargs):
-        args = (sys.executable, "-m", "scrapy.cmdline") + new_args
+        args = (sys.executable, "-m", "scrapy.cmdline", *new_args)
         p = subprocess.Popen(
             args,
             cwd=popen_kwargs.pop("cwd", self.cwd),
@@ -652,6 +654,24 @@ class MiscCommandsTest(CommandTest):
     def test_list(self):
         self.assertEqual(0, self.call("list"))
 
+    def test_command_not_found(self):
+        na_msg = """
+The list command is not available from this location.
+These commands are only available from within a project: check, crawl, edit, list, parse.
+"""
+        not_found_msg = """
+Unknown command: abc
+"""
+        params = [
+            ("list", 0, na_msg),
+            ("abc", 0, not_found_msg),
+            ("abc", 1, not_found_msg),
+        ]
+        for cmdname, inproject, message in params:
+            with mock.patch("sys.stdout", new=StringIO()) as out:
+                _print_unknown_command_msg(Settings(), cmdname, inproject)
+                self.assertEqual(out.getvalue().strip(), message.strip())
+
 
 class RunSpiderCommandTest(CommandTest):
     spider_filename = "myspider.py"
@@ -1143,3 +1163,29 @@ class HelpMessageTest(CommandTest):
         for command in self.commands:
             _, out, _ = self.proc(command, "-h")
             self.assertIn("Usage", out)
+
+
+class PopCommandNameTest(unittest.TestCase):
+    def test_valid_command(self):
+        argv = ["scrapy", "crawl", "my_spider"]
+        command = _pop_command_name(argv)
+        self.assertEqual(command, "crawl")
+        self.assertEqual(argv, ["scrapy", "my_spider"])
+
+    def test_no_command(self):
+        argv = ["scrapy"]
+        command = _pop_command_name(argv)
+        self.assertIsNone(command)
+        self.assertEqual(argv, ["scrapy"])
+
+    def test_option_before_command(self):
+        argv = ["scrapy", "-h", "crawl"]
+        command = _pop_command_name(argv)
+        self.assertEqual(command, "crawl")
+        self.assertEqual(argv, ["scrapy", "-h"])
+
+    def test_option_after_command(self):
+        argv = ["scrapy", "crawl", "-h"]
+        command = _pop_command_name(argv)
+        self.assertEqual(command, "crawl")
+        self.assertEqual(argv, ["scrapy", "-h"])
diff --git a/tests/test_contracts.py b/tests/test_contracts.py
index 743889234..f7581707b 100644
--- a/tests/test_contracts.py
+++ b/tests/test_contracts.py
@@ -517,8 +517,8 @@ class ContractsManagerTest(unittest.TestCase):
                 super().__init__(*args, **kwargs)
                 self.visited = 0
 
-            def start_requests(s):
-                return self.conman.from_spider(s, self.results)
+            def start_requests(self_):  # pylint: disable=no-self-argument
+                return self.conman.from_spider(self_, self.results)
 
             def parse_first(self, response):
                 self.visited += 1
diff --git a/tests/test_crawler.py b/tests/test_crawler.py
index f3e5ebf5d..6c3fe96b0 100644
--- a/tests/test_crawler.py
+++ b/tests/test_crawler.py
@@ -1,5 +1,4 @@
 import logging
-import os
 import platform
 import re
 import signal
@@ -643,11 +642,10 @@ class CrawlerRunnerHasSpider(unittest.TestCase):
 
 class ScriptRunnerMixin:
     script_dir: Path
-    cwd = os.getcwd()
 
     def get_script_args(self, script_name: str, *script_args: str) -> list[str]:
         script_path = self.script_dir / script_name
-        return [sys.executable, str(script_path)] + list(script_args)
+        return [sys.executable, str(script_path), *script_args]
 
     def run_script(self, script_name: str, *script_args: str) -> str:
         args = self.get_script_args(script_name, *script_args)
diff --git a/tests/test_downloader_handlers.py b/tests/test_downloader_handlers.py
index 8ecba41bf..0dcbeaec1 100644
--- a/tests/test_downloader_handlers.py
+++ b/tests/test_downloader_handlers.py
@@ -116,7 +116,7 @@ class FileTestCase(unittest.TestCase):
 
     def tearDown(self):
         os.close(self.fd)
-        os.remove(self.tmpname)
+        Path(self.tmpname).unlink()
 
     def test_download(self):
         def _test(response):
@@ -530,9 +530,10 @@ class Http11TestCase(HttpTestCase):
         d = self.download_request(request, Spider("foo"))
 
         def checkDataLoss(failure):
-            if failure.check(ResponseFailed):
-                if any(r.check(_DataLoss) for r in failure.value.reasons):
-                    return None
+            if failure.check(ResponseFailed) and any(
+                r.check(_DataLoss) for r in failure.value.reasons
+            ):
+                return None
             return failure
 
         d.addCallback(lambda _: self.fail("No DataLoss exception"))
diff --git a/tests/test_downloadermiddleware_httpcache.py b/tests/test_downloadermiddleware_httpcache.py
index f80eff3e6..ec4e87ffb 100644
--- a/tests/test_downloadermiddleware_httpcache.py
+++ b/tests/test_downloadermiddleware_httpcache.py
@@ -566,7 +566,3 @@ class RFC2616PolicyTest(DefaultStorageTest):
                 res2 = self._process_requestresponse(mw, req0, None)
                 self.assertEqualResponse(res1, res2)
                 assert "cached" in res2.flags
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_downloadermiddleware_offsite.py b/tests/test_downloadermiddleware_offsite.py
index fec56a39f..23a1d06da 100644
--- a/tests/test_downloadermiddleware_offsite.py
+++ b/tests/test_downloadermiddleware_offsite.py
@@ -64,6 +64,37 @@ def test_process_request_dont_filter(value, filtered):
         assert mw.process_request(request, spider) is None
 
 
+@pytest.mark.parametrize(
+    ("allow_offsite", "dont_filter", "filtered"),
+    (
+        (True, UNSET, False),
+        (True, None, False),
+        (True, False, False),
+        (True, True, False),
+        (False, UNSET, True),
+        (False, None, True),
+        (False, False, True),
+        (False, True, False),
+    ),
+)
+def test_process_request_allow_offsite(allow_offsite, dont_filter, filtered):
+    crawler = get_crawler(Spider)
+    spider = crawler._create_spider(name="a", allowed_domains=["a.example"])
+    mw = OffsiteMiddleware.from_crawler(crawler)
+    mw.spider_opened(spider)
+    kwargs = {"meta": {}}
+    if allow_offsite is not UNSET:
+        kwargs["meta"]["allow_offsite"] = allow_offsite
+    if dont_filter is not UNSET:
+        kwargs["dont_filter"] = dont_filter
+    request = Request("https://b.example", **kwargs)
+    if filtered:
+        with pytest.raises(IgnoreRequest):
+            mw.process_request(request, spider)
+    else:
+        assert mw.process_request(request, spider) is None
+
+
 @pytest.mark.parametrize(
     "value",
     (
diff --git a/tests/test_downloadermiddleware_redirect.py b/tests/test_downloadermiddleware_redirect.py
index e37da9715..7b19ab781 100644
--- a/tests/test_downloadermiddleware_redirect.py
+++ b/tests/test_downloadermiddleware_redirect.py
@@ -1314,7 +1314,3 @@ def test_meta_refresh_schemes(url, location, target):
     else:
         assert isinstance(redirect, Request)
         assert redirect.url == target
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_downloadermiddleware_retry.py b/tests/test_downloadermiddleware_retry.py
index c99f19b03..6b9b39413 100644
--- a/tests/test_downloadermiddleware_retry.py
+++ b/tests/test_downloadermiddleware_retry.py
@@ -114,7 +114,7 @@ class RetryTest(unittest.TestCase):
     def test_exception_to_retry_added(self):
         exc = ValueError
         settings_dict = {
-            "RETRY_EXCEPTIONS": list(RETRY_EXCEPTIONS) + [exc],
+            "RETRY_EXCEPTIONS": [*RETRY_EXCEPTIONS, exc],
         }
         crawler = get_crawler(Spider, settings_dict=settings_dict)
         mw = RetryMiddleware.from_crawler(crawler)
@@ -643,7 +643,3 @@ class GetRetryRequestTest(unittest.TestCase):
             f"{stats_key}/reason_count/{expected_reason}",
         ):
             self.assertEqual(spider.crawler.stats.get_value(stat), 1)
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_downloaderslotssettings.py b/tests/test_downloaderslotssettings.py
index 55f9ecac9..879bc8697 100644
--- a/tests/test_downloaderslotssettings.py
+++ b/tests/test_downloaderslotssettings.py
@@ -31,7 +31,7 @@ class DownloaderSlotsSettingsTestSpider(MetaSpider):
     def start_requests(self):
         self.times = {None: []}
 
-        slots = list(self.custom_settings.get("DOWNLOAD_SLOTS", {}).keys()) + [None]
+        slots = [*self.custom_settings.get("DOWNLOAD_SLOTS", {}), None]
 
         for slot in slots:
             url = self.mockserver.url(f"/?downloader_slot={slot}")
diff --git a/tests/test_dupefilters.py b/tests/test_dupefilters.py
index 4fd648f48..703c23529 100644
--- a/tests/test_dupefilters.py
+++ b/tests/test_dupefilters.py
@@ -4,11 +4,13 @@ import sys
 import tempfile
 import unittest
 from pathlib import Path
+from warnings import catch_warnings
 
 from testfixtures import LogCapture
 
 from scrapy.core.scheduler import Scheduler
-from scrapy.dupefilters import RFPDupeFilter
+from scrapy.dupefilters import BaseDupeFilter, RFPDupeFilter
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import Request
 from scrapy.utils.python import to_bytes
 from scrapy.utils.test import get_crawler
@@ -252,3 +254,18 @@ class RFPDupeFilterTest(unittest.TestCase):
             )
 
             dupefilter.close("finished")
+
+
+class BaseDupeFilterTestCase(unittest.TestCase):
+    def test_log_deprecation(self):
+        dupefilter = _get_dupefilter(
+            settings={"DUPEFILTER_CLASS": BaseDupeFilter},
+        )
+        with catch_warnings(record=True) as warning_list:
+            dupefilter.log(None, None)
+        self.assertEqual(len(warning_list), 1)
+        self.assertEqual(
+            str(warning_list[0].message),
+            "Calling BaseDupeFilter.log() is deprecated.",
+        )
+        self.assertEqual(warning_list[0].category, ScrapyDeprecationWarning)
diff --git a/tests/test_exporters.py b/tests/test_exporters.py
index fa9389044..1fbacfdfc 100644
--- a/tests/test_exporters.py
+++ b/tests/test_exporters.py
@@ -116,7 +116,7 @@ class BaseItemExporterTest(unittest.TestCase):
         )
 
         ie = self._get_exporter(fields_to_export=["name"], encoding="latin-1")
-        _, name = list(ie._get_serialized_fields(self.i))[0]
+        _, name = next(iter(ie._get_serialized_fields(self.i)))
         assert isinstance(name, str)
         self.assertEqual(name, "John\xa3")
 
@@ -216,7 +216,9 @@ class PprintItemExporterTest(BaseItemExporterTest):
         return PprintItemExporter(self.output, **kwargs)
 
     def _check_output(self):
-        self._assert_expected_item(eval(self.output.getvalue()))
+        self._assert_expected_item(
+            eval(self.output.getvalue())  # pylint: disable=eval-used
+        )
 
 
 class PprintItemExporterDataclassTest(PprintItemExporterTest):
@@ -699,7 +701,3 @@ class CustomExporterItemTest(unittest.TestCase):
 
 class CustomExporterDataclassTest(CustomExporterItemTest):
     item_class = TestDataClass
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py
index b087aaab1..0f149f172 100644
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@@ -756,7 +756,7 @@ class FeedExportTest(FeedExportTestBase):
                 )
 
         finally:
-            for file_path in FEEDS.keys():
+            for file_path in FEEDS:
                 if not Path(file_path).exists():
                     continue
 
@@ -1229,15 +1229,13 @@ class FeedExportTest(FeedExportTestBase):
 
         class CustomFilter2(scrapy.extensions.feedexport.ItemFilter):
             def accepts(self, item):
-                if "foo" not in item.fields:
-                    return False
-                return True
+                return "foo" in item.fields
 
         class CustomFilter3(scrapy.extensions.feedexport.ItemFilter):
             def accepts(self, item):
-                if isinstance(item, tuple(self.item_classes)) and item["foo"] == "bar1":
-                    return True
-                return False
+                return (
+                    isinstance(item, tuple(self.item_classes)) and item["foo"] == "bar1"
+                )
 
         formats = {
             "json": b'[\n{"foo": "bar1", "egg": "spam1"}\n]',
diff --git a/tests/test_http_request.py b/tests/test_http_request.py
index 9997b7ab3..34d3b25d5 100644
--- a/tests/test_http_request.py
+++ b/tests/test_http_request.py
@@ -1488,10 +1488,7 @@ def _buildresponse(body, **kwargs):
 
 
 def _qs(req, encoding="utf-8", to_unicode=False):
-    if req.method == "POST":
-        qs = req.body
-    else:
-        qs = req.url.partition("?")[2]
+    qs = req.body if req.method == "POST" else req.url.partition("?")[2]
     uqs = unquote_to_bytes(qs)
     if to_unicode:
         uqs = uqs.decode(encoding)
@@ -1717,7 +1714,3 @@ class JsonRequestTest(RequestTest):
     def tearDown(self):
         warnings.resetwarnings()
         super().tearDown()
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_http_response.py b/tests/test_http_response.py
index 679cc8238..0730cff3a 100644
--- a/tests/test_http_response.py
+++ b/tests/test_http_response.py
@@ -960,7 +960,7 @@ class XmlResponseTest(TextResponseTest):
 
 
 class CustomResponse(TextResponse):
-    attributes = TextResponse.attributes + ("foo", "bar")
+    attributes = (*TextResponse.attributes, "foo", "bar")
 
     def __init__(self, *args, **kwargs) -> None:
         self.foo = kwargs.pop("foo", None)
diff --git a/tests/test_item.py b/tests/test_item.py
index 13243b67f..480412841 100644
--- a/tests/test_item.py
+++ b/tests/test_item.py
@@ -1,7 +1,8 @@
 import unittest
+from abc import ABCMeta
 from unittest import mock
 
-from scrapy.item import ABCMeta, Field, Item, ItemMeta
+from scrapy.item import Field, Item, ItemMeta
 
 
 class ItemTest(unittest.TestCase):
@@ -54,7 +55,7 @@ class ItemTest(unittest.TestCase):
 
         self.assertEqual(itemrepr, "{'name': 'John Doe', 'number': 123}")
 
-        i2 = eval(itemrepr)
+        i2 = eval(itemrepr)  # pylint: disable=eval-used
         self.assertEqual(i2["name"], "John Doe")
         self.assertEqual(i2["number"], 123)
 
@@ -296,7 +297,3 @@ class ItemMetaClassCellRegression(unittest.TestCase):
                 # TypeError: __class__ set to <class '__main__.MyItem'>
                 # defining 'MyItem' as <class '__main__.MyItem'>
                 super().__init__(*args, **kwargs)
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_link.py b/tests/test_link.py
index 7ba0851ae..35723bbd6 100644
--- a/tests/test_link.py
+++ b/tests/test_link.py
@@ -49,7 +49,7 @@ class LinkTest(unittest.TestCase):
         l1 = Link(
             "http://www.example.com", text="test", fragment="something", nofollow=True
         )
-        l2 = eval(repr(l1))
+        l2 = eval(repr(l1))  # pylint: disable=eval-used
         self._assert_same_links(l1, l2)
 
     def test_bytes_url(self):
diff --git a/tests/test_loader.py b/tests/test_loader.py
index aca428bbe..824d7aecf 100644
--- a/tests/test_loader.py
+++ b/tests/test_loader.py
@@ -586,7 +586,3 @@ class FunctionProcessorTestCase(unittest.TestCase):
         lo.add_value("foo", "  bar  ")
         lo.add_value("foo", ["  asdf  ", "  qwerty  "])
         self.assertEqual(dict(lo.load_item()), {"foo": ["BAR", "ASDF", "QWERTY"]})
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_loader_deprecated.py b/tests/test_loader_deprecated.py
index 4bf22f6a0..8d4bd6bc1 100644
--- a/tests/test_loader_deprecated.py
+++ b/tests/test_loader_deprecated.py
@@ -715,7 +715,3 @@ class FunctionProcessorTestCase(unittest.TestCase):
         lo.add_value("foo", "  bar  ")
         lo.add_value("foo", ["  asdf  ", "  qwerty  "])
         self.assertEqual(dict(lo.load_item()), {"foo": ["BAR", "ASDF", "QWERTY"]})
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_logformatter.py b/tests/test_logformatter.py
index 5a92521cc..e5d077858 100644
--- a/tests/test_logformatter.py
+++ b/tests/test_logformatter.py
@@ -1,5 +1,7 @@
+import logging
 import unittest
 
+import pytest
 from testfixtures import LogCapture
 from twisted.internet import defer
 from twisted.python.failure import Failure
@@ -26,6 +28,7 @@ class LogFormatterTestCase(unittest.TestCase):
     def setUp(self):
         self.formatter = LogFormatter()
         self.spider = Spider("default")
+        self.spider.crawler = get_crawler()
 
     def test_crawled_with_referer(self):
         req = Request("http://www.example.com")
@@ -68,6 +71,62 @@ class LogFormatterTestCase(unittest.TestCase):
         assert all(isinstance(x, str) for x in lines)
         self.assertEqual(lines, ["Dropped: \u2018", "{}"])
 
+    def test_dropitem_default_log_level(self):
+        item = {}
+        exception = DropItem("Test drop")
+        response = Response("http://www.example.com")
+        spider = Spider("foo")
+        spider.crawler = get_crawler(Spider)
+
+        logkws = self.formatter.dropped(item, exception, response, spider)
+        self.assertEqual(logkws["level"], logging.WARNING)
+
+        spider.crawler.settings.frozen = False
+        spider.crawler.settings["DEFAULT_DROPITEM_LOG_LEVEL"] = logging.INFO
+        spider.crawler.settings.frozen = True
+        logkws = self.formatter.dropped(item, exception, response, spider)
+        self.assertEqual(logkws["level"], logging.INFO)
+
+        spider.crawler.settings.frozen = False
+        spider.crawler.settings["DEFAULT_DROPITEM_LOG_LEVEL"] = "INFO"
+        spider.crawler.settings.frozen = True
+        logkws = self.formatter.dropped(item, exception, response, spider)
+        self.assertEqual(logkws["level"], logging.INFO)
+
+        spider.crawler.settings.frozen = False
+        spider.crawler.settings["DEFAULT_DROPITEM_LOG_LEVEL"] = 10
+        spider.crawler.settings.frozen = True
+        logkws = self.formatter.dropped(item, exception, response, spider)
+        self.assertEqual(logkws["level"], logging.DEBUG)
+
+        spider.crawler.settings.frozen = False
+        spider.crawler.settings["DEFAULT_DROPITEM_LOG_LEVEL"] = 0
+        spider.crawler.settings.frozen = True
+        logkws = self.formatter.dropped(item, exception, response, spider)
+        self.assertEqual(logkws["level"], logging.NOTSET)
+
+        unsupported_value = object()
+        spider.crawler.settings.frozen = False
+        spider.crawler.settings["DEFAULT_DROPITEM_LOG_LEVEL"] = unsupported_value
+        spider.crawler.settings.frozen = True
+        logkws = self.formatter.dropped(item, exception, response, spider)
+        self.assertEqual(logkws["level"], unsupported_value)
+
+        with pytest.raises(TypeError):
+            logging.log(logkws["level"], "message")
+
+    def test_dropitem_custom_log_level(self):
+        item = {}
+        response = Response("http://www.example.com")
+
+        exception = DropItem("Test drop", log_level="INFO")
+        logkws = self.formatter.dropped(item, exception, response, self.spider)
+        self.assertEqual(logkws["level"], logging.INFO)
+
+        exception = DropItem("Test drop", log_level="ERROR")
+        logkws = self.formatter.dropped(item, exception, response, self.spider)
+        self.assertEqual(logkws["level"], logging.ERROR)
+
     def test_item_error(self):
         # In practice, the complete traceback is shown by passing the
         # 'exc_info' argument to the logging function
@@ -145,6 +204,7 @@ class LogformatterSubclassTest(LogFormatterTestCase):
     def setUp(self):
         self.formatter = LogFormatterSubclass()
         self.spider = Spider("default")
+        self.spider.crawler = get_crawler(Spider)
 
     def test_crawled_with_referer(self):
         req = Request("http://www.example.com")
@@ -234,7 +294,3 @@ class ShowOrSkipMessagesTestCase(TwistedTestCase):
         self.assertNotIn("Scraped from <200 http://127.0.0.1:", str(lc))
         self.assertNotIn("Crawled (200) <GET http://127.0.0.1:", str(lc))
         self.assertNotIn("Dropped: Ignoring item", str(lc))
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_logstats.py b/tests/test_logstats.py
index d87285df7..a4b002e34 100644
--- a/tests/test_logstats.py
+++ b/tests/test_logstats.py
@@ -47,7 +47,7 @@ class TestLogStats(unittest.TestCase):
 
         # Simulate when spider closes after running for 30 mins
         self.stats.set_value("start_time", datetime.fromtimestamp(1655100172))
-        self.stats.set_value("finished_time", datetime.fromtimestamp(1655101972))
+        self.stats.set_value("finish_time", datetime.fromtimestamp(1655101972))
         logstats.spider_closed(self.spider, "test reason")
         self.assertEqual(self.stats.get_value("responses_per_minute"), 172.9)
         self.assertEqual(self.stats.get_value("items_per_minute"), 116.4)
@@ -60,3 +60,12 @@ class TestLogStats(unittest.TestCase):
         logstats.spider_closed(self.spider, "test reason")
         self.assertIsNone(self.stats.get_value("responses_per_minute"))
         self.assertIsNone(self.stats.get_value("items_per_minute"))
+
+    def test_stats_calculation_no_elapsed_time(self):
+        """The stat values should be None since the elapsed time is 0."""
+        logstats = LogStats.from_crawler(self.crawler)
+        self.stats.set_value("start_time", datetime.fromtimestamp(1655100172))
+        self.stats.set_value("finish_time", datetime.fromtimestamp(1655100172))
+        logstats.spider_closed(self.spider, "test reason")
+        self.assertIsNone(self.stats.get_value("responses_per_minute"))
+        self.assertIsNone(self.stats.get_value("items_per_minute"))
diff --git a/tests/test_mail.py b/tests/test_mail.py
index c6af2b1b8..cf40c342e 100644
--- a/tests/test_mail.py
+++ b/tests/test_mail.py
@@ -157,7 +157,3 @@ class MailSenderTest(unittest.TestCase):
 
         context = factory.buildProtocol("test@scrapy.org").context
         self.assertIsInstance(context, ClientTLSOptions)
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py
index a6c5f0a94..4c3fc36b6 100644
--- a/tests/test_pipeline_files.py
+++ b/tests/test_pipeline_files.py
@@ -634,19 +634,21 @@ class TestGCSFilesStore(unittest.TestCase):
             import google.cloud.storage  # noqa: F401
         except ModuleNotFoundError:
             raise unittest.SkipTest("google-cloud-storage is not installed")
-        with mock.patch("google.cloud.storage") as _:
-            with mock.patch("scrapy.pipelines.files.time") as _:
-                uri = "gs://my_bucket/my_prefix/"
-                store = GCSFilesStore(uri)
-                store.bucket = mock.Mock()
-                path = "full/my_data.txt"
-                yield store.persist_file(
-                    path, mock.Mock(), info=None, meta=None, headers=None
-                )
-                yield store.stat_file(path, info=None)
-                expected_blob_path = store.prefix + path
-                store.bucket.blob.assert_called_with(expected_blob_path)
-                store.bucket.get_blob.assert_called_with(expected_blob_path)
+        with (
+            mock.patch("google.cloud.storage"),
+            mock.patch("scrapy.pipelines.files.time"),
+        ):
+            uri = "gs://my_bucket/my_prefix/"
+            store = GCSFilesStore(uri)
+            store.bucket = mock.Mock()
+            path = "full/my_data.txt"
+            yield store.persist_file(
+                path, mock.Mock(), info=None, meta=None, headers=None
+            )
+            yield store.stat_file(path, info=None)
+            expected_blob_path = store.prefix + path
+            store.bucket.blob.assert_called_with(expected_blob_path)
+            store.bucket.get_blob.assert_called_with(expected_blob_path)
 
 
 class TestFTPFileStore(unittest.TestCase):
diff --git a/tests/test_responsetypes.py b/tests/test_responsetypes.py
index 7be8150fc..f9f56ff97 100644
--- a/tests/test_responsetypes.py
+++ b/tests/test_responsetypes.py
@@ -126,7 +126,3 @@ class ResponseTypesTest(unittest.TestCase):
         self.assertEqual(
             responsetypes.mimetypes.guess_type("x.scrapytest")[0], "x-scrapy/test"
         )
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py
index 8bd1480ad..3ac330ae2 100644
--- a/tests/test_scheduler.py
+++ b/tests/test_scheduler.py
@@ -277,14 +277,12 @@ class DownloaderAwareSchedulerTestMixin:
         downloader = self.mock_crawler.engine.downloader
         while self.scheduler.has_pending_requests():
             request = self.scheduler.next_request()
-            # pylint: disable=protected-access
             slot = downloader.get_slot_key(request)
             dequeued_slots.append(slot)
             downloader.increment(slot)
             requests.append(request)
 
         for request in requests:
-            # pylint: disable=protected-access
             slot = downloader.get_slot_key(request)
             downloader.decrement(slot)
 
diff --git a/tests/test_scrapy__getattr__.py b/tests/test_scrapy__getattr__.py
index 979c42267..443e26a3c 100644
--- a/tests/test_scrapy__getattr__.py
+++ b/tests/test_scrapy__getattr__.py
@@ -3,7 +3,7 @@ import warnings
 
 def test_deprecated_twisted_version():
     with warnings.catch_warnings(record=True) as warns:
-        from scrapy import twisted_version
+        from scrapy import twisted_version  # pylint: disable=no-name-in-module
 
         assert twisted_version is not None
         assert isinstance(twisted_version, tuple)
diff --git a/tests/test_selector.py b/tests/test_selector.py
index 1b5f3f018..857c7d626 100644
--- a/tests/test_selector.py
+++ b/tests/test_selector.py
@@ -264,7 +264,7 @@ class JMESPathTestCase(unittest.TestCase):
         )
 
     @pytest.mark.skipif(PARSEL_18_PLUS, reason="parsel >= 1.8 supports jmespath")
-    def test_jmespath_not_available(my_json_page) -> None:
+    def test_jmespath_not_available(self) -> None:
         body = """
         {
             "website": {"name": "Example"}
diff --git a/tests/test_settings/__init__.py b/tests/test_settings/__init__.py
index 503c29e32..5c8a19d9b 100644
--- a/tests/test_settings/__init__.py
+++ b/tests/test_settings/__init__.py
@@ -170,7 +170,7 @@ class BaseSettingsTest(unittest.TestCase):
 
         self.assertCountEqual(self.settings.attributes.keys(), ctrl_attributes.keys())
 
-        for key in ctrl_attributes.keys():
+        for key in ctrl_attributes:
             attr = self.settings.attributes[key]
             ctrl_attr = ctrl_attributes[key]
             self.assertEqual(attr.value, ctrl_attr.value)
@@ -497,7 +497,3 @@ class SettingsTest(unittest.TestCase):
         self.assertEqual(
             str(error.exception), "Trying to modify an immutable Settings object"
         )
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_spiderloader/__init__.py b/tests/test_spiderloader/__init__.py
index 9b53b9b96..705f722b3 100644
--- a/tests/test_spiderloader/__init__.py
+++ b/tests/test_spiderloader/__init__.py
@@ -1,3 +1,4 @@
+import contextlib
 import shutil
 import sys
 import tempfile
@@ -22,10 +23,8 @@ module_dir = Path(__file__).resolve().parent
 
 
 def _copytree(source: Path, target: Path):
-    try:
+    with contextlib.suppress(shutil.Error):
         shutil.copytree(source, target)
-    except shutil.Error:
-        pass
 
 
 class SpiderLoaderTest(unittest.TestCase):
@@ -98,6 +97,25 @@ class SpiderLoaderTest(unittest.TestCase):
         self.spider_loader = SpiderLoader.from_settings(settings)
         assert len(self.spider_loader._spiders) == 0
 
+    def test_load_spider_module_from_addons(self):
+        module = "tests.test_spiderloader.spiders_from_addons.spider0"
+
+        class SpiderModuleAddon:
+            @classmethod
+            def update_pre_crawler_settings(cls, settings):
+                settings.set(
+                    "SPIDER_MODULES",
+                    [module],
+                    "project",
+                )
+
+        runner = CrawlerRunner({"ADDONS": {SpiderModuleAddon: 1}})
+
+        crawler = runner.create_crawler("spider_from_addon")
+        self.assertTrue(issubclass(crawler.spidercls, scrapy.Spider))
+        self.assertEqual(crawler.spidercls.name, "spider_from_addon")
+        self.assertTrue(len(crawler.settings["SPIDER_MODULES"]) == 1)
+
     def test_crawler_runner_loading(self):
         module = "tests.test_spiderloader.test_spiders.spider1"
         runner = CrawlerRunner(
diff --git a/tests/test_spiderloader/spiders_from_addons/__init__.py b/tests/test_spiderloader/spiders_from_addons/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/test_spiderloader/spiders_from_addons/spider0.py b/tests/test_spiderloader/spiders_from_addons/spider0.py
new file mode 100644
index 000000000..45c3f64a7
--- /dev/null
+++ b/tests/test_spiderloader/spiders_from_addons/spider0.py
@@ -0,0 +1,6 @@
+from scrapy.spiders import Spider
+
+
+class SpiderFromAddon(Spider):
+    name = "spider_from_addon"
+    allowed_domains = ["scrapy1.org", "scrapy3.org"]
diff --git a/tests/test_spidermiddleware_offsite.py b/tests/test_spidermiddleware_offsite.py
index 837f1c2c8..906928e01 100644
--- a/tests/test_spidermiddleware_offsite.py
+++ b/tests/test_spidermiddleware_offsite.py
@@ -29,6 +29,7 @@ class TestOffsiteMiddleware(TestCase):
             Request("http://scrapy.org/1"),
             Request("http://sub.scrapy.org/1"),
             Request("http://offsite.tld/letmepass", dont_filter=True),
+            Request("http://offsite-2.tld/allow", meta={"allow_offsite": True}),
             Request("http://scrapy.test.org/"),
             Request("http://scrapy.test.org:8000/"),
         ]
diff --git a/tests/test_spidermiddleware_referer.py b/tests/test_spidermiddleware_referer.py
index 23b0c17c6..4945ac25d 100644
--- a/tests/test_spidermiddleware_referer.py
+++ b/tests/test_spidermiddleware_referer.py
@@ -891,13 +891,11 @@ class TestSettingsPolicyByName(TestCase):
         # test parsing without space(s) after the comma
         settings1 = Settings(
             {
-                "REFERRER_POLICY": ",".join(
-                    [
-                        "some-custom-unknown-policy",
-                        POLICY_SAME_ORIGIN,
-                        POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN,
-                        "another-custom-unknown-policy",
-                    ]
+                "REFERRER_POLICY": (
+                    f"some-custom-unknown-policy,"
+                    f"{POLICY_SAME_ORIGIN},"
+                    f"{POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN},"
+                    f"another-custom-unknown-policy"
                 )
             }
         )
@@ -907,12 +905,10 @@ class TestSettingsPolicyByName(TestCase):
         # test parsing with space(s) after the comma
         settings2 = Settings(
             {
-                "REFERRER_POLICY": ",    ".join(
-                    [
-                        POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN,
-                        "another-custom-unknown-policy",
-                        POLICY_UNSAFE_URL,
-                    ]
+                "REFERRER_POLICY": (
+                    f"{POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGIN},"
+                    f"    another-custom-unknown-policy,"
+                    f"    {POLICY_UNSAFE_URL}"
                 )
             }
         )
@@ -922,12 +918,10 @@ class TestSettingsPolicyByName(TestCase):
     def test_multiple_policy_tokens_all_invalid(self):
         settings = Settings(
             {
-                "REFERRER_POLICY": ",".join(
-                    [
-                        "some-custom-unknown-policy",
-                        "another-custom-unknown-policy",
-                        "yet-another-custom-unknown-policy",
-                    ]
+                "REFERRER_POLICY": (
+                    "some-custom-unknown-policy,"
+                    "another-custom-unknown-policy,"
+                    "yet-another-custom-unknown-policy"
                 )
             }
         )
diff --git a/tests/test_squeues_request.py b/tests/test_squeues_request.py
index 02ea8027f..04eeae4dc 100644
--- a/tests/test_squeues_request.py
+++ b/tests/test_squeues_request.py
@@ -1,3 +1,7 @@
+"""
+Queues that handle requests
+"""
+
 import shutil
 import tempfile
 import unittest
@@ -16,10 +20,6 @@ from scrapy.squeues import (
 )
 from scrapy.utils.test import get_crawler
 
-"""
-Queues that handle requests
-"""
-
 
 class BaseQueueTestCase(unittest.TestCase):
     def setUp(self):
diff --git a/tests/test_utils_conf.py b/tests/test_utils_conf.py
index 2ce7948eb..cbea41129 100644
--- a/tests/test_utils_conf.py
+++ b/tests/test_utils_conf.py
@@ -177,7 +177,3 @@ class FeedExportConfigTestCase(unittest.TestCase):
                 "item_export_kwargs": {},
             },
         )
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_console.py b/tests/test_utils_console.py
index dabd6054d..0bc86e1b9 100644
--- a/tests/test_utils_console.py
+++ b/tests/test_utils_console.py
@@ -38,7 +38,3 @@ class UtilsConsoleTestCase(unittest.TestCase):
         # default shell should be 'ipython'
         shell = get_shell_embed_func()
         self.assertEqual(shell.__name__, "_embed_ipython_shell")
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_datatypes.py b/tests/test_utils_datatypes.py
index 5a76593c3..e80381671 100644
--- a/tests/test_utils_datatypes.py
+++ b/tests/test_utils_datatypes.py
@@ -372,7 +372,3 @@ class LocalWeakReferencedCacheTest(unittest.TestCase):
         for i, r in enumerate(refs):
             self.assertIn(r, cache)
             self.assertEqual(cache[r], i)
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_deprecate.py b/tests/test_utils_deprecate.py
index eedb6f6af..dc5fbd3c3 100644
--- a/tests/test_utils_deprecate.py
+++ b/tests/test_utils_deprecate.py
@@ -259,12 +259,14 @@ class WarnWhenSubclassedTest(unittest.TestCase):
         self.assertIn("foo.Bar", str(w[1].message))
 
     def test_inspect_stack(self):
-        with mock.patch("inspect.stack", side_effect=IndexError):
-            with warnings.catch_warnings(record=True) as w:
-                DeprecatedName = create_deprecated_class("DeprecatedName", NewName)
+        with (
+            mock.patch("inspect.stack", side_effect=IndexError),
+            warnings.catch_warnings(record=True) as w,
+        ):
+            DeprecatedName = create_deprecated_class("DeprecatedName", NewName)
 
-                class SubClass(DeprecatedName):
-                    pass
+            class SubClass(DeprecatedName):
+                pass
 
         self.assertIn("Error detecting parent module", str(w[0].message))
 
diff --git a/tests/test_utils_httpobj.py b/tests/test_utils_httpobj.py
index b824972d5..741e69559 100644
--- a/tests/test_utils_httpobj.py
+++ b/tests/test_utils_httpobj.py
@@ -20,7 +20,3 @@ class HttpobjUtilsTest(unittest.TestCase):
         assert req1a is req1b
         assert req1a is not req2
         assert req1a is not req2
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_iterators.py b/tests/test_utils_iterators.py
index 4c81e3a2f..12507c6a3 100644
--- a/tests/test_utils_iterators.py
+++ b/tests/test_utils_iterators.py
@@ -366,7 +366,7 @@ class UtilsCsvTestCase(unittest.TestCase):
 
         # explicit type check cuz' we no like stinkin' autocasting! yarrr
         for result_row in result:
-            self.assertTrue(all(isinstance(k, str) for k in result_row.keys()))
+            self.assertTrue(all(isinstance(k, str) for k in result_row))
             self.assertTrue(all(isinstance(v, str) for v in result_row.values()))
 
     def test_csviter_delimiter(self):
diff --git a/tests/test_utils_misc/__init__.py b/tests/test_utils_misc/__init__.py
index f71b2b034..478c1e73a 100644
--- a/tests/test_utils_misc/__init__.py
+++ b/tests/test_utils_misc/__init__.py
@@ -216,7 +216,3 @@ class UtilsMiscTestCase(unittest.TestCase):
         assert rel_has_nofollow("nofollowfoo") is False
         assert rel_has_nofollow("foonofollow") is False
         assert rel_has_nofollow("ugc,  ,  nofollow") is True
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_project.py b/tests/test_utils_project.py
index 3831f4c21..1d149d48d 100644
--- a/tests/test_utils_project.py
+++ b/tests/test_utils_project.py
@@ -12,7 +12,7 @@ from scrapy.utils.project import data_path, get_project_settings
 
 @contextlib.contextmanager
 def inside_a_project():
-    prev_dir = os.getcwd()
+    prev_dir = Path.cwd()
     project_dir = tempfile.mkdtemp()
 
     try:
diff --git a/tests/test_utils_request.py b/tests/test_utils_request.py
index 0a3e3b00b..51bca9a31 100644
--- a/tests/test_utils_request.py
+++ b/tests/test_utils_request.py
@@ -481,7 +481,3 @@ class RequestToCurlTest(unittest.TestCase):
             " --data-raw '{\"foo\": \"bar\"}' --cookie 'foo=bar'"
         )
         self._test_request(request_object, expected_curl_command)
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_sitemap.py b/tests/test_utils_sitemap.py
index ce0de0722..69a459d8b 100644
--- a/tests/test_utils_sitemap.py
+++ b/tests/test_utils_sitemap.py
@@ -295,7 +295,3 @@ Disallow: /forum/active/
         )
 
         self.assertEqual(list(s), [{"loc": "http://127.0.0.1:8000/"}])
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_spider.py b/tests/test_utils_spider.py
index ae59d0137..df8f37103 100644
--- a/tests/test_utils_spider.py
+++ b/tests/test_utils_spider.py
@@ -30,7 +30,3 @@ class UtilsSpidersTestCase(unittest.TestCase):
 
         it = iter_spider_classes(tests.test_utils_spider)
         self.assertEqual(set(it), {MySpider1, MySpider2})
-
-
-if __name__ == "__main__":
-    unittest.main()
diff --git a/tests/test_utils_template.py b/tests/test_utils_template.py
index fc42c0d2f..5fbbd74da 100644
--- a/tests/test_utils_template.py
+++ b/tests/test_utils_template.py
@@ -33,7 +33,3 @@ class UtilsRenderTemplateFileTestCase(unittest.TestCase):
 
         render_path.unlink()
         assert not render_path.exists()  # Failure of test itself
-
-
-if "__main__" == __name__:
-    unittest.main()
diff --git a/tests/test_utils_trackref.py b/tests/test_utils_trackref.py
index ef07d625f..58efad585 100644
--- a/tests/test_utils_trackref.py
+++ b/tests/test_utils_trackref.py
@@ -61,11 +61,11 @@ Foo                                 1   oldest: 0s ago\n\n""",
         )
 
     def test_get_oldest(self):
-        o1 = Foo()  # noqa: F841
+        o1 = Foo()
 
         o1_time = time()
 
-        o2 = Bar()  # noqa: F841
+        o2 = Bar()
 
         o3_time = time()
         if o3_time <= o1_time:
@@ -80,9 +80,9 @@ Foo                                 1   oldest: 0s ago\n\n""",
         self.assertIsNone(trackref.get_oldest("XXX"))
 
     def test_iter_all(self):
-        o1 = Foo()  # noqa: F841
+        o1 = Foo()
         o2 = Bar()  # noqa: F841
-        o3 = Foo()  # noqa: F841
+        o3 = Foo()
         self.assertEqual(
             set(trackref.iter_all("Foo")),
             {o1, o3},
diff --git a/tests/test_utils_url.py b/tests/test_utils_url.py
index 62e2b5c1e..314082742 100644
--- a/tests/test_utils_url.py
+++ b/tests/test_utils_url.py
@@ -6,7 +6,7 @@ import pytest
 from scrapy.linkextractors import IGNORED_EXTENSIONS
 from scrapy.spiders import Spider
 from scrapy.utils.misc import arg_to_iter
-from scrapy.utils.url import (
+from scrapy.utils.url import (  # type: ignore[attr-defined]
     _is_filesystem_path,
     _public_w3lib_objects,
     add_http_if_no_scheme,
@@ -630,7 +630,3 @@ def test_deprecated_imports_from_w3lib(obj_name):
         getattr(import_module("scrapy.utils.url"), obj_name)
 
         assert message in warns[0].message.args
-
-
-if __name__ == "__main__":
-    unittest.main()