Repository: sphinx-doc/sphinx

Issue:
Flaky unit test: test_auth_header_uses_first_match
### Describe the bug

The `test_auth_header_uses_first_match` test seems to fail intermittently, particularly when threaded test HTTP servers are in use (as in #11340).

### How to Reproduce

Visible in continuous build output in jayaddison/sphinx#2.

### Environment Information

```text
N/A
```


### Sphinx extensions

```python
N/A
```


### Additional context

_No response_

Discussion:
jayaddison: (I suppose technically #11392 should be merged first, or this should be merged into #11392 -- because threads aren't in use in the mainline codebase yet, so the pull request title is inaccurate at the moment, and would be in the commit history too)
jayaddison: I'm sorta unsatisfied and not entirely convinced by this explanation, but it does seem that the algorithm used by `headers.as_string` method is non-trivial complexity-wise; from the [relevant documentation](https://docs.python.org/3/library/http.server.html#http.server.BaseHTTPRequestHandler.headers) it's essentially an `email.message.Message`-like object, and the latter has [had a few bugs](https://github.com/python/cpython/pulls?q=is%3Apr+email+folding+is%3Aclosed) and [has been rewritten](https://github.com/python/cpython/pull/4693) at least once.

Would that complexity be enough to put some of the unit tests near the client timeout threshold? ... I don't know, [0.05s](https://github.com/sphinx-doc/sphinx/blob/fff7d120f0e24869b7923b18c4e97b916215eed9/tests/roots/test-linkcheck-localserver/conf.py#L2) seems like it should be a lot of compute-time budget.  :shrug: 

I've removed a call to the method in 81ce0f0dcdf03788d19666e295f9ed88b94327a5 and on a sample size of one, it allowed the two instances of `test_linkcheck_request_headers` on Py3.8 (two because we run the tests against two versions of `docutils`) to pass when they both failed on a previous version of the code (although I did add a merge-latest between the two).
jayaddison: Further suggestions and feedback welcome; otherwise I think that this should be ready for review (no further changes planned at the moment).
jayaddison: Thank you @francoisfreitag!

Patch:
diff --git a/doc/faq.rst b/doc/faq.rst
index 68eb4745f..85ccb245e 100644
--- a/doc/faq.rst
+++ b/doc/faq.rst
@@ -40,7 +40,7 @@ How do I...
 For many more extensions and other contributed stuff, see the sphinx-contrib_
 repository.
 
-.. _sphinx-contrib: https://bitbucket.org/birkenfeld/sphinx-contrib/
+.. _sphinx-contrib: https://github.com/sphinx-contrib/
 
 .. _usingwith:
 
@@ -63,7 +63,8 @@ Doxygen
 
 SCons
    Glenn Hutchings has written a SCons build script to build Sphinx
-   documentation; it is hosted here: https://bitbucket.org/zondo/sphinx-scons
+   documentation; it is hosted here:
+   https://bitbucket-archive.softwareheritage.org/projects/zo/zondo/sphinx-scons.html
 
 PyPI
    Jannis Leidel wrote a `setuptools command
@@ -77,8 +78,9 @@ GitHub Pages
    GitHub Pages on building HTML document automatically.
 
 MediaWiki
-   See https://bitbucket.org/kevindunn/sphinx-wiki/wiki/Home, a project by
-   Kevin Dunn.
+   See `sphinx-wiki`_, a project by Kevin Dunn.
+
+   .. _sphinx-wiki: https://bitbucket-archive.softwareheritage.org/projects/ke/kevindunn/sphinx-wiki.html
 
 Google Analytics
    You can use a custom ``layout.html`` template, like this:
diff --git a/doc/man/sphinx-build.rst b/doc/man/sphinx-build.rst
index a66423010..a1fb9be40 100644
--- a/doc/man/sphinx-build.rst
+++ b/doc/man/sphinx-build.rst
@@ -201,10 +201,19 @@ Options
    references.  See the config value :confval:`nitpick_ignore` for a way to
    exclude some references as "known missing".
 
-.. option:: -N
+.. option:: -N, --no-color
 
    Do not emit colored output.
 
+   .. versionchanged:: 1.6
+      Add ``--no-color`` long option.
+
+.. option:: --color
+
+   Emit colored output. Auto-detected by default.
+
+   .. versionadded:: 1.6
+
 .. option:: -v
 
    Increase verbosity (loglevel).  This option can be given up to three times
diff --git a/doc/usage/restructuredtext/domains.rst b/doc/usage/restructuredtext/domains.rst
index cbece86e8..23ac543a4 100644
--- a/doc/usage/restructuredtext/domains.rst
+++ b/doc/usage/restructuredtext/domains.rst
@@ -2179,5 +2179,5 @@ Jinja_, Operation_, and Scala_.
 .. _MATLAB: https://pypi.org/project/sphinxcontrib-matlabdomain/
 .. _Operation: https://pypi.org/project/sphinxcontrib-operationdomain/
 .. _PHP: https://pypi.org/project/sphinxcontrib-phpdomain/
-.. _Ruby: https://bitbucket.org/birkenfeld/sphinx-contrib/src/default/rubydomain
+.. _Ruby: https://github.com/sphinx-contrib/rubydomain
 .. _Scala: https://pypi.org/project/sphinxcontrib-scaladomain/
diff --git a/sphinx/builders/__init__.py b/sphinx/builders/__init__.py
index a5d5a1ef6..b6471d3d5 100644
--- a/sphinx/builders/__init__.py
+++ b/sphinx/builders/__init__.py
@@ -496,11 +496,7 @@ def read_doc(self, docname: str, *, _cache: bool = True) -> None:
             doctree = publisher.document
 
         # store time of reading, for outdated files detection
-        # (Some filesystems have coarse timestamp resolution;
-        # therefore time.time() can be older than filesystem's timestamp.
-        # For example, FAT32 has 2sec timestamp resolution.)
-        self.env.all_docs[docname] = max(time.time(),
-                                         path.getmtime(self.env.doc2path(docname)))
+        self.env.all_docs[docname] = time.time_ns() // 1_000
 
         # cleanup
         self.env.temp_data.clear()
diff --git a/sphinx/builders/_epub_base.py b/sphinx/builders/_epub_base.py
index dfe4877ec..1609c4995 100644
--- a/sphinx/builders/_epub_base.py
+++ b/sphinx/builders/_epub_base.py
@@ -65,9 +65,9 @@
     '.svg': 'image/svg+xml',
     '.jpg': 'image/jpeg',
     '.jpeg': 'image/jpeg',
-    '.otf': 'application/x-font-otf',
-    '.ttf': 'application/x-font-ttf',
-    '.woff': 'application/font-woff',
+    '.otf': 'font/otf',
+    '.ttf': 'font/ttf',
+    '.woff': 'font/woff',
 }
 
 VECTOR_GRAPHICS_EXTENSIONS = ('.svg',)
diff --git a/sphinx/builders/gettext.py b/sphinx/builders/gettext.py
index 697b75e40..ee2f8bcd6 100644
--- a/sphinx/builders/gettext.py
+++ b/sphinx/builders/gettext.py
@@ -4,7 +4,7 @@
 
 from codecs import open
 from collections import defaultdict
-from datetime import datetime, timedelta, tzinfo
+from datetime import datetime, timedelta, timezone, tzinfo
 from os import getenv, path, walk
 from time import time
 from typing import Any, Generator, Iterable
@@ -163,8 +163,10 @@ def write_doc(self, docname: str, doctree: nodes.document) -> None:
 
 # determine tzoffset once to remain unaffected by DST change during build
 timestamp = time()
-tzdelta = datetime.fromtimestamp(timestamp) - \
-    datetime.utcfromtimestamp(timestamp)
+local_time = datetime.fromtimestamp(timestamp)
+utc_time = datetime.fromtimestamp(timestamp, tz=timezone.utc)
+tzdelta = local_time - utc_time.replace(tzinfo=None)
+
 # set timestamp from SOURCE_DATE_EPOCH if set
 # see https://reproducible-builds.org/specs/source-date-epoch/
 source_date_epoch = getenv('SOURCE_DATE_EPOCH')
diff --git a/sphinx/builders/linkcheck.py b/sphinx/builders/linkcheck.py
index 6c99f96e6..479a5b877 100644
--- a/sphinx/builders/linkcheck.py
+++ b/sphinx/builders/linkcheck.py
@@ -13,12 +13,12 @@
 from os import path
 from queue import PriorityQueue, Queue
 from threading import Thread
-from typing import Any, Generator, NamedTuple, Tuple, Union, cast
-from urllib.parse import unquote, urlparse, urlunparse
+from typing import Any, Callable, Generator, Iterator, NamedTuple, Tuple, Union, cast
+from urllib.parse import unquote, urlparse, urlsplit, urlunparse
 
 from docutils import nodes
 from requests import Response
-from requests.exceptions import ConnectionError, HTTPError, TooManyRedirects
+from requests.exceptions import ConnectionError, HTTPError, SSLError, TooManyRedirects
 
 from sphinx.application import Sphinx
 from sphinx.builders.dummy import DummyBuilder
@@ -72,7 +72,7 @@ class RateLimit(NamedTuple):
 
 
 class AnchorCheckParser(HTMLParser):
-    """Specialized HTML parser that looks for a specific anchor."""
+    """Specialised HTML parser that looks for a specific anchor."""
 
     def __init__(self, search_anchor: str) -> None:
         super().__init__()
@@ -87,11 +87,10 @@ def handle_starttag(self, tag: Any, attrs: Any) -> None:
                 break
 
 
-def check_anchor(response: requests.requests.Response, anchor: str) -> bool:
-    """Reads HTML data from a response object `response` searching for `anchor`.
-    Returns True if anchor was found, False otherwise.
-    """
-    parser = AnchorCheckParser(anchor)
+def contains_anchor(response: Response, anchor: str) -> bool:
+    """Determine if an anchor is contained within an HTTP response."""
+
+    parser = AnchorCheckParser(unquote(anchor))
     # Read file in chunks. If we find a matching anchor, we break
     # the loop early in hopes not to have to download the whole thing.
     for chunk in response.iter_content(chunk_size=4096, decode_unicode=True):
@@ -271,7 +270,7 @@ def run(self) -> None:
             kwargs['timeout'] = self.config.linkcheck_timeout
 
         def get_request_headers() -> dict[str, str]:
-            url = urlparse(uri)
+            url = urlsplit(uri)
             candidates = [f"{url.scheme}://{url.netloc}",
                           f"{url.scheme}://{url.netloc}/",
                           uri,
@@ -286,16 +285,11 @@ def get_request_headers() -> dict[str, str]:
             return {}
 
         def check_uri() -> tuple[str, str, int]:
-            # split off anchor
-            if '#' in uri:
-                req_url, anchor = uri.split('#', 1)
-                for rex in self.anchors_ignore:
-                    if rex.match(anchor):
-                        anchor = None
-                        break
-            else:
-                req_url = uri
-                anchor = None
+            req_url, delimiter, anchor = uri.partition('#')
+            for rex in self.anchors_ignore if delimiter and anchor else []:
+                if rex.match(anchor):
+                    anchor = ''
+                    break
 
             # handle non-ASCII URIs
             try:
@@ -313,71 +307,87 @@ def check_uri() -> tuple[str, str, int]:
             # update request headers for the URL
             kwargs['headers'] = get_request_headers()
 
-            try:
-                if anchor and self.config.linkcheck_anchors:
-                    # Read the whole document and see if #anchor exists
-                    with requests.get(req_url, stream=True, config=self.config, auth=auth_info,
-                                      **kwargs) as response:
-                        response.raise_for_status()
-                        found = check_anchor(response, unquote(anchor))
-
-                    if not found:
-                        raise Exception(__("Anchor '%s' not found") % anchor)
-                else:
-                    try:
-                        # try a HEAD request first, which should be easier on
-                        # the server and the network
-                        with requests.head(req_url, allow_redirects=True, config=self.config,
-                                           auth=auth_info, **kwargs) as response:
-                            response.raise_for_status()
+            # Linkcheck HTTP request logic:
+            #
+            # - Attempt HTTP HEAD before HTTP GET unless page content is required.
+            # - Follow server-issued HTTP redirects.
+            # - Respect server-issued HTTP 429 back-offs.
+            error_message = None
+            status_code = -1
+            response_url = retry_after = ''
+            for retrieval_method, retrieval_kwargs in _retrieval_methods(
+                    self.config.linkcheck_anchors, anchor,
+            ):
+                try:
+                    with retrieval_method(url=req_url, auth=auth_info, config=self.config,
+                                          **retrieval_kwargs, **kwargs) as response:
+                        if response.ok and anchor and not contains_anchor(response, anchor):
+                            raise Exception(__(f'Anchor {anchor!r} not found'))
+
+                    # Copy data we need from the (closed) response
+                    status_code = response.status_code
+                    redirect_status_code = response.history[-1].status_code if response.history else None  # NoQA: E501
+                    retry_after = response.headers.get('Retry-After')
+                    response_url = f'{response.url}'
+                    response.raise_for_status()
+                    del response
+                    break
+
+                except SSLError as err:
+                    # SSL failure; report that the link is broken.
+                    return 'broken', str(err), 0
+
+                except (ConnectionError, TooManyRedirects) as err:
                     # Servers drop the connection on HEAD requests, causing
                     # ConnectionError.
-                    except (ConnectionError, HTTPError, TooManyRedirects) as err:
-                        if isinstance(err, HTTPError) and err.response.status_code == 429:
-                            raise
-                        # retry with GET request if that fails, some servers
-                        # don't like HEAD requests.
-                        with requests.get(req_url, stream=True, config=self.config,
-                                          auth=auth_info, **kwargs) as response:
-                            response.raise_for_status()
-            except HTTPError as err:
-                if err.response.status_code == 401:
-                    # We'll take "Unauthorized" as working.
-                    return 'working', ' - unauthorized', 0
-                elif err.response.status_code == 429:
-                    next_check = self.limit_rate(err.response)
-                    if next_check is not None:
-                        self.wqueue.put(CheckRequest(next_check, hyperlink), False)
-                        return 'rate-limited', '', 0
-                    return 'broken', str(err), 0
-                elif err.response.status_code == 503:
-                    # We'll take "Service Unavailable" as ignored.
-                    return 'ignored', str(err), 0
-                else:
+                    error_message = str(err)
+                    continue
+
+                except HTTPError as err:
+                    error_message = str(err)
+
+                    # Unauthorised: the reference probably exists
+                    if status_code == 401:
+                        return 'working', 'unauthorized', 0
+
+                    # Rate limiting; back-off if allowed, or report failure otherwise
+                    if status_code == 429:
+                        if next_check := self.limit_rate(response_url, retry_after):
+                            self.wqueue.put(CheckRequest(next_check, hyperlink), False)
+                            return 'rate-limited', '', 0
+                        return 'broken', error_message, 0
+
+                    # Don't claim success/failure during server-side outages
+                    if status_code == 503:
+                        return 'ignored', 'service unavailable', 0
+
+                    # For most HTTP failures, continue attempting alternate retrieval methods
+                    continue
+
+                except Exception as err:
+                    # Unhandled exception (intermittent or permanent); report that
+                    # the link is broken.
                     return 'broken', str(err), 0
-            except Exception as err:
-                return 'broken', str(err), 0
+
             else:
-                netloc = urlparse(req_url).netloc
-                try:
-                    del self.rate_limits[netloc]
-                except KeyError:
-                    pass
-            if response.url.rstrip('/') == req_url.rstrip('/'):
+                # All available retrieval methods have been exhausted; report
+                # that the link is broken.
+                return 'broken', error_message, 0
+
+            # Success; clear rate limits for the origin
+            netloc = urlsplit(req_url).netloc
+            try:
+                del self.rate_limits[netloc]
+            except KeyError:
+                pass
+
+            if ((response_url.rstrip('/') == req_url.rstrip('/'))
+                    or allowed_redirect(req_url, response_url)):
                 return 'working', '', 0
+            elif redirect_status_code is not None:
+                return 'redirected', response_url, redirect_status_code
             else:
-                new_url = response.url
-                if anchor:
-                    new_url += '#' + anchor
-
-                if allowed_redirect(req_url, new_url):
-                    return 'working', '', 0
-                elif response.history:
-                    # history contains any redirects, get last
-                    code = response.history[-1].status_code
-                    return 'redirected', new_url, code
-                else:
-                    return 'redirected', new_url, 0
+                return 'redirected', response_url, 0
 
         def allowed_redirect(url: str, new_url: str) -> bool:
             return any(
@@ -428,7 +438,7 @@ def check(docname: str) -> tuple[str, str, int]:
 
             if uri is None:
                 break
-            netloc = urlparse(uri).netloc
+            netloc = urlsplit(uri).netloc
             try:
                 # Refresh rate limit.
                 # When there are many links in the queue, workers are all stuck waiting
@@ -451,9 +461,8 @@ def check(docname: str) -> tuple[str, str, int]:
                 self.rqueue.put(CheckResult(uri, docname, lineno, status, info, code))
             self.wqueue.task_done()
 
-    def limit_rate(self, response: Response) -> float | None:
+    def limit_rate(self, response_url: str, retry_after: str) -> float | None:
         next_check = None
-        retry_after = response.headers.get("Retry-After")
         if retry_after:
             try:
                 # Integer: time to wait before next attempt.
@@ -471,7 +480,7 @@ def limit_rate(self, response: Response) -> float | None:
                     delay = (until - datetime.now(timezone.utc)).total_seconds()
             else:
                 next_check = time.time() + delay
-        netloc = urlparse(response.url).netloc
+        netloc = urlsplit(response_url).netloc
         if next_check is None:
             max_delay = self.config.linkcheck_rate_limit_timeout
             try:
@@ -490,6 +499,15 @@ def limit_rate(self, response: Response) -> float | None:
         return next_check
 
 
+def _retrieval_methods(
+    linkcheck_anchors: bool,
+    anchor: str,
+) -> Iterator[tuple[Callable, dict[str, bool]]]:
+    if not linkcheck_anchors or not anchor:
+        yield requests.head, {'allow_redirects': True}
+    yield requests.get, {'stream': True}
+
+
 class HyperlinkCollector(SphinxPostTransform):
     builders = ('linkcheck',)
     default_priority = 800
diff --git a/sphinx/environment/__init__.py b/sphinx/environment/__init__.py
index b1d0dc4b4..d2beffdd9 100644
--- a/sphinx/environment/__init__.py
+++ b/sphinx/environment/__init__.py
@@ -7,7 +7,7 @@
 import pickle
 from collections import defaultdict
 from copy import copy
-from datetime import datetime
+from datetime import datetime, timezone
 from os import path
 from typing import TYPE_CHECKING, Any, Callable, Generator, Iterator
 
@@ -55,7 +55,7 @@
 
 # This is increased every time an environment attribute is added
 # or changed to properly invalidate pickle files.
-ENV_VERSION = 57
+ENV_VERSION = 58
 
 # config status
 CONFIG_OK = 1
@@ -166,9 +166,9 @@ def __init__(self, app: Sphinx):
         # All "docnames" here are /-separated and relative and exclude
         # the source suffix.
 
-        # docname -> mtime at the time of reading
+        # docname -> time of reading (in integer microseconds)
         # contains all read docnames
-        self.all_docs: dict[str, float] = {}
+        self.all_docs: dict[str, int] = {}
         # docname -> set of dependent file
         # names, relative to documentation root
         self.dependencies: dict[str, set[str]] = defaultdict(set)
@@ -481,12 +481,14 @@ def get_outdated_files(self, config_changed: bool) -> tuple[set[str], set[str],
                     continue
                 # check the mtime of the document
                 mtime = self.all_docs[docname]
-                newmtime = path.getmtime(self.doc2path(docname))
+                newmtime = _last_modified_time(self.doc2path(docname))
                 if newmtime > mtime:
+                    # convert integer microseconds to floating-point seconds,
+                    # and then to timezone-aware datetime objects.
+                    mtime_dt = datetime.fromtimestamp(mtime / 1_000_000, tz=timezone.utc)
+                    newmtime_dt = datetime.fromtimestamp(mtime / 1_000_000, tz=timezone.utc)
                     logger.debug('[build target] outdated %r: %s -> %s',
-                                 docname,
-                                 datetime.utcfromtimestamp(mtime),
-                                 datetime.utcfromtimestamp(newmtime))
+                                 docname, mtime_dt, newmtime_dt)
                     changed.add(docname)
                     continue
                 # finally, check the mtime of dependencies
@@ -497,7 +499,7 @@ def get_outdated_files(self, config_changed: bool) -> tuple[set[str], set[str],
                         if not path.isfile(deppath):
                             changed.add(docname)
                             break
-                        depmtime = path.getmtime(deppath)
+                        depmtime = _last_modified_time(deppath)
                         if depmtime > mtime:
                             changed.add(docname)
                             break
@@ -728,3 +730,18 @@ def check_consistency(self) -> None:
         for domain in self.domains.values():
             domain.check_consistency()
         self.events.emit('env-check-consistency', self)
+
+
+def _last_modified_time(filename: str | os.PathLike[str]) -> int:
+    """Return the last modified time of ``filename``.
+
+    The time is returned as integer microseconds.
+    The lowest common denominator of modern file-systems seems to be
+    microsecond-level precision.
+
+    We prefer to err on the side of re-rendering a file,
+    so we round up to the nearest microsecond.
+    """
+
+    # upside-down floor division to get the ceiling
+    return -(os.stat(filename).st_mtime_ns // -1_000)
diff --git a/sphinx/io.py b/sphinx/io.py
index 335502e57..efd2b4ea9 100644
--- a/sphinx/io.py
+++ b/sphinx/io.py
@@ -182,7 +182,7 @@ def create_publisher(app: Sphinx, filetype: str) -> Publisher:
     defaults = {"traceback": True, **app.env.settings}
     # Set default settings
     if docutils.__version_info__[:2] >= (0, 19):
-        pub.get_settings(**defaults)  # type: ignore[arg-type]
+        pub.get_settings(**defaults)
     else:
-        pub.settings = pub.setup_option_parser(**defaults).get_default_values()  # type: ignore
+        pub.settings = pub.setup_option_parser(**defaults).get_default_values()
     return pub
diff --git a/sphinx/writers/html5.py b/sphinx/writers/html5.py
index e7d932286..5aea3ddb8 100644
--- a/sphinx/writers/html5.py
+++ b/sphinx/writers/html5.py
@@ -887,3 +887,12 @@ def depart_math_block(self, node: Element, math_env: str = '') -> None:
         _, depart = self.builder.app.registry.html_block_math_renderers[name]
         if depart:  # type: ignore[truthy-function]
             depart(self, node)
+
+    # See Docutils r9413
+    # Re-instate the footnote-reference class
+    def visit_footnote_reference(self, node):
+        href = '#' + node['refid']
+        classes = ['footnote-reference', self.settings.footnote_references]
+        self.body.append(self.starttag(node, 'a', suffix='', classes=classes,
+                                       role='doc-noteref', href=href))
+        self.body.append('<span class="fn-bracket">[</span>')


Test patch:
diff --git a/sphinx/testing/util.py b/sphinx/testing/util.py
index 887f8dbfe..66cfc5172 100644
--- a/sphinx/testing/util.py
+++ b/sphinx/testing/util.py
@@ -112,8 +112,9 @@ def __init__(
         parallel: int = 0,
     ) -> None:
 
+        self.docutils_conf_path = srcdir / 'docutils.conf'
         if docutilsconf is not None:
-            (srcdir / 'docutils.conf').write_text(docutilsconf)
+            self.docutils_conf_path.write_text(docutilsconf)
 
         if builddir is None:
             builddir = srcdir / '_build'
@@ -154,6 +155,10 @@ def cleanup(self, doctrees: bool = False) -> None:
                method not in self._saved_nodeclasses:
                 delattr(nodes.GenericNodeVisitor, 'visit_' + method[6:])
                 delattr(nodes.GenericNodeVisitor, 'depart_' + method[6:])
+        try:
+            os.remove(self.docutils_conf_path)
+        except FileNotFoundError:
+            pass
 
     def __repr__(self) -> str:
         return f'<{self.__class__.__name__} buildername={self.builder.name!r}>'
diff --git a/tests/test_build_linkcheck.py b/tests/test_build_linkcheck.py
index 260cf2c42..7bc4703d6 100644
--- a/tests/test_build_linkcheck.py
+++ b/tests/test_build_linkcheck.py
@@ -2,13 +2,13 @@
 
 from __future__ import annotations
 
-import base64
 import http.server
 import json
 import re
 import textwrap
 import time
 import wsgiref.handlers
+from base64 import b64encode
 from datetime import datetime
 from os import path
 from queue import Queue
@@ -18,6 +18,7 @@
 
 from sphinx.builders.linkcheck import HyperlinkAvailabilityCheckWorker, RateLimit
 from sphinx.testing.util import strip_escseq
+from sphinx.util import requests
 from sphinx.util.console import strip_colors
 
 from .utils import CERT_FILE, http_server, https_server
@@ -194,16 +195,44 @@ def do_GET(self):
     )
 
 
-def capture_headers_handler(records):
-    class HeadersDumperHandler(http.server.BaseHTTPRequestHandler):
+def custom_handler(valid_credentials=(), success_criteria=lambda _: True):
+    """
+    Returns an HTTP request handler that authenticates the client and then determines
+    an appropriate HTTP response code, based on caller-provided credentials and optional
+    success criteria, respectively.
+    """
+    expected_token = None
+    if valid_credentials:
+        assert len(valid_credentials) == 2, "expected a pair of strings as credentials"
+        expected_token = b64encode(":".join(valid_credentials).encode()).decode("utf-8")
+        del valid_credentials
+
+    class CustomHandler(http.server.BaseHTTPRequestHandler):
+        def authenticated(method):
+            def method_if_authenticated(self):
+                if expected_token is None:
+                    return method(self)
+                elif self.headers["Authorization"] == f"Basic {expected_token}":
+                    return method(self)
+                else:
+                    self.send_response(403, "Forbidden")
+                    self.end_headers()
+
+            return method_if_authenticated
+
+        @authenticated
         def do_HEAD(self):
             self.do_GET()
 
+        @authenticated
         def do_GET(self):
-            self.send_response(200, "OK")
+            if success_criteria(self):
+                self.send_response(200, "OK")
+            else:
+                self.send_response(400, "Bad Request")
             self.end_headers()
-            records.append(self.headers.as_string())
-    return HeadersDumperHandler
+
+    return CustomHandler
 
 
 @pytest.mark.sphinx(
@@ -214,25 +243,29 @@ def do_GET(self):
         (r'.*local.*', ('user2', 'hunter2')),
     ]})
 def test_auth_header_uses_first_match(app):
-    records = []
-    with http_server(capture_headers_handler(records)):
+    with http_server(custom_handler(valid_credentials=("user1", "password"))):
         app.build()
 
-    stdout = "\n".join(records)
-    encoded_auth = base64.b64encode(b'user1:password').decode('ascii')
-    assert f"Authorization: Basic {encoded_auth}\n" in stdout
+    with open(app.outdir / "output.json", encoding="utf-8") as fp:
+        content = json.load(fp)
+
+    assert content["status"] == "working"
 
 
 @pytest.mark.sphinx(
     'linkcheck', testroot='linkcheck-localserver', freshenv=True,
     confoverrides={'linkcheck_auth': [(r'^$', ('user1', 'password'))]})
 def test_auth_header_no_match(app):
-    records = []
-    with http_server(capture_headers_handler(records)):
+    with http_server(custom_handler(valid_credentials=("user1", "password"))):
         app.build()
 
-    stdout = "\n".join(records)
-    assert "Authorization" not in stdout
+    with open(app.outdir / "output.json", encoding="utf-8") as fp:
+        content = json.load(fp)
+
+    # TODO: should this test's webserver return HTTP 401 here?
+    # https://github.com/sphinx-doc/sphinx/issues/11433
+    assert content["info"] == "403 Client Error: Forbidden for url: http://localhost:7777/"
+    assert content["status"] == "broken"
 
 
 @pytest.mark.sphinx(
@@ -246,14 +279,20 @@ def test_auth_header_no_match(app):
         },
     }})
 def test_linkcheck_request_headers(app):
-    records = []
-    with http_server(capture_headers_handler(records)):
+    def check_headers(self):
+        if "X-Secret" in self.headers:
+            return False
+        if self.headers["Accept"] != "text/html":
+            return False
+        return True
+
+    with http_server(custom_handler(success_criteria=check_headers)):
         app.build()
 
-    stdout = "\n".join(records)
-    assert "Accept: text/html\n" in stdout
-    assert "X-Secret" not in stdout
-    assert "sesami" not in stdout
+    with open(app.outdir / "output.json", encoding="utf-8") as fp:
+        content = json.load(fp)
+
+    assert content["status"] == "working"
 
 
 @pytest.mark.sphinx(
@@ -263,14 +302,20 @@ def test_linkcheck_request_headers(app):
         "*": {"X-Secret": "open sesami"},
     }})
 def test_linkcheck_request_headers_no_slash(app):
-    records = []
-    with http_server(capture_headers_handler(records)):
+    def check_headers(self):
+        if "X-Secret" in self.headers:
+            return False
+        if self.headers["Accept"] != "application/json":
+            return False
+        return True
+
+    with http_server(custom_handler(success_criteria=check_headers)):
         app.build()
 
-    stdout = "\n".join(records)
-    assert "Accept: application/json\n" in stdout
-    assert "X-Secret" not in stdout
-    assert "sesami" not in stdout
+    with open(app.outdir / "output.json", encoding="utf-8") as fp:
+        content = json.load(fp)
+
+    assert content["status"] == "working"
 
 
 @pytest.mark.sphinx(
@@ -280,13 +325,20 @@ def test_linkcheck_request_headers_no_slash(app):
         "*": {"X-Secret": "open sesami"},
     }})
 def test_linkcheck_request_headers_default(app):
-    records = []
-    with http_server(capture_headers_handler(records)):
+    def check_headers(self):
+        if self.headers["X-Secret"] != "open sesami":
+            return False
+        if self.headers["Accept"] == "application/json":
+            return False
+        return True
+
+    with http_server(custom_handler(success_criteria=check_headers)):
         app.build()
 
-    stdout = "\n".join(records)
-    assert "Accepts: application/json\n" not in stdout
-    assert "X-Secret: open sesami\n" in stdout
+    with open(app.outdir / "output.json", encoding="utf-8") as fp:
+        content = json.load(fp)
+
+    assert content["status"] == "working"
 
 
 def make_redirect_handler(*, support_head):
@@ -394,7 +446,9 @@ def do_GET(self):
 def test_invalid_ssl(app):
     # Link indicates SSL should be used (https) but the server does not handle it.
     with http_server(OKHandler):
-        app.build()
+        with mock.patch("sphinx.builders.linkcheck.requests.get", wraps=requests.get) as get_request:
+            app.build()
+            assert not get_request.called
 
     with open(app.outdir / 'output.json', encoding='utf-8') as fp:
         content = json.load(fp)
@@ -637,14 +691,14 @@ class FakeResponse:
 def test_limit_rate_default_sleep(app):
     worker = HyperlinkAvailabilityCheckWorker(app.env, app.config, Queue(), Queue(), {})
     with mock.patch('time.time', return_value=0.0):
-        next_check = worker.limit_rate(FakeResponse())
+        next_check = worker.limit_rate(FakeResponse.url, FakeResponse.headers.get("Retry-After"))
     assert next_check == 60.0
 
 
 def test_limit_rate_user_max_delay(app):
     app.config.linkcheck_rate_limit_timeout = 0.0
     worker = HyperlinkAvailabilityCheckWorker(app.env, app.config, Queue(), Queue(), {})
-    next_check = worker.limit_rate(FakeResponse())
+    next_check = worker.limit_rate(FakeResponse.url, FakeResponse.headers.get("Retry-After"))
     assert next_check is None
 
 
@@ -653,7 +707,7 @@ def test_limit_rate_doubles_previous_wait_time(app):
     worker = HyperlinkAvailabilityCheckWorker(app.env, app.config, Queue(), Queue(),
                                               rate_limits)
     with mock.patch('time.time', return_value=0.0):
-        next_check = worker.limit_rate(FakeResponse())
+        next_check = worker.limit_rate(FakeResponse.url, FakeResponse.headers.get("Retry-After"))
     assert next_check == 120.0
 
 
@@ -663,7 +717,7 @@ def test_limit_rate_clips_wait_time_to_max_time(app):
     worker = HyperlinkAvailabilityCheckWorker(app.env, app.config, Queue(), Queue(),
                                               rate_limits)
     with mock.patch('time.time', return_value=0.0):
-        next_check = worker.limit_rate(FakeResponse())
+        next_check = worker.limit_rate(FakeResponse.url, FakeResponse.headers.get("Retry-After"))
     assert next_check == 90.0
 
 
@@ -672,7 +726,7 @@ def test_limit_rate_bails_out_after_waiting_max_time(app):
     rate_limits = {"localhost": RateLimit(90.0, 0.0)}
     worker = HyperlinkAvailabilityCheckWorker(app.env, app.config, Queue(), Queue(),
                                               rate_limits)
-    next_check = worker.limit_rate(FakeResponse())
+    next_check = worker.limit_rate(FakeResponse.url, FakeResponse.headers.get("Retry-After"))
     assert next_check is None
 
 
diff --git a/tests/test_intl.py b/tests/test_intl.py
index 54fa113c1..b34ce5e33 100644
--- a/tests/test_intl.py
+++ b/tests/test_intl.py
@@ -5,6 +5,7 @@
 
 import os
 import re
+import time
 
 import pygments
 import pytest
@@ -629,6 +630,7 @@ def get_number_of_update_targets(app_):
     # phase1: build document with non-gettext builder and generate mo file in srcdir
     app0 = make_app('dummy', *args, **kwargs)
     app0.build()
+    time.sleep(0.01)
     assert (app0.srcdir / 'xx' / 'LC_MESSAGES' / 'bom.mo').exists()
     # Since it is after the build, the number of documents to be updated is 0
     assert get_number_of_update_targets(app0) == 0
@@ -646,6 +648,7 @@ def get_number_of_update_targets(app_):
     # The mo file in the srcdir directory is retained.
     app = make_app('gettext', *args, **kwargs)
     app.build()
+    time.sleep(0.01)
     # Since it is after the build, the number of documents to be updated is 0
     assert get_number_of_update_targets(app) == 0
     # Even if the timestamp of the mo file is updated, the number of documents