Repository: sphinx-doc/sphinx

Issue:
Support `linkcheck_ignore` in link redirection
**Is your feature request related to a problem? Please describe.**
Specifying a domain in `linkcheck_ignore` works well for links containing this domain but it doesnn't for links which redirect to a link to the domain to be ignored.
For example, the following configuration:
```python
linkcheck_ignore = [
    "https://onlinelibrary.wiley.com",  # 403 Client Error: Forbidden for url
]
```
works perfectly for links like `https://onlinelibrary.wiley.com/doi/10.1002/jemt.20597` but not for `https://doi.org/10.1002/jemt.20597`, which redirect to `https://onlinelibrary.wiley.com/doi/10.1002/jemt.20597`

**Describe the solution you'd like**
The `linkcheck_ignore` configuration parameters should also apply to redirect links.

**Additional context**
See for example https://github.com/hyperspy/hyperspy/pull/3108. This typically happen for DOI links, which are by design permanent url and redirect to urls which can changed. In this case, the DOI should be used in favour of the redirect url however, the `linkcheck_ignore` will not be effective on the redirect url.



Discussion:
Sounds very reasonable to me. When a user expects all links to a domain (or path) to be ignored, linkcheck should also ignore the redirections pointing to that domain.
I tried to understand why Wiley URLs have this problem but was not successful, so I asked for help upstream:  https://github.com/psf/requests/issues/6471

Meanwhile having the option to ignore specific redirect sites of DOI links would a good idea.

<details><summary>Here is what I have tried here if someone is interested</summary>
If I for instance visit `https://doi.org/10.1002/jccs.200600142` with my browser, everything is fine. But both requests and Sphinx fail:

```sh
python -c "import requests; print(requests.head('https://doi.org/10.1002/jccs.200600142', allow_redirects=True)); import sphinx.util.requests; print(sphinx.util.requests.head('https://doi.org/10.1002/jccs.200600142', allow_redirects=True))"
<Response [403]>
<Response [403]>
```

I also tried accepting cookies and changing the user-agent, which also did not help:
```py
import requests
with requests.Session() as s:
    print(s.get('https://doi.org/10.1002/jccs.200600142', allow_redirects=True, headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0'}))
```

I posted this upstream: https://github.com/psf/requests/issues/6471
</details>
Unsure what problem Wiley URLs have? They are always replying with a 403. The issue here is that developers can instruct linkcheck to ignore URLs matching a regexp pattern, but that URL is only ignored if it appears in the documentation, not if it comes as the result of a redirect.

So:
```python
# conf.py
linkcheck_ignore = [
    "https://onlinelibrary.wiley.com",  # 403 Client Error: Forbidden for url
]
```
```rst
.. doc.rst
.. this link is ignored by linkcheck, it matches the pattern from linkcheck_ignore
`direct link <https://onlinelibrary.wiley.com/doi/10.1002/jemt.20597>`_
.. doi.org redirects to Wiley, but linkcheck does not check the linkcheck_ignore during the redirection chain, so the following shows as a broken link for linkcheck
`indirect link <https://doi.org/10.1002/jemt.20597>`_
I'm interested in implementing this feature, but there's a detail about the existing implementation that I think is important to consider first:

The Sphinx `linkcheck` builder currently uses the `requests` HTTP client to handle following of redirects, and so the logic to determine whether one-or-more-redirects will be followed is encapsulated by that library, and a naive application of `linkcheck_ignore` rules would only apply after a final response URL is resolved (or a timeout occurs, or an error code is returned, etc).

My intuition for a feature like this is that we'd ideally want to ignore redirections as soon as they suggest navigating through an ignored path -- that is, we'd follow the initial hyperlink, and if it tells us to go to a known-ignorable URL, we'd stop immediately and return the `ignore` status code for that link.  I believe that'd be preferable because it'd imply that we'd generate less network traffic, we'd spend less time linkchecking, and we wouldn't attempt to initiate traffic to domains we've configured as ignored.

To do so, however, we'd probably want to adjust Sphinx's `linkchecker` so that `requests` doesn't follow redirects on its behalf.  That's OK... but it potentially has other consequences; we'd want to enforce a maximum-redirect limit (similar to the long-standing [`requests.sessions.DEFAULT_REDIRECT_LIMIT`](https://github.com/psf/requests/blob/22701d149ad9585cc01b3f9bda4e78cd77ffb996/requests/sessions.py#L34)), for example.  And `requests` itself will have had a wealth of experience dealing with unusual behaviours and bugreports related to redirects.  So... as a maintainer I'm initially a bit cautious (an understatement) about this.

We _could_ perhaps apply the naive solution, and ignore URLs after `requests` has followed all redirects.  Perhaps that's OK - I just don't love the idea that some traffic may in fact navigate through ignored URLs.  Hiding those by marking them as `ignored` -- even though they weren't -- feels potentially worse than doing nothing.



Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 1d61320343c..12646e59989 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -40,6 +40,8 @@ Bugs fixed
   Patch by Jean-François B.
 * #13096: HTML Search: check that query terms exist as properties in
   term indices before accessing them.
+* #11233: linkcheck: match redirect URIs against :confval:`linkcheck_ignore` by
+  overriding session-level ``requests.get_redirect_target``.
 
 Testing
 -------
diff --git a/doc/usage/configuration.rst b/doc/usage/configuration.rst
index 69c9ccaa143..6b23825a3fb 100644
--- a/doc/usage/configuration.rst
+++ b/doc/usage/configuration.rst
@@ -3709,6 +3709,9 @@ and which failures and redirects it ignores.
    A list of regular expressions that match URIs that should not be checked
    when doing a ``linkcheck`` build.
 
+   Server-issued redirects that match :confval:`ignored URIs <linkcheck_ignore>`
+   will not be followed.
+
    Example:
 
    .. code-block:: python
diff --git a/sphinx/builders/linkcheck.py b/sphinx/builders/linkcheck.py
index 7e4ac98ec61..d1ad8692d83 100644
--- a/sphinx/builders/linkcheck.py
+++ b/sphinx/builders/linkcheck.py
@@ -398,7 +398,9 @@ def __init__(
         self.tls_verify = config.tls_verify
         self.tls_cacerts = config.tls_cacerts
 
-        self._session = requests._Session()
+        self._session = requests._Session(
+            _ignored_redirects=tuple(map(re.compile, config.linkcheck_ignore))
+        )
 
         super().__init__(daemon=True)
 
@@ -570,6 +572,14 @@ def _check_uri(self, uri: str, hyperlink: Hyperlink) -> _URIProperties:
                 error_message = str(err)
                 continue
 
+            except requests._IgnoredRedirection as err:
+                # A redirection to an ignored URI was attempted; report it appropriately
+                return (
+                    _Status.IGNORED,
+                    f'ignored redirect: {err.destination}',
+                    err.status_code,
+                )
+
             except HTTPError as err:
                 error_message = str(err)
 
diff --git a/sphinx/util/requests.py b/sphinx/util/requests.py
index a647bf5ba50..b439ce437e8 100644
--- a/sphinx/util/requests.py
+++ b/sphinx/util/requests.py
@@ -3,20 +3,34 @@
 from __future__ import annotations
 
 import warnings
-from typing import Any
-from urllib.parse import urlsplit
+from typing import TYPE_CHECKING
+from urllib.parse import urljoin, urlsplit
 
 import requests
 from urllib3.exceptions import InsecureRequestWarning
 
 import sphinx
 
+if TYPE_CHECKING:
+    import re
+    from collections.abc import Sequence
+    from typing import Any
+
+
 _USER_AGENT = (
     f'Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0 '
     f'Sphinx/{sphinx.__version__}'
 )
 
 
+class _IgnoredRedirection(Exception):
+    """Sphinx-internal exception raised when an HTTP redirect is ignored"""
+
+    def __init__(self, destination: str, status_code: int) -> None:
+        self.destination = destination
+        self.status_code = status_code
+
+
 def _get_tls_cacert(url: str, certs: str | dict[str, str] | None) -> str | bool:
     """Get additional CA cert for a specific URL."""
     if not certs:
@@ -50,6 +64,23 @@ def head(url: str, **kwargs: Any) -> requests.Response:
 
 
 class _Session(requests.Session):
+    _ignored_redirects: Sequence[re.Pattern[str]]
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        self._ignored_redirects = kwargs.pop('_ignored_redirects', ())
+        super().__init__(*args, **kwargs)
+
+    def get_redirect_target(self, resp: requests.Response) -> str | None:
+        """Overrides the default requests.Session.get_redirect_target"""
+        # do not follow redirections that match ignored URI patterns
+        if resp.is_redirect:
+            destination = urljoin(resp.url, resp.headers['location'])
+            if any(pat.match(destination) for pat in self._ignored_redirects):
+                raise _IgnoredRedirection(
+                    destination=destination, status_code=resp.status_code
+                )
+        return super().get_redirect_target(resp)
+
     def request(  # type: ignore[override]
         self,
         method: str,


Test patch:
diff --git a/tests/test_builders/test_build_linkcheck.py b/tests/test_builders/test_build_linkcheck.py
index d35672025be..f1cc1c88ae1 100644
--- a/tests/test_builders/test_build_linkcheck.py
+++ b/tests/test_builders/test_build_linkcheck.py
@@ -926,7 +926,7 @@ class InfiniteRedirectOnHeadHandler(BaseHTTPRequestHandler):
 
     def do_HEAD(self):
         self.send_response(302, 'Found')
-        self.send_header('Location', '/')
+        self.send_header('Location', '/redirected')
         self.send_header('Content-Length', '0')
         self.end_headers()
 
@@ -966,6 +966,55 @@ def test_TooManyRedirects_on_HEAD(app, monkeypatch):
     }
 
 
+@pytest.mark.sphinx('linkcheck', testroot='linkcheck-localserver')
+def test_ignore_local_redirection(app):
+    with serve_application(app, InfiniteRedirectOnHeadHandler) as address:
+        app.config.linkcheck_ignore = [f'http://{address}/redirected']
+        app.build()
+
+    with open(app.outdir / 'output.json', encoding='utf-8') as fp:
+        content = json.load(fp)
+    assert content == {
+        'code': 302,
+        'status': 'ignored',
+        'filename': 'index.rst',
+        'lineno': 1,
+        'uri': f'http://{address}/',
+        'info': f'ignored redirect: http://{address}/redirected',
+    }
+
+
+class RemoteDomainRedirectHandler(InfiniteRedirectOnHeadHandler):
+    protocol_version = 'HTTP/1.1'
+
+    def do_GET(self):
+        self.send_response(301, 'Found')
+        if self.path == '/':
+            self.send_header('Location', '/local')
+        elif self.path == '/local':
+            self.send_header('Location', 'http://example.test/migrated')
+        self.send_header('Content-Length', '0')
+        self.end_headers()
+
+
+@pytest.mark.sphinx('linkcheck', testroot='linkcheck-localserver')
+def test_ignore_remote_redirection(app):
+    with serve_application(app, RemoteDomainRedirectHandler) as address:
+        app.config.linkcheck_ignore = ['http://example.test']
+        app.build()
+
+    with open(app.outdir / 'output.json', encoding='utf-8') as fp:
+        content = json.load(fp)
+    assert content == {
+        'code': 301,
+        'status': 'ignored',
+        'filename': 'index.rst',
+        'lineno': 1,
+        'uri': f'http://{address}/',
+        'info': 'ignored redirect: http://example.test/migrated',
+    }
+
+
 def make_retry_after_handler(
     responses: list[tuple[int, str | None]],
 ) -> type[BaseHTTPRequestHandler]: