Repository: pypa/pip

Issue:
Recursive loops in requirements files not detected
### Description

If a requirements file references itself (directly or indirectly), an RecursionError is raised. 


### Expected behavior

The RecursionError isn't strictly *wrong*, but the cause of the error may not be obvious as it results in a hard interpreter crash. It would be desirable to detect the recursion, and report the source of the problem rather than crashing the Python interpreter.

### pip version

24.0

### Python version

3.10 (but should affect any version)

### OS

macOS (but should affect any OS)

### How to Reproduce

1. Construct a `requirements.txt` that contains:
```
-r requirements.txt
```
2. Run `pip install -r requirements.txt`

This is the simplest case; other loops (such as requirements files that include each other, and cycles of 3 or more requirements files) also cause the problem.

### Output

```shell
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 342, in run
    reqs = self.get_requirements(args, options, finder, session)
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 433, in get_requirements
    for parsed_req in parse_requirements(
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 156, in parse_requirements
    for parsed_line in parser.parse(filename, constraint):
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 337, in parse
    yield from self._parse_and_recurse(filename, constraint)
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 366, in _parse_and_recurse
    yield from self._parse_and_recurse(req_path, nested_constraint)
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 366, in _parse_and_recurse
    yield from self._parse_and_recurse(req_path, nested_constraint)
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 366, in _parse_and_recurse
    yield from self._parse_and_recurse(req_path, nested_constraint)
  [Previous line repeated 972 more times]
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 342, in _parse_and_recurse
    for line in self._parse_file(filename, constraint):
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 379, in _parse_file
    args_str, opts = self._line_parser(line)
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 398, in parse_line
    parser = build_parser()
  File "/Users/rkm/beeware/toga/venv3.10/lib/python3.10/site-packages/pip/_internal/req/req_file.py", line 444, in build_parser
    parser = optparse.OptionParser(add_help_option=False)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/optparse.py", line 1191, in __init__
    self.set_usage(usage)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/optparse.py", line 1266, in set_usage
    self.usage = _("%prog [options]")
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/gettext.py", line 742, in gettext
    return dgettext(_current_domain, message)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/gettext.py", line 666, in dgettext
    t = translation(domain, _localedirs.get(domain, None))
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/gettext.py", line 587, in translation
    mofiles = find(domain, localedir, languages, all=True)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/gettext.py", line 558, in find
    for nelang in _expand_lang(lang):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/gettext.py", line 213, in _expand_lang
    loc = locale.normalize(loc)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/locale.py", line 461, in normalize
    return _replace_encoding(code, encoding)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/locale.py", line 368, in _replace_encoding
    norm_encoding = encodings.normalize_encoding(encoding)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/encodings/__init__.py", line 55, in normalize_encoding
    if isinstance(encoding, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).

Discussion:
pradyunsg: > I am not sure if I am allowed to put out patches for issues with "needs triage" labels, but seems like simple fix and I got to scratch my own itch too.

It absolutely is! :)



Patch:
diff --git a/news/12653.feature.rst b/news/12653.feature.rst
new file mode 100644
index 000000000..83e1a46e6
--- /dev/null
+++ b/news/12653.feature.rst
@@ -0,0 +1,2 @@
+Detect recursively referencing requirements files and help users identify
+the source.
diff --git a/src/pip/_internal/req/req_file.py b/src/pip/_internal/req/req_file.py
index 53ad8674c..55c0726f3 100644
--- a/src/pip/_internal/req/req_file.py
+++ b/src/pip/_internal/req/req_file.py
@@ -324,11 +324,15 @@ class RequirementsFileParser:
     ) -> None:
         self._session = session
         self._line_parser = line_parser
+        self._parsed_files: dict[str, Optional[str]] = {}
 
     def parse(
         self, filename: str, constraint: bool
     ) -> Generator[ParsedLine, None, None]:
         """Parse a given file, yielding parsed lines."""
+        self._parsed_files[os.path.abspath(filename)] = (
+            None  # The primary requirements file passed
+        )
         yield from self._parse_and_recurse(filename, constraint)
 
     def _parse_and_recurse(
@@ -353,11 +357,25 @@ class RequirementsFileParser:
                 # original file and nested file are paths
                 elif not SCHEME_RE.search(req_path):
                     # do a join so relative paths work
-                    req_path = os.path.join(
-                        os.path.dirname(filename),
-                        req_path,
+                    # and then abspath so that we can identify recursive references
+                    req_path = os.path.abspath(
+                        os.path.join(
+                            os.path.dirname(filename),
+                            req_path,
+                        )
                     )
-
+                if req_path in self._parsed_files:
+                    initial_file = self._parsed_files[req_path]
+                    tail = (
+                        f" and again in {initial_file}"
+                        if initial_file is not None
+                        else ""
+                    )
+                    raise RequirementsFileParseError(
+                        f"{req_path} recursively references itself in {filename}{tail}"
+                    )
+                # Keeping a track where was each file first included in
+                self._parsed_files[req_path] = filename
                 yield from self._parse_and_recurse(req_path, nested_constraint)
             else:
                 yield line


Test patch:
diff --git a/tests/unit/test_req_file.py b/tests/unit/test_req_file.py
index 236b666fb..1ddccdee5 100644
--- a/tests/unit/test_req_file.py
+++ b/tests/unit/test_req_file.py
@@ -1,6 +1,7 @@
 import collections
 import logging
 import os
+import re
 import textwrap
 from optparse import Values
 from pathlib import Path
@@ -345,6 +346,63 @@ class TestProcessLine:
         assert reqs[0].name == req_name
         assert reqs[0].constraint
 
+    def test_recursive_requirements_file(
+        self, tmpdir: Path, session: PipSession
+    ) -> None:
+        req_files: list[Path] = []
+        req_file_count = 4
+        for i in range(req_file_count):
+            req_file = tmpdir / f"{i}.txt"
+            req_file.write_text(f"-r {(i+1) % req_file_count}.txt")
+            req_files.append(req_file)
+
+        # When the passed requirements file recursively references itself
+        with pytest.raises(
+            RequirementsFileParseError,
+            match=(
+                f"{re.escape(str(req_files[0]))} recursively references itself"
+                f" in {re.escape(str(req_files[req_file_count - 1]))}"
+            ),
+        ):
+            list(parse_requirements(filename=str(req_files[0]), session=session))
+
+        # When one of other the requirements file recursively references itself
+        req_files[req_file_count - 1].write_text(
+            # Just name since they are in the same folder
+            f"-r {req_files[req_file_count - 2].name}"
+        )
+        with pytest.raises(
+            RequirementsFileParseError,
+            match=(
+                f"{re.escape(str(req_files[req_file_count - 2]))} recursively"
+                " references itself in"
+                f" {re.escape(str(req_files[req_file_count - 1]))} and again in"
+                f" {re.escape(str(req_files[req_file_count - 3]))}"
+            ),
+        ):
+            list(parse_requirements(filename=str(req_files[0]), session=session))
+
+    def test_recursive_relative_requirements_file(
+        self, tmpdir: Path, session: PipSession
+    ) -> None:
+        root_req_file = tmpdir / "root.txt"
+        (tmpdir / "nest" / "nest").mkdir(parents=True)
+        level_1_req_file = tmpdir / "nest" / "level_1.txt"
+        level_2_req_file = tmpdir / "nest" / "nest" / "level_2.txt"
+
+        root_req_file.write_text("-r nest/level_1.txt")
+        level_1_req_file.write_text("-r nest/level_2.txt")
+        level_2_req_file.write_text("-r ../../root.txt")
+
+        with pytest.raises(
+            RequirementsFileParseError,
+            match=(
+                f"{re.escape(str(root_req_file))} recursively references itself in"
+                f" {re.escape(str(level_2_req_file))}"
+            ),
+        ):
+            list(parse_requirements(filename=str(root_req_file), session=session))
+
     def test_options_on_a_requirement_line(self, line_processor: LineProcessor) -> None:
         line = (
             'SomeProject --global-option="yo3" --global-option "yo4" '