Repository: tox-dev/tox

Issue:
tox4: ValueError when installing deps from requirements path containing spaces.
In tox4, attempting to install deps from requirements.txt files where the full path contains spaces fails with a `ValueError`.

This is induced by setting the `deps` option under `[testenv]` to `-r {toxinidir}/tests/requirements.txt`, where `toxinidir` expands to a path containing spaces. The issue is not limited to `toxinidir`, as providing the full path fails too. 

The traceback is:

```python
Traceback (most recent call last):
  File "/home/domdf/Python/01 GitHub Repos/03 Libraries/wordle/venv/lib/python3.8/site-packages/tox/session/cmd/run/single.py", line 41, in _evaluate
    tox_env.ensure_setup(recreate=recreate)
  File "/home/domdf/Python/01 GitHub Repos/03 Libraries/wordle/venv/lib/python3.8/site-packages/tox/tox_env/api.py", line 169, in ensure_setup
    self.setup()
  File "/home/domdf/Python/01 GitHub Repos/03 Libraries/wordle/venv/lib/python3.8/site-packages/tox/tox_env/python/runner.py", line 44, in setup
    self.install_deps()
  File "/home/domdf/Python/01 GitHub Repos/03 Libraries/wordle/venv/lib/python3.8/site-packages/tox/tox_env/python/runner.py", line 68, in install_deps
    requirement_file_content = requirements_file.validate_and_expand()
  File "/home/domdf/Python/01 GitHub Repos/03 Libraries/wordle/venv/lib/python3.8/site-packages/tox/tox_env/python/req_file.py", line 94, in validate_and_expand
    raise ValueError(line)
ValueError: -r /home/domdf/Python/01 GitHub Repos/03 Libraries/wordle/tests/requirements.txt
py36: FAIL ✖ in 0.42 seconds
  py36: FAIL code 2 (0.42 seconds)
  evaluation failed :( (0.49 seconds)

```

This works fine in tox 3.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/tox-dev/tox/pull/1828?src=pr&el=h1) Report
> Merging [#1828](https://codecov.io/gh/tox-dev/tox/pull/1828?src=pr&el=desc) (9a4204c) into [rewrite](https://codecov.io/gh/tox-dev/tox/commit/d9c5b9cc6c4aa6a4cd477b3dbb5b8e09efcb922a?el=desc) (d9c5b9c) will **increase** coverage by `0.13%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/tox-dev/tox/pull/1828/graphs/tree.svg?width=650&height=150&src=pr&token=DYodAwDCZ5)](https://codecov.io/gh/tox-dev/tox/pull/1828?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           rewrite    #1828      +/-   ##
===========================================
+ Coverage    99.80%   99.94%   +0.13%     
===========================================
  Files          133      132       -1     
  Lines         7195     7120      -75     
  Branches       713      693      -20     
===========================================
- Hits          7181     7116      -65     
+ Misses           3        2       -1     
+ Partials        11        2       -9     
```

| Flag | Coverage Δ | |
|---|---|---|
| tests | `99.94% <100.00%> (+0.13%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/tox-dev/tox/pull/1828?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/tox/tox\_env/python/req\_file.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC90b3hfZW52L3B5dGhvbi9yZXFfZmlsZS5weQ==) | `98.05% <100.00%> (-1.95%)` | :arrow_down: |
| [tests/tox\_env/python/test\_req\_file.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-dGVzdHMvdG94X2Vudi9weXRob24vdGVzdF9yZXFfZmlsZS5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/tox/tox\_env/api.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC90b3hfZW52L2FwaS5weQ==) | `100.00% <0.00%> (ø)` | |
| [src/tox/execute/request.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC9leGVjdXRlL3JlcXVlc3QucHk=) | `100.00% <0.00%> (ø)` | |
| [...ecute/local\_sub\_process/read\_via\_thread\_windows.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC9leGVjdXRlL2xvY2FsX3N1Yl9wcm9jZXNzL3JlYWRfdmlhX3RocmVhZF93aW5kb3dzLnB5) | | |
| [src/tox/tox\_env/python/api.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC90b3hfZW52L3B5dGhvbi9hcGkucHk=) | `100.00% <0.00%> (+0.68%)` | :arrow_up: |
| [src/tox/config/loader/str\_convert.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC9jb25maWcvbG9hZGVyL3N0cl9jb252ZXJ0LnB5) | `100.00% <0.00%> (+1.47%)` | :arrow_up: |
| [src/tox/util/spinner.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC91dGlsL3NwaW5uZXIucHk=) | `100.00% <0.00%> (+2.30%)` | :arrow_up: |
| [src/tox/execute/local\_sub\_process/\_\_init\_\_.py](https://codecov.io/gh/tox-dev/tox/pull/1828/diff?src=pr&el=tree#diff-c3JjL3RveC9leGVjdXRlL2xvY2FsX3N1Yl9wcm9jZXNzL19faW5pdF9fLnB5) | `98.51% <0.00%> (+3.03%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/tox-dev/tox/pull/1828?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/tox-dev/tox/pull/1828?src=pr&el=footer). Last update [d9c5b9c...9a4204c](https://codecov.io/gh/tox-dev/tox/pull/1828?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/docs/changelog/1792.bugfix.rst b/docs/changelog/1792.bugfix.rst
new file mode 100644
index 00000000..550172ba
--- /dev/null
+++ b/docs/changelog/1792.bugfix.rst
@@ -0,0 +1,2 @@
+When specifying requirements/editable/constraint paths within ``deps`` escape space, unless already escaped to support
+running specifying transitive requirements files within deps - by :user:`gaborbernat`.
diff --git a/docs/changelog/1828.bugfix.rst b/docs/changelog/1828.bugfix.rst
new file mode 100644
index 00000000..96ce0b8a
--- /dev/null
+++ b/docs/changelog/1828.bugfix.rst
@@ -0,0 +1,2 @@
+Raise ``ValueError`` with descriptive message when a requirements file specified does not exist
+- by :user:`gaborbernat`.
diff --git a/src/tox/tox_env/python/req_file.py b/src/tox/tox_env/python/req_file.py
index 2796e492..cd690646 100644
--- a/src/tox/tox_env/python/req_file.py
+++ b/src/tox/tox_env/python/req_file.py
@@ -56,13 +56,21 @@ class RequirementsFile:
         ],
     }
 
-    def __init__(self, raw: str, allow_short_req_file: bool = True, root: Optional[Path] = None) -> None:
+    def __init__(self, raw: str, within_tox_ini: bool = True, root: Optional[Path] = None) -> None:
         self._root = Path().cwd() if root is None else root
-        if allow_short_req_file:  # patch for tox<4 supporting requirement/constraint files via -rreq.txt/-creq.txt
+        if within_tox_ini:  # patch the content coming from tox.ini
             lines: List[str] = []
             for line in raw.splitlines():
+                # for tox<4 supporting requirement/constraint files via -rreq.txt/-creq.txt
                 if len(line) >= 3 and (line.startswith("-r") or line.startswith("-c")) and not line[2].isspace():
                     line = f"{line[:2]} {line[2:]}"
+                # escape spaces
+                escape_for = ("-c", "--constraint", "-r", "--requirement", "-f", "--find-links" "-e", "--editable")
+                escape_match = next((e for e in escape_for if line.startswith(e) and line[len(e)].isspace()), None)
+                if escape_match is not None:
+                    # escape not already escaped spaces
+                    escaped = re.sub(r"(?<!\\)(\s)", r"\\\1", line[len(escape_match) + 1 :])
+                    line = f"{line[:len(escape_match)]} {escaped}"
                 lines.append(line)
             adjusted = "\n".join(lines)
             raw = f"{adjusted}\n" if raw.endswith("\\\n") else adjusted  # preserve trailing newline if input has it
@@ -86,7 +94,7 @@ class RequirementsFile:
             if not line:
                 continue
             if line.startswith("-"):  # handle flags
-                words = re.findall(r"\S+", line)
+                words = [i for i in re.split(r"(?<!\\)\s", line) if i]
                 first = words[0]
                 if first in self.VALID_OPTIONS["no_arg"]:
                     if len(words) != 1:
@@ -98,10 +106,14 @@ class RequirementsFile:
                         raise ValueError(line)
                     else:
                         if first in ("-r", "--requirement", "-c", "--constraint"):
-                            path = Path(line[len(first) + 1 :].strip())
+                            raw_path = line[len(first) + 1 :].strip()
+                            unescaped_path = re.sub(r"\\(\s)", r"\1", raw_path)
+                            path = Path(unescaped_path)
                             if not path.is_absolute():
                                 path = ini_dir / path
-                            req_file = RequirementsFile(path.read_text(), allow_short_req_file=False, root=self.root)
+                            if not path.exists():
+                                raise ValueError(f"requirement file path {str(path)!r} does not exist")
+                            req_file = RequirementsFile(path.read_text(), within_tox_ini=False, root=self.root)
                             result.extend(req_file.validate_and_expand())
                         else:
                             result.append(" ".join(words))


Test patch:
diff --git a/tests/tox_env/python/test_req_file.py b/tests/tox_env/python/test_req_file.py
index 970c47cb..cb112178 100644
--- a/tests/tox_env/python/test_req_file.py
+++ b/tests/tox_env/python/test_req_file.py
@@ -126,7 +126,6 @@ def test_requirements_txt_transitive(tmp_path: Path, flag: str) -> None:
     "raw",
     [
         "--pre something",
-        "-r one two",
         "--missing",
         "-k",
         "magic+https://git.example.com/MyProject#egg=MyProject",
@@ -138,6 +137,12 @@ def test_bad_line(raw: str) -> None:
         req.validate_and_expand()
 
 
+def test_requirements_file_missing() -> None:
+    req = RequirementsFile("-r one two")
+    with pytest.raises(ValueError, match="requirement file path '.*one two' does not exist"):
+        req.validate_and_expand()
+
+
 def test_legacy_requirement_file(tmp_path: Path, monkeypatch: MonkeyPatch) -> None:
     monkeypatch.chdir(tmp_path)
     (tmp_path / "requirement.txt").write_text("a")
@@ -160,3 +165,19 @@ def test_constraint_txt_expanded(tmp_path: Path, flag: str) -> None:
     assert req.validate_and_expand() == ["magic", "magical"]
     with req.with_file() as filename:
         assert filename.read_text() == f"{flag} other.txt"
+
+
+@pytest.mark.parametrize("escape_upfront", [True, False])
+def test_req_path_with_space(tmp_path: Path, escape_upfront: bool) -> None:
+    req_file = tmp_path / "a b"
+    req_file.write_text("c")
+    path = f"-r {str(req_file)}"
+    if escape_upfront:
+        path = f'{path[:-len("a b")]}a\\ b'
+    req = RequirementsFile(path)
+
+    # must be escaped within the requirements file
+    assert "a\\ b" in str(req)
+
+    # but still unroll during transitive dependencies
+    assert req.validate_and_expand() == ["c"]