Repository: python-poetry/poetry

Issue:
Poetry does not properly resolve install-requires for certain GIT packages
<!--
    Hi there! Thank you for submitting a bug report!

    Before you submit your issue, please review and follow the instructions at
    https://python-poetry.org/docs/contributing/#reporting-bugs. Fully fill out the template below; insufficient
    information or bad reproduction instructions will impair the ability of others to help you.
-->

<!-- All the below information must be provided for others to understand and help with your issue. -->
- **Poetry version**: 1.7.1
- **Python version**: 3.8.18
- **OS version and name**: Ubuntu
- **pyproject.toml**: 
```
[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = ""
authors = ["Romain <romain@email.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.8"
clip = {git = "https://github.com/openai/CLIP.git"}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```
<!-- All the below steps should be completed before submitting your issue. Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) stable Poetry version, installed using a [recommended method](https://python-poetry.org/docs/#installation).
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] I have consulted the [FAQ](https://python-poetry.org/docs/faq/) and [blog](https://python-poetry.org/blog/) for any relevant entries or release notes.
- [ ] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option) and have included the output below.

## Issue
The issue is that when determining the install-requires dependency, it analyzes the setup.py file and notices that there is a install-requires but cannot parse it (it's a list comprehension and not a list). The end result is that this is then silently ignored here: https://github.com/python-poetry/poetry/blob/master/src/poetry/inspection/info.py#L565 because it doesn't check if install-requires is filled (and in fairness, shouldn't since some packages don't have that).

The crux of the issue is basically that in the SetupReader, there is no distinction between "value absent" and "value can't be read".

The proper behavior is probably to fallback to building the package if allowed in the case when the field exists but can't be extracted.

Steps to reproduce:
```
poetry new poetry-demo
cd poetry-demo
poetry add "git+https://github.com/openai/CLIP.git"
```

If you look at the CLIP project, you will see it has some requirements like torch but the poetry lock file lists no such thing and looks like this: 
```
[[package]]
name = "clip"
version = "1.0"
description = ""
optional = false
python-versions = "*"
files = []
develop = false

[package.extras]
dev = ["pytest"]

[package.source]
type = "git"
url = "https://github.com/openai/CLIP.git"
reference = "HEAD"
resolved_reference = "a1d071733d7111c9c014f024669f959182114e33"

[metadata]
lock-version = "2.0"
python-versions = "^3.8"
content-hash = "ca6e63d2418191bea92517a4e1a4b2427c67dbcfe40147d16556cd0c674f34c3"
```



Discussion:
Transferring from https://github.com/python-poetry/poetry/issues/4325#issuecomment-1949880958:

Here, we only use the information from setup.py if it is "complete" (otherwise we do a build):
https://github.com/python-poetry/poetry/blob/48a76fd1e27a7e329b8987407c409ee3eb28347c/src/poetry/inspection/info.py#L583-L586

So the question is why don't we try a build even though we were not able to parse `install_requires`? That's because we were able to parse `extra_require` and both are merged to `requires_dist`. Thus, we could improve the heuristics.

Nevertheless, it might make more sense to convince projects to specifiy their dependencies in a [more static way](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#declaring-required-dependency) when using setuptools.
Would it be reasonable to add a field to the git dependency specification that indicates "hey, this repo does some funky stuff, skip the heuristics, just build it and get the dependencies and other metadata from the output, I'm okay with that being slower" ?

While I agree that static configuration is "the right way (tm)", at least from my experience, a large chunk of the ecosystem hasn't, and has no interest in switching. The pragmatic approach would be to have some sort of "escape hatch" like this. Otherwise, poetry is basically unusable if one has to fork such a project.
Unnecessary complexity.  If this is something you're interested in, it would be just as simple to fix it.

Somewhere round about [here](https://github.com/python-poetry/poetry/blob/e015ee534eb09c7d0add6a8440c944ff132ff6f4/src/poetry/utils/setup_reader.py#L224), poetry is just ignoring things that it doesn't understand.

Instead have it signal that it can't understand them: by returning `None` or throwing an exception, or whatever.
I will readily admit that I am a total novice to poetry's code base but in general I feel that having heuristics that "produce the wrong result" is probably not a great behavior. I agree with @dimbleby that at a minimum, a failure to parse should raise an exception (it would at least inform the user that poetry cannot be used safely/correctly with that repository). My personal preference though would be:
- use heuristics to see if you can skip building
- if heuristics fail (it should be detectable) fallback to building the package

It does seem that right now, in *some* cases, the package is built just not in all failure modes. I think this is what @radoering is saying -- ie: the heuristic is failing to detect that it couldn't parse the information properly. The behavior of "fallback to build" is the right one imho, but failure cases need to be more properly detected.

The code already falls back to building, if reading `setup.py` fails.  It should be straightforward to arrange that the fix I suggest has the desired behaviour.

But this is open source: someone who wants this has to do the work!
To be frank, relying on heuristics while demanding others rework their project to use static configuration for reliability and repeatability seems at least a little hypocritical, especially when the justification for both is "it works now, why change?". Heuristics are great up until they don't do what you want or do what you don't want. Being able to statically configure what you expect from poetry, and have it fail if it can't follow through seems like the best choice. The problem with heuristics is that there will always be edge cases that you can't predict, and there is always the possibility that a user might want to force a build for reasons that cannot be conceived of right now. If it failed here, even with this fix, what gives you confidence it won't fail in other ways? Simplicity is great, unless you're simply wrong.
This isn't a question of heuristics, it's a straight bug.  Poetry is trying to read setup.py, thinks it knows what it is doing, and it gets it wrong.

If this is important to you, please submit a pull request.

Resolving patch:
diff --git a/src/poetry/utils/setup_reader.py b/src/poetry/utils/setup_reader.py
--- a/src/poetry/utils/setup_reader.py
+++ b/src/poetry/utils/setup_reader.py
@@ -14,6 +14,10 @@
     from pathlib import Path
 
 
+class SetupReaderError(Exception):
+    pass
+
+
 class SetupReader:
     """
     Class that reads a setup.py file without executing it.
@@ -192,113 +196,95 @@ def _find_sub_setup_call(
         return None
 
     def _find_install_requires(self, call: ast.Call, body: list[ast.stmt]) -> list[str]:
-        install_requires: list[str] = []
         value = self._find_in_call(call, "install_requires")
         if value is None:
             # Trying to find in kwargs
             kwargs = self._find_call_kwargs(call)
 
             if kwargs is None or not isinstance(kwargs, ast.Name):
-                return install_requires
+                return []
 
             variable = self._find_variable_in_body(body, kwargs.id)
-            if not isinstance(variable, (ast.Dict, ast.Call)):
-                return install_requires
-
-            if isinstance(variable, ast.Call):
-                if not isinstance(variable.func, ast.Name):
-                    return install_requires
 
-                if variable.func.id != "dict":
-                    return install_requires
+            if isinstance(variable, ast.Dict):
+                value = self._find_in_dict(variable, "install_requires")
 
+            elif (
+                isinstance(variable, ast.Call)
+                and isinstance(variable.func, ast.Name)
+                and variable.func.id == "dict"
+            ):
                 value = self._find_in_call(variable, "install_requires")
+
             else:
-                value = self._find_in_dict(variable, "install_requires")
+                raise SetupReaderError(f"Cannot handle variable {variable}")
 
         if value is None:
-            return install_requires
+            return []
 
-        if isinstance(value, ast.List):
-            for el in value.elts:
-                if isinstance(el, ast.Constant) and isinstance(el.value, str):
-                    install_requires.append(el.value)
-        elif isinstance(value, ast.Name):
-            variable = self._find_variable_in_body(body, value.id)
+        if isinstance(value, ast.Name):
+            value = self._find_variable_in_body(body, value.id)
 
-            if variable is not None and isinstance(variable, ast.List):
-                for el in variable.elts:
-                    if isinstance(el, ast.Constant) and isinstance(el.value, str):
-                        install_requires.append(el.value)
+        if isinstance(value, ast.Constant) and value.value is None:
+            return []
 
-        return install_requires
+        if isinstance(value, ast.List):
+            return string_list_values(value)
+
+        raise SetupReaderError(f"Cannot handle value of type {type(value)}")
 
     def _find_extras_require(
         self, call: ast.Call, body: list[ast.stmt]
     ) -> dict[str, list[str]]:
-        extras_require: dict[str, list[str]] = {}
         value = self._find_in_call(call, "extras_require")
         if value is None:
             # Trying to find in kwargs
             kwargs = self._find_call_kwargs(call)
 
             if kwargs is None or not isinstance(kwargs, ast.Name):
-                return extras_require
+                return {}
 
             variable = self._find_variable_in_body(body, kwargs.id)
-            if not isinstance(variable, (ast.Dict, ast.Call)):
-                return extras_require
-
-            if isinstance(variable, ast.Call):
-                if not isinstance(variable.func, ast.Name):
-                    return extras_require
-
-                if variable.func.id != "dict":
-                    return extras_require
+            if isinstance(variable, ast.Dict):
+                value = self._find_in_dict(variable, "extras_require")
 
+            elif (
+                isinstance(variable, ast.Call)
+                and isinstance(variable.func, ast.Name)
+                and variable.func.id == "dict"
+            ):
                 value = self._find_in_call(variable, "extras_require")
+
             else:
-                value = self._find_in_dict(variable, "extras_require")
+                raise SetupReaderError(f"Cannot handle variable {variable}")
 
         if value is None:
-            return extras_require
+            return {}
+
+        if isinstance(value, ast.Name):
+            value = self._find_variable_in_body(body, value.id)
+
+        if isinstance(value, ast.Constant) and value.value is None:
+            return {}
 
         if isinstance(value, ast.Dict):
+            extras_require: dict[str, list[str]] = {}
             val: ast.expr | None
             for key, val in zip(value.keys, value.values):
                 if not isinstance(key, ast.Constant) or not isinstance(key.value, str):
-                    continue
+                    raise SetupReaderError(f"Cannot handle key {key}")
 
                 if isinstance(val, ast.Name):
                     val = self._find_variable_in_body(body, val.id)
 
-                if isinstance(val, ast.List):
-                    extras_require[key.value] = [
-                        e.value
-                        for e in val.elts
-                        if isinstance(e, ast.Constant) and isinstance(e.value, str)
-                    ]
-        elif isinstance(value, ast.Name):
-            variable = self._find_variable_in_body(body, value.id)
-
-            if variable is None or not isinstance(variable, ast.Dict):
-                return extras_require
-
-            for key, val in zip(variable.keys, variable.values):
-                if not isinstance(key, ast.Constant) or not isinstance(key.value, str):
-                    continue
+                if not isinstance(val, ast.List):
+                    raise SetupReaderError(f"Cannot handle value of type {type(val)}")
 
-                if isinstance(val, ast.Name):
-                    val = self._find_variable_in_body(body, val.id)
+                extras_require[key.value] = string_list_values(val)
 
-                if isinstance(val, ast.List):
-                    extras_require[key.value] = [
-                        e.value
-                        for e in val.elts
-                        if isinstance(e, ast.Constant) and isinstance(e.value, str)
-                    ]
+            return extras_require
 
-        return extras_require
+        raise SetupReaderError(f"Cannot handle value of type {type(value)}")
 
     def _find_single_string(
         self, call: ast.Call, body: list[ast.stmt], name: str
@@ -383,3 +369,15 @@ def _find_in_dict(self, dict_: ast.Dict, name: str) -> ast.expr | None:
                 return val
 
         return None
+
+
+def string_list_values(value: ast.List) -> list[str]:
+    strings = []
+    for element in value.elts:
+        if isinstance(element, ast.Constant) and isinstance(element.value, str):
+            strings.append(element.value)
+
+        else:
+            raise SetupReaderError("Found non-string element in list")
+
+    return strings


Test patch:
diff --git a/tests/inspection/test_info.py b/tests/inspection/test_info.py
--- a/tests/inspection/test_info.py
+++ b/tests/inspection/test_info.py
@@ -303,12 +303,9 @@ def test_info_setup_complex_pep517_legacy(
 def test_info_setup_complex_disable_build(
     mocker: MockerFixture, demo_setup_complex: Path
 ) -> None:
-    spy = mocker.spy(VirtualEnv, "run")
-    info = PackageInfo.from_directory(demo_setup_complex, disable_build=True)
-    assert spy.call_count == 0
-    assert info.name == "demo"
-    assert info.version == "0.1.0"
-    assert info.requires_dist is None
+    # Cannot extract install_requires from list comprehension.
+    with pytest.raises(PackageInfoError):
+        PackageInfo.from_directory(demo_setup_complex, disable_build=True)
 
 
 @pytest.mark.network