Repository: jazzband/pip-tools

Issue:
Mysterious error with optional dependencies in pyproject.toml
I have been following the guide in the README file on how to use pip-tools with pyproject.toml. Unfortunately, if I add a ``project.optional-dependencies`` section to pyproject.toml, *regardless of what its contents are*, I get the following error:

```
(base) PS C:\Users\adewar\code\Solidity-GUI> pip-compile -v pyproject.toml
C:\ProgramData\Miniconda3\lib\site-packages\_distutils_hack\__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Creating virtualenv isolated environment...
find interpreter for spec PythonSpec(path=C:\ProgramData\Miniconda3\python.exe)
proposed PythonInfo(spec=CPython3.9.13.final.0-64, exe=C:\ProgramData\Miniconda3\python.exe, platform=win32, version='3.9.13 (main, Oct 13 2022, 21:23:06) [MSC v.1916 64 bit (AMD64)]', encoding_fs_io=utf-8-utf-8)
create virtual environment via CPython3Windows(dest=C:\Users\adewar\AppData\Local\Temp\build-env-m1awcc93, clear=False, no_vcs_ignore=False, global=False)
add seed packages via FromAppData(download=False, pip=bundle, via=copy, app_data_dir=C:\Users\adewar\AppData\Local\pypa\virtualenv)
Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
Getting build dependencies for wheel...
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse C:\Users\adewar\code\Solidity-GUI\pyproject.toml
```

#### Environment Versions

1. Windows 10 (replicated on Arch Linux)
1. Python version: `python 3.9.13`
1. pip version: `pip 22.2.2 from C:\ProgramData\Miniconda3\lib\site-packages\pip (python 3.9)`
1. pip-tools version: `pip-compile, version 6.9.0`

#### Steps to replicate

1. Add a `project.optional-dependencies` section to your `pyproject.toml`
2. Run `pip-compile pyproject.toml`



Discussion:
I've dug in a bit more and it's weirder than I thought. This following pyproject.toml doesn't work:

```
[project]
name = "SolidityGUI"
version = "0.1"
authors = ["author1"]
dependencies = ["pygmsh", "pyside6", "vtk"]

[project.optional-dependencies]
dev = ["black"]
```

However, if you remove the ``authors`` line, it works fine again:

```
[project]
name = "SolidityGUI"
version = "0.1"
dependencies = ["pygmsh", "pyside6", "vtk"]

[project.optional-dependencies]
dev = ["black"]
```

How odd is that?!
Thanks! Can you post the entire `pyproject.toml`, including the `build-system` section?
That *is* the entire ``pyproject.toml``. Is the ``build-system`` section a requirement that I'm missing...?
It usually *should* be there if it's describing an installable package, but I'm not sure if it ought to be strictly mandatory here. Looking at [PEP 518](https://peps.python.org/pep-0518/#build-system-table):

> For the vast majority of Python projects that rely upon setuptools, the `pyproject.toml` file will be:

```toml
[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel"]  # PEP 508 specifications.
```
> ...

> Tools should not require the existence of the `[build-system]` table. A `pyproject.toml` file may be used to store configuration details other than build-related data and thus lack a `[build-system]` table legitimately. If the file exists but is lacking the `[build-system]` table then the default values as specified above should be used. If the table is specified but is missing required fields then the tool should consider it an error.

My thought at the moment is that if you add the above sample section and it works, then this is a bug, since when omitted

> . . . the default values as specified above should be used.
A colleague has tracked down the source of the problem. If you check ``pyproject.toml`` with [validate-pyproject](https://pypi.org/project/validate-pyproject) you get:
```
[ERROR] project.authors[{data__authors_x}] must be object
```

Putting the ``authors`` field into the correct format fixes things, i.e.:
```py
authors = [{ name = "author1", email = "author1@email.com" }]
```
I'm glad you've found the root of the issue. I'll close this since there's no action required regarding pip-tools. Thanks for the issue!
Thanks for your help @atugushev. It may still be worth opening a separate issue re having better error reporting in the case where the ``pyproject.toml`` file is malformed, but obviously it's a less important problem.
```
$ pip-compile pyproject.toml
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /private/var/folders/l0/lnq1ghps5yqc4vkgszlcz92m0000gp/T/tmp.X3gWYzsJ/foo/pyproject.toml
```

If you think that this could be improved on the [build](https://github.com/pypa/build/blob/7b002bbc48e965fd677d51ea88b1adc276ba506d/src/build/__init__.py#L302) side feel free to report on https://github.com/pypa/build/issues.
Ouch, sorry. This is the pip-tools' message:

https://github.com/jazzband/pip-tools/blob/eff84765725fc15579d6626851910350d580ec8b/piptools/scripts/compile.py#L479

---

`pip install .`:

```
      ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
      configuration error: `project.authors[{data__authors_x}]` must be object
```

---

`python -m build`:

```
ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
configuration error: `project.authors[{data__authors_x}]` must be object

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist
```


> It may still be worth opening a separate issue re having better error reporting in the case where the `pyproject.toml` file is malformed, but obviously it's a less important problem.

Agreed. Feel free to open a separate issue on the pip-tools tracker.
FYI, there is a tracking issue https://github.com/jazzband/pip-tools/issues/1583
I've stumbled upon this issue a couple of times in the last month. Here is the error I see:

```shell
$ pip-compile pyproject.toml -v
Creating venv isolated environment...
Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
Getting build dependencies for wheel...
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/OSS/opensearch-logger/pyproject.toml
```

## Setup

Environment:
* Python 3.11
* MacOS 13.0.1
* arm64

The `pyproject.toml` file itself is super simple and passes validation using the `validate-pyproject` package:

```toml
[build-system]
requires = [
    "flit_core >=3.2,<4",
]
build-backend = "flit_core.buildapi"

[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]
```

## What causes it

It's definitely not a `pip-compile` problem as it just calls `project_wheel_metadata` from the `build` package and then catches the error and reports. 

https://github.com/jazzband/pip-tools/blob/eff84765725fc15579d6626851910350d580ec8b/piptools/scripts/compile.py#L474-L476

The `build` package, in turn, does lots of stuff and ends up calling a Python script called `_in_process.py` from the pep517 package.

I have no idea how it is supposed to work, but it looks like the `build` package create a one-time temporary virtual environment in which this `_in_process.py` script is then invoked. Here is the command with which `build` package tries to call a subprocess in my case:

```python
['/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-rj6bf2id/bin/python', '/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpb6wdunjz']
```

Obviously, when something wrong happens in that subprocess script all we get at the end is:

```
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/Playground/pip-tools-bug/pyproject.toml
```

But in reality, the real error does not ever get propagated to the parent process and we don't see it. I've caught it by instrumenting the `_in_process.py` file in my virtual environment (`lib/python3.11/site-packages/pep517/in_process/_in_process.py`) with print statements.

```python
Exception: description must be specified under [project] or listed as a dynamic field
_in_process.py was called with args: ['/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpttsp4wmf']
Exception: description must be specified under [project] or listed as a dynamic field
  File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 338, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/buildapi.py", line 23, in get_requires_for_build_wheel
    info = read_flit_config(pyproj_toml)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 79, in read_flit_config
    return prep_toml_config(d, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 106, in prep_toml_config
    loaded_cfg = read_pep621_metadata(d['project'], path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
    raise ConfigError(
```

Now I can see that the underlying error actually comes from the `flit` package, complaining about lack of the `description` field in the `[project]` section of my `pyproject.toml` file.

## How to get the same output

Here is what I manually added to the `_in_process.py` file in `pep517` package in my venv

https://github.com/pypa/pyproject-hooks/blob/4c9325f4e594bfc7178452d0d01eb8da6c3dbbdb/src/pyproject_hooks/_in_process/_in_process.py#L322

```python
def main():
    # begin: added
    from pathlib import Path
    log_file = Path(__file__).parent / "log.txt"
    with log_file.open("a") as f:
        try:
    # end: added
            if len(sys.argv) < 3:
                sys.exit("Needs args: hook_name, control_dir")
            hook_name = sys.argv[1]
            control_dir = sys.argv[2]
            if hook_name not in HOOK_NAMES:
                sys.exit("Unknown hook: %s" % hook_name)
            hook = globals()[hook_name]

            hook_input = read_json(pjoin(control_dir, 'input.json'))

            json_out = {'unsupported': False, 'return_val': None}
            try:
                json_out['return_val'] = hook(**hook_input['kwargs'])
            except BackendUnavailable as e:
                json_out['no_backend'] = True
                json_out['traceback'] = e.traceback
            except BackendInvalid as e:
                json_out['backend_invalid'] = True
                json_out['backend_error'] = e.message
            except GotUnsupportedOperation as e:
                json_out['unsupported'] = True
                json_out['traceback'] = e.traceback
            except HookMissing as e:
                json_out['hook_missing'] = True
                json_out['missing_hook_name'] = e.hook_name or hook_name

            write_json(json_out, pjoin(control_dir, 'output.json'), indent=2)
        # begin: more_added
        except Exception as e:
            f.write(f"Exception: {e}\n")
            import traceback
            traceback.print_tb(e.__traceback__, file=f)
        # end: more_added
```

## Fixing the issue

I can address the complaints of `flit` by adding missing things:

* Adding `description` field
* Making sure a module `opensearch_logger` exists in the same directory where `pyproject.toml` is placed

And voila:

```yaml
#
# This file is autogenerated by pip-compile with python 3.11
# To update, run:
#
#    pip-compile pyproject.toml
#
certifi==2022.9.24
    # via
    #   opensearch-py
    #   requests
charset-normalizer==2.1.1
    # via requests
idna==3.4
    # via requests
opensearch-py==2.0.0
    # via opensearch-logger (pyproject.toml)
requests==2.28.1
    # via opensearch-py
urllib3==1.26.13
    # via
    #   opensearch-py
    #   requests
```

Alternatively, should I swap the build system to classic setuptools, it compiles successfully even without said fixes.

```toml
[build-system]
requires = [
    "setuptools",
    "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]
```

## Conclusion

I don't understand why such system was chosen by creators of build system (not pip-tools). This is the reason we don't see an underlying problem from the chosen build system. It just does not get propagated to higher level tools, such as `build` or `pip-tools`. 

P.S. was so happy to see @atugushev in the comments of this issue. Cheers!

@vduseev thanks for chiming in and for the additional context!

Digging into the issue I found that `build.utils.project_wheel_metadata` [uses](https://github.com/pypa/build/blob/4475cf1bf2362cfba8dd8b10ac97b42d066db502/src/build/util.py#L46) `pep517.quiet_subprocess_runner` which [suppresses](https://github.com/pypa/pyproject-hooks/blob/233c8e193b44d1d64bff1c2348e0d4452246c80f/src/pyproject_hooks/_impl.py#L63) output. With `pep517.default_subprocess_runner` It shows the actual error.

Patched `build`:

```diff
diff --git src/build/util.py src/build/util.py
index 9675393..a435479 100644
--- src/build/util.py
+++ src/build/util.py
@@ -43,7 +43,7 @@ def project_wheel_metadata(
     """
     builder = build.ProjectBuilder(
         os.fspath(srcdir),
-        runner=pep517.quiet_subprocess_runner,
+        runner=pep517.default_subprocess_runner,
     )

     if not isolated:
```

`pip-compile` output:

```python
root@35878ab01f42:/foo# pip-compile pyproject.toml
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 351, in <module>
    main()
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 333, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/usr/local/lib/python3.10/site-packages/flit_core/buildapi.py", line 47, in prepare_metadata_for_build_wheel
    ini_info = read_flit_config(pyproj_toml)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 79, in read_flit_config
    return prep_toml_config(d, path)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 106, in prep_toml_config
    loaded_cfg = read_pep621_metadata(d['project'], path)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
    raise ConfigError(
flit_core.config.ConfigError: description must be specified under [project] or listed as a dynamic field
Backend subprocess exited when trying to invoke prepare_metadata_for_build_wheel
Failed to parse /foo/pyproject.toml
```

---

@pradyunsg I wonder if would it be okay to introduce one of the following params in `build.utils. project_wheel_metadata()`? For example:
- `project_wheel_metadata(quiet=False)` - do not suppress the output
- `project_wheel_metadata(runner=my_runner)` - use my runner
> FYI, there is a tracking issue #1583

FTR, while it looks related to #1583 it's affected by a different part of the code. So I've reopened the issue.
FTR: tracking issue https://github.com/pypa/build/issues/553 for `project_wheel_metadata` improvement.
> @pradyunsg I wonder if would it be okay to introduce one of the following params in `build.utils. project_wheel_metadata()`?

I just saw this, and I would've suggested that you do exactly what you've done. :)

I had a bad whitespace, a '\t' (tab) in a quoted string, that caused this issue; TOML Linter found it.
I found another case today where this error can occur:.  I started with a working `pyproject.toml`, then I created two subdirectories.

The clue for me was that `pip install --editable .` gave an error "multiple top-level packages discovered in a flat-layout".

Running validate-projected returned no errors.

```
$ cat pyproject.toml 
[project]
name = 'test'
description = 'test'
version = '0.1.0'
authors = [ ]
dependencies = [ 'simplejson ' ]
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
    # via test (pyproject.toml)
$ mkdir foo
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
    # via test (pyproject.toml)
$ mkdir bar
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /home/pbrannan/tmp/test-env/pyproject.toml
$ ./env/bin/python -mvalidate_pyproject pyproject.toml
Valid file: pyproject.toml
```
Regarding that last example, it [seems to be about the way setuptools auto-discovers the project structure](https://github.com/pypa/setuptools/issues/3197#issuecomment-1078770109), and can be avoided with

```toml
[tool.setuptools]
py-modules = []
```

---

Aside from that I'll note that the alternative runner selection is now in `build`, but there has not yet been a release including it.
pip-compile omitted all error logs, I have to use `pip install -e .` to see what went wrong with the pyproject.toml file. My problem was wrong syntax for fields `license` and `requires-python`
> Regarding that last example, it [seems to be about the way setuptools auto-discovers the project structure](https://github.com/pypa/setuptools/issues/3197#issuecomment-1078770109), and can be avoided with
> 
> ```toml
> [tool.setuptools]
> py-modules = []
> ```
> 
> Aside from that I'll note that the alternative runner selection is now in `build`, but there has not yet been a release including it.

I also experienced this problem. I have a valid `pyproject.toml` with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre's solution above is what worked for me. Thanks @AndydeCleyre! 
for the record, `build` has implemented switching runner from the default `quiet` one, but it's not yet included in any releases of `build`.
So the errors come because pip-compile swallows output of `pip install -e .`
Running the command will reveal the error.

In the case of a [flatlayout](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#flat-layout) with multiple packages, mentioned by @cout 
one can explicitly specify which folders in the flat layout must be included as packages

```
[tool.setuptools]
packages = ["package1", "package2"]
```
> > Regarding that last example, it [seems to be about the way setuptools auto-discovers the project structure](https://github.com/pypa/setuptools/issues/3197#issuecomment-1078770109), and can be avoided with
> > ```toml
> > [tool.setuptools]
> > py-modules = []
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Aside from that I'll note that the alternative runner selection is now in `build`, but there has not yet been a release including it.
> 
> I also experienced this problem. I have a valid `pyproject.toml` with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre's solution above is what worked for me. Thanks @AndydeCleyre!

this also solved the problem for me.

Resolving patch:
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -15,6 +15,7 @@
 from pip._internal.req import InstallRequirement
 from pip._internal.req.constructors import install_req_from_line
 from pip._internal.utils.misc import redact_auth_from_url
+from pyproject_hooks import default_subprocess_runner, quiet_subprocess_runner
 
 from .._compat import parse_requirements
 from ..cache import DependencyCache
@@ -323,6 +324,11 @@ def cli(
                 metadata = project_wheel_metadata(
                     os.path.dirname(os.path.abspath(src_file)),
                     isolated=build_isolation,
+                    runner=(
+                        default_subprocess_runner
+                        if verbose
+                        else quiet_subprocess_runner
+                    ),
                 )
             except BuildBackendException as e:
                 log.error(str(e))


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -2582,6 +2582,40 @@ def test_input_formats(fake_dists, runner, make_module, fname, content):
     assert "extra ==" not in out.stderr
 
 
+@pytest.mark.parametrize("verbose_option", (True, False))
+def test_error_in_pyproject_toml(
+    fake_dists, runner, make_module, capfd, verbose_option
+):
+    """
+    Test that an error in pyproject.toml is reported.
+    """
+    fname = "pyproject.toml"
+    invalid_content = dedent(
+        """\
+        [project]
+        invalid = "metadata"
+        """
+    )
+    meta_path = make_module(fname=fname, content=invalid_content)
+
+    options = []
+    if verbose_option:
+        options = ["--verbose"]
+
+    options.extend(
+        ["-n", "--no-build-isolation", "--find-links", fake_dists, meta_path]
+    )
+
+    out = runner.invoke(cli, options)
+
+    assert out.exit_code == 2, out.stderr
+    captured = capfd.readouterr()
+
+    assert (
+        "`project` must contain ['name'] properties" in captured.err
+    ) is verbose_option
+
+
 @pytest.mark.network
 @pytest.mark.parametrize(("fname", "content"), METADATA_TEST_CASES)
 def test_one_extra(fake_dists, runner, make_module, fname, content):