Repository: pypa/setuptools

Issue:
[BUG] sys.exit(0) in setup.py breaks PEP 517 hooks API
### setuptools version

68.0.0

### Python version

3.11

### OS

Linux

### Additional environment information

I believe this will affect any OS and any Python version.

### Description

If a setup.py file calls `sys.exit(0)`, building it through the PEP 517 hooks interface will fail. `sys.exit()` raises a `SystemExit` exception, which propagates up out of the hook function. Normally, unhandled exceptions indicate that something went wrong, and that's OK - the frontend sees that the backend exited with an error, and displays its stdout/stderr for debugging. But in this case the exception is used even though nothing is wrong.

[PEP 517](https://peps.python.org/pep-0517/) says:

> If a hook raises an exception, or causes the process to terminate, then this indicates an error.

We could interpret that 100% literally: an exception was raised, therefore the backend has failed, but that's clearly not the intention - `sys.exit(0)` indicates that execution of `setup.py` completed successfully. **Edit:** it's like `return` for scripts. :slightly_smiling_face: 

This issue has been bounced around several packaging related projects already, most recently https://github.com/pypa/pyproject-hooks/issues/173 .

### Expected behavior

I think setuptools hooks implementation should catch and ignore `SystemExit` with an `err.code` attribute of 0 or None (the non-error cases), on the assumption that this means running `setup.py` finished successfully.

I had a go at doing this in `pyproject-hooks`, the machinery that calls the hooks, but the hooks have to return something, so catching the error outside setuptools isn't enough - we don't know what it would have returned.

### How to Reproduce

Save the following code as `setup.py` in a folder by itself and then run `python -m build path/to/folder/`:

```python
from setuptools import setup
import sys

setup(
    name="pep517-test-setup-py-support",
    version="1.0",
)
sys.exit(0)
```

### Output

```console
$ python -m build tests/samples/setup-py/
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools, wheel)
* Getting build dependencies for sdist...
running egg_info
writing pep517_test_setup_py_support.egg-info/PKG-INFO
writing dependency_links to pep517_test_setup_py_support.egg-info/dependency_links.txt
writing top-level names to pep517_test_setup_py_support.egg-info/top_level.txt
reading manifest file 'pep517_test_setup_py_support.egg-info/SOURCES.txt'
writing manifest file 'pep517_test_setup_py_support.egg-info/SOURCES.txt'

Traceback (most recent call last):
  File "/home/takluyver/.local/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line 19, in read_json
    with open(path, encoding='utf-8') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpe0ghw0d1/output.json'

ERROR Backend operation failed: FileNotFoundError(2, 'No such file or directory')
```



Discussion:
Hi @takluyver, thank you for filing this issue.

Due to my personal bandwidth, I will treat this with low priority, but if anyone would like to give it a go and propose a PR I can review it.

The rationale for the low priority is that running `python setup.py` directly has been deprecated for a while, therefore adding `sys.exit` to the `setup.py` no longer make that much sense...

I also think that a fix, if implemented, should target the legacy backend, but not in the regular one. Do you agree?
> The rationale for the low priority is that running python setup.py directly has been deprecated for a while, therefore adding sys.exit to the setup.py no longer make that much sense...

I'm not entirely sure about that. It may still make sense as a way to exit the script early for some reason - `sys.exit(0)` is to a script as `return` is to a function. If all your `setup.py` script does is call `setuptools.setup()`, like my minimal reproducer, then it's certainly unnecessary, but you can put arbitrary code in `setup.py`, and projects have done all sorts of things with that.

Edit: it's also a pattern in some places to have the exit code in a variable or a return value, and unconditionally call `sys.exit(exit_code)` at the end of a script.

> I also think that a fix, if implemented, should target the legacy backend, but not in the regular one. Do you agree?

I disagree: I would fix it in both backends. For one thing, that's simply easier: the relevant code is shared between them, so it's a smaller change to fix both than to introduce a distinction. It could also cause confusion when something that worked with the legacy backend fails when someone selects the regular backend.

The alternative would be to declare that `setup.py` should never call `sys.exit()` for non-error cases, and catch this happening and turn it into an explicit error so that people have to go and 'fix' it. But I don't really see what the point is. Making it an error and pushing people to change seems like more work for setuptools than just fixing it. :wink: 
The fix for the legacy may consist of simply wrapping https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py#L340 to catch SystemExit and then disregarding it in case of success (0) and throwing some failure exception in case of non-zero exit status (potentially even `OSError`).

I'll be happy to provide a PR.
> and throwing some failure exception in case of non-zero exit status (potentially even OSError).

I'd probably just re-raise the `SystemExit` error unchanged for the non-zero cases, on the assumption that the script has already provided some messages indicating what went wrong.
That's the thing rethrowing SystemExit may break pyproject-hooks handling of the sub-subprocess death, which results in a very obscure pip error saying "FileNotFound 'output.json' ". It'll also technically break PEP-517.

Per comment in `build_meta.py` ideally we should be using an isolated interpreter as well :smile::

https://github.com/pypa/setuptools/blob/cd9c6a4150d87dc6a451379aea99ca65e938e451/setuptools/build_meta.py#L11-L14
> I'm not entirely sure about that. It may still make sense as a way to exit the script early for some reason - `sys.exit(0)` is to a script as `return` is to a function.

Conceptually we are moving away from `setup.py` being seen as an executable script. That is deprecated, and that is why I think it does not make much sense to have `sys.exit(0)` in this new mental model. Going forward `setup.py` should only be a configuration file that culminates with a call to `setup(...)` as a form of passing build parameters...


> I'll be happy to provide a PR.

Thank you very much @arcivanov.
> rethrowing SystemExit may break pyproject-hooks handling of the sub-subprocess death, which results in a very obscure pip error saying "FileNotFound 'output.json' "

I think it's OK to rethrow it when there's a non-zero exit code - the issue with the FileNotFoundError is only with code 0 (or None) if I've followed it correctly.
I so far settled on `CalledProcessError`, which is plausible for any backend to throw. I'll submit a PR and if it's not desirable in practice we'll rewrite it. 
Thank you very much @arcivanov, I think the PR is very welcome.

I was just thinking about other things that use `.py` as configuration files.
For example, what does `sys.exit(0)` even mean in the context of pytest's `conftest.py` and tox's `toxfile.py`? I think there is a closer parallel between those and `setup.py` instead of a regular executable script...
But `setup.py` was never a configuration file per se - you ran it as `python setup.py` and it was exit with certain exit code.
In my specific case I was calling another `setup.py` from mine and passing exit code from the other process.
It worked until the version of pip 23.1 when the mechanism changed and pip stopped calling `setup.py` as a process landing us on this issue.

Resolving patch:
diff --git a/setuptools/build_meta.py b/setuptools/build_meta.py
--- a/setuptools/build_meta.py
+++ b/setuptools/build_meta.py
@@ -337,7 +337,19 @@ def run_setup(self, setup_script='setup.py'):
         with _open_setup_script(__file__) as f:
             code = f.read().replace(r'\r\n', r'\n')
 
-        exec(code, locals())
+        try:
+            exec(code, locals())
+        except SystemExit as e:
+            if e.code:
+                raise
+            # We ignore exit code indicating success
+            SetuptoolsDeprecationWarning.emit(
+                "Running `setup.py` directly as CLI tool is deprecated.",
+                "Please avoid using `sys.exit(0)` or similar statements "
+                "that don't fit in the paradigm of a configuration file.",
+                see_url="https://blog.ganssle.io/articles/2021/10/"
+                "setup-py-deprecated.html",
+            )
 
     def get_requires_for_build_wheel(self, config_settings=None):
         return self._get_build_requires(config_settings, requirements=['wheel'])


Test patch:
diff --git a/setuptools/tests/test_build_meta.py b/setuptools/tests/test_build_meta.py
--- a/setuptools/tests/test_build_meta.py
+++ b/setuptools/tests/test_build_meta.py
@@ -958,3 +958,26 @@ def test_legacy_editable_install(venv, tmpdir, tmpdir_cwd):
     cmd = ["pip", "install", "--no-build-isolation", "-e", "."]
     output = str(venv.run(cmd, cwd=tmpdir, env=env), "utf-8").lower()
     assert "running setup.py develop for myproj" in output
+
+
+@pytest.mark.filterwarnings("ignore::setuptools.SetuptoolsDeprecationWarning")
+def test_sys_exit_0_in_setuppy(monkeypatch, tmp_path):
+    """Setuptools should be resilent to setup.py with ``sys.exit(0)`` (#3973)."""
+    monkeypatch.chdir(tmp_path)
+    setuppy = """
+        import sys, setuptools
+        setuptools.setup(name='foo', version='0.0.0')
+        sys.exit(0)
+        """
+    (tmp_path / "setup.py").write_text(DALS(setuppy), encoding="utf-8")
+    backend = BuildBackend(backend_name="setuptools.build_meta")
+    assert backend.get_requires_for_build_wheel() == ["wheel"]
+
+
+def test_system_exit_in_setuppy(monkeypatch, tmp_path):
+    monkeypatch.chdir(tmp_path)
+    setuppy = "import sys; sys.exit('some error')"
+    (tmp_path / "setup.py").write_text(setuppy, encoding="utf-8")
+    with pytest.raises(SystemExit, match="some error"):
+        backend = BuildBackend(backend_name="setuptools.build_meta")
+        backend.get_requires_for_build_wheel()