Repository: pypa/setuptools

Issue:
[BUG] Setuptools 65 does not copy binary files (.so) in editable install
### setuptools version

65.0.0

### Python version

3.9

### OS

Ubunty 20.04

### Additional environment information

Standard setup using Cythonized sources.  Cython to C then compiled to .so/.pyd files.

### Description

As of 65 the binary extensions are not being copied back into the tree when using an editable install.  This means that they are not available when importing the library.  The install command is very vanilla

```
python -m pip install -e .
```


### Expected behavior

The .so/.pyd files would be in the same directory as their target (which is the same as the .pyx files and their compiled c version).

### How to Reproduce

1. `pip install -e ,` for a project containing Cython
2. Check tree for missing extension modules

On linux,

```
git clone https://github.com/bashtage/arch
cd arch
python -m pip install -e .
ls arch/univariate/recursions*
````

Note that the .so is missing when using 65.  If using an older setuptools, e.g., 63.
### Output

```console
The error depends on whether the binary modules are required.
```



Discussion:
I have a similar issue since `setuptools>=64`, and I may have found the root cause.

Looks like you're doing something similar to me here:
https://github.com/bashtage/arch/blob/1b141cb7710b9c8ba2192038d9ea6158e75c201d/setup.py#L20

Basically, since this commit: https://github.com/pypa/setuptools/commit/075a3c801910a6b65dccf8b9eb26a0637315627a, `setuptools` added a way to export the output mapping, which is later used here: https://github.com/pypa/setuptools/blob/7021149ad29c30e4cccfa076dba30ede6ae205ee/setuptools/command/editable_wheel.py#L250 during editable installs.

**Why is failing for our similar use-cases:**

We both need to avoid using Cython when not available or could not be used, even if available.

I can see 3 different use-cases:

1. Do not use Cython `build_ext` as is not available
2. Do not use Cython even if available, as I can't use it. (`c` files are provided instead)
3. Feel free to use Cython, as is available and can be used.

On `setuptools<64.0.0`, a custom `build_ext` `cmdclass` based on top of `distutils.command.build_ext` ( case 1 and 2) or on top of `Cython.Distutils.build_ext` (case 3) worked great.

That's hacky, worked in past, but may not be the best way to do that.

**Additional:**
Looks like `setuptools` added a check on their side, here:
https://github.com/pypa/setuptools/blob/e6b911708e900f57c98e3b5b03af6de2953dbc76/setuptools/command/build_ext.py#L16-L24

That may be enough for case 1, but not for case 2.

(I'm currently investigating a way that avoids me to fall into case 2)

PS: Adding a switch that avoids the usage of `Cython.Distutils.build_ext` may be great BTW.
Thanks.  Please let me know if you find a good solution - I have just pinned <64 for now.
Thank you very much @misl6 for the comprehensive summary.

Could you please expand a little bit more about the motivation of the use case 2 (which I believe is related to the suggestion of the switch you made...)?

In the case of a project with `pyproject.toml` the use case 3 should aways be the case, right? And if someone decides to call `python setup.py bdist_wheel` directly I suppose setuptools will automatically choose the most appropriate method depending on the availability of `Cython`, right?

---

Hi @bashtage , I believe a good solution here is to just use `setuptools.command.build_ext` directly, as it will automatically use Cython if it is installed, or fallback to pre-generated `.c` files otherwise.

Resolving patch:
diff --git a/setuptools/command/editable_wheel.py b/setuptools/command/editable_wheel.py
--- a/setuptools/command/editable_wheel.py
+++ b/setuptools/command/editable_wheel.py
@@ -237,6 +237,8 @@ def _set_editable_mode(self):
             cmd = dist.get_command_obj(cmd_name)
             if hasattr(cmd, "editable_mode"):
                 cmd.editable_mode = True
+            elif hasattr(cmd, "inplace"):
+                cmd.inplace = True  # backward compatibility with distutils
 
     def _collect_build_outputs(self) -> Tuple[List[str], Dict[str, str]]:
         files: List[str] = []


Test patch:
diff --git a/setuptools/tests/test_editable_install.py b/setuptools/tests/test_editable_install.py
--- a/setuptools/tests/test_editable_install.py
+++ b/setuptools/tests/test_editable_install.py
@@ -5,6 +5,7 @@
 import platform
 from copy import deepcopy
 from importlib import import_module
+from importlib.machinery import EXTENSION_SUFFIXES
 from pathlib import Path
 from textwrap import dedent
 from unittest.mock import Mock
@@ -28,6 +29,7 @@
     editable_wheel,
 )
 from setuptools.dist import Distribution
+from setuptools.extension import Extension
 
 
 @pytest.fixture(params=["strict", "lenient"])
@@ -872,9 +874,40 @@ def test_access_plat_name(self, tmpdir_cwd):
         cmd = editable_wheel(dist)
         cmd.ensure_finalized()
         cmd.run()
-        wheel_file = str(next(Path().glob('dist/*')))
+        wheel_file = str(next(Path().glob('dist/*.whl')))
         assert "editable" in wheel_file
-        assert wheel_file.endswith(".whl")
+
+
+class TestCustomBuildExt:
+    def install_custom_build_ext_distutils(self, dist):
+        from distutils.command.build_ext import build_ext as build_ext_cls
+
+        class MyBuildExt(build_ext_cls):
+            pass
+
+        dist.cmdclass["build_ext"] = MyBuildExt
+
+    @pytest.mark.skipif(
+        sys.platform != "linux", reason="compilers may fail without correct setup",
+    )
+    def test_distutils_leave_inplace_files(self, tmpdir_cwd):
+        jaraco.path.build({"module.c": ""})
+        attrs = {
+            "ext_modules": [Extension("module", ["module.c"])],
+        }
+        dist = Distribution(attrs)
+        dist.script_name = "setup.py"
+        dist.set_defaults()
+        self.install_custom_build_ext_distutils(dist)
+        cmd = editable_wheel(dist)
+        cmd.ensure_finalized()
+        cmd.run()
+        wheel_file = str(next(Path().glob('dist/*.whl')))
+        assert "editable" in wheel_file
+        files = [p for p in Path().glob("module.*") if p.suffix != ".c"]
+        assert len(files) == 1
+        name = files[0].name
+        assert any(name.endswith(ext) for ext in EXTENSION_SUFFIXES)
 
 
 def install_project(name, venv, tmp_path, files, *opts):