Repository: onnx/onnx

Issue:
Build error with the latest setuptool
# Bug Report

### Is the issue related to model conversion?

No

### Describe the bug

Building from source with `pip install -e .` resulted in an error with the latest setuptool.

Running

```sh
 SETUPTOOLS_ENABLE_FEATURES="legacy-editable" pip install -e .
```

works.

### System information

- OS Platform and Distribution (*e.g. Linux Ubuntu 16.04*): Linux Ubuntu
- ONNX version (*e.g. 1.7*):  master
- Python version: 3.10.7
- GCC/Compiler version (if compiling from source): 
- CMake version:
- Protobuf version:
- Visual Studio version (if applicable):

```
Package                      Version   Editable project location
---------------------------- --------- -------------------------
absl-py                      1.2.0
astunparse                   1.6.3
cachetools                   5.2.0
certifi                      2022.9.14
charset-normalizer           2.1.1
contourpy                    1.0.5
cycler                       0.11.0
etils                        0.8.0
flatbuffers                  2.0.7
fonttools                    4.37.3
gast                         0.4.0
google-auth                  2.11.1
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
grpcio                       1.48.1
h5py                         3.7.0
idna                         3.4
importlib-resources          5.9.0
jax                          0.3.17
jaxlib                       0.3.15
joblib                       1.2.0
keras                        2.10.0
Keras-Preprocessing          1.1.2
kiwisolver                   1.4.4
libclang                     14.0.6
Markdown                     3.4.1
MarkupSafe                   2.1.1
matplotlib                   3.6.0
numpy                        1.23.3
oauthlib                     3.2.1
onnx                         1.12.0    /workspaces/onnx
opt-einsum                   3.3.0
packaging                    21.3
pandas                       1.5.0
Pillow                       9.2.0
pip                          22.2.2
plotly                       5.10.0
protobuf                     3.19.5
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pyparsing                    3.0.9
python-dateutil              2.8.2
pytz                         2022.2.1
requests                     2.28.1
requests-oauthlib            1.3.1
rsa                          4.9
scikit-learn                 1.1.2
scipy                        1.9.1
seaborn                      0.12.0
setuptools                   65.3.0
six                          1.16.0
tenacity                     8.0.1
tensorboard                  2.10.0
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.10.0
tensorflow-estimator         2.10.0
tensorflow-io-gcs-filesystem 0.27.0
termcolor                    2.0.1
threadpoolctl                3.1.0
torch                        1.12.1
typing_extensions            4.3.0
urllib3                      1.26.12
Werkzeug                     2.2.2
wheel                        0.37.1
wrapt                        1.14.1
zipp                         3.8.1
```


### Reproduction instructions

`pip install -e .`

<!--
- Describe the code to reproduce the behavior.
```
import onnx
model = onnx.load('model.onnx')
...
```
- Attach the ONNX model to the issue (where applicable)-->

### Expected behavior
<!-- A clear and concise description of what you expected to happen. -->

### Notes


```python
Building wheels for collected packages: onnx
  Building editable for onnx (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building editable for onnx (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [114 lines of output]
      running editable_wheel
      creating /tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info
      writing /tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/entry_points.txt
      writing requirements to /tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/requires.txt
      writing top-level names to /tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/top_level.txt
      writing manifest file '/tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.c' under directory 'onnx'
      adding license file 'LICENSE'
      writing manifest file '/tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx.egg-info/SOURCES.txt'
      creating '/tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx-1.12.0.dist-info'
      adding license file "LICENSE" (matched pattern "LICENSE")
      creating /tmp/pip-wheel-2mha55tp/tmpkm5xxdfk/onnx-1.12.0.dist-info/WHEEL
      /tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build_py
      running create_version
      running cmake_build
      Extra cmake args: ['-DONNX_USE_PROTOBUF_SHARED_LIBS=ON']
      Using cmake args: ['/usr/bin/cmake', '-DPYTHON_INCLUDE_DIR=/usr/local/python/3.10.7/include/python3.10', '-DPYTHON_EXECUTABLE=/usr/local/python/3.10.7/bin/python3.10', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-310-x86_64-linux-gnu.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '-DONNX_USE_PROTOBUF_SHARED_LIBS=ON', '/workspaces/onnx']
      Generated: /workspaces/onnx/.setuptools-cmake-build/onnx/onnx-ml.proto
      Generated: /workspaces/onnx/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
      Generated: /workspaces/onnx/.setuptools-cmake-build/onnx/onnx-data.proto
      -- Could NOT find pybind11 (missing: pybind11_DIR)
      -- pybind11 v2.9.1
      --
      -- ******** Summary ********
      --   CMake version             : 3.16.3
      --   CMake command             : /usr/bin/cmake
      --   System                    : Linux
      --   C++ compiler              : /usr/bin/c++
      --   C++ compiler version      : 9.4.0
      --   CXX flags                 :  -Wnon-virtual-dtor
      --   Build type                : Release
      --   Compile definitions       : __STDC_FORMAT_MACROS
      --   CMAKE_PREFIX_PATH         :
      --   CMAKE_INSTALL_PREFIX      : /usr/local
      --   CMAKE_MODULE_PATH         :
      --
      --   ONNX version              : 1.12.0
      --   ONNX NAMESPACE            : onnx
      --   ONNX_USE_LITE_PROTO       : OFF
      --   USE_PROTOBUF_SHARED_LIBS  : ON
      --   Protobuf_USE_STATIC_LIBS  : OFF
      --   ONNX_DISABLE_EXCEPTIONS   : OFF
      --   ONNX_WERROR               : OFF
      --   ONNX_BUILD_TESTS          : OFF
      --   ONNX_BUILD_BENCHMARKS     : OFF
      --
      --   Protobuf compiler         : /usr/bin/protoc
      --   Protobuf includes         : /usr/include
      --   Protobuf libraries        : /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread
      --   BUILD_ONNX_PYTHON         : ON
      --     Python version        :
      --     Python executable     : /usr/local/python/3.10.7/bin/python3.10
      --     Python includes       : /usr/local/python/3.10.7/include/python3.10
      -- Configuring done
      -- Generating done
      -- Build files have been written to: /workspaces/onnx/.setuptools-cmake-build
      [  3%] Built target gen_onnx_proto
      [  6%] Built target gen_onnx_data_proto
      [  9%] Built target gen_onnx_operators_proto
      [ 24%] Built target onnx_proto
      [ 96%] Built target onnx
      [100%] Built target onnx_cpp2py_export
      running build_ext
      copying /workspaces/onnx/.setuptools-cmake-build/onnx_cpp2py_export.cpython-310-x86_64-linux-gnu.so -> /tmp/tmpksx5ht08.build-lib/onnx
      Traceback (most recent call last):
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/_distutils/file_util.py", line 40, in _copy_file_contents
          fdst = open(dst, 'wb')
      FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpksx5ht08.build-lib/onnx/onnx_cpp2py_export.cpython-310-x86_64-linux-gnu.so'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 140, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 330, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 261, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 288, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "<string>", line 255, in run
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "<string>", line 272, in build_extensions
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 351, in copy_file
          return file_util.copy_file(
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/_distutils/file_util.py", line 163, in copy_file
          _copy_file_contents(src, dst)
        File "/tmp/pip-build-env-5rbu00q6/overlay/lib/python3.10/site-packages/setuptools/_distutils/file_util.py", line 42, in _copy_file_contents
          raise DistutilsFileError(
      distutils.errors.DistutilsFileError: could not create '/tmp/tmpksx5ht08.build-lib/onnx/onnx_cpp2py_export.cpython-310-x86_64-linux-gnu.so': No such file or directory
      error: Support for editable installs via PEP 660 was recently introduced
      in `setuptools`. If you are seeing this error, please report to:
      
      https://github.com/pypa/setuptools/issues
      
      Meanwhile you can try the legacy behavior by setting an
      environment variable and trying to install again:
      
      SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for onnx
Failed to build onnx
ERROR: Could not build wheels for onnx, which is required to install pyproject.toml-based projects
```



Discussion:
This break happened after this PR: https://github.com/onnx/onnx/pull/4361, probably related to introduced pyproject.toml file. Still investigating.
https://github.com/onnx/onnx/pull/4558 would be a quick workaround to make onnx build forcely use old legacy development mode for now, but in the long term we should modify the existing build logic to enable the latest editable_mode if it is beneficial.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -26,13 +26,11 @@
 import setuptools.command.develop
 
 TOP_DIR = os.path.realpath(os.path.dirname(__file__))
-SRC_DIR = os.path.join(TOP_DIR, "onnx")
 CMAKE_BUILD_DIR = os.path.join(TOP_DIR, ".setuptools-cmake-build")
 
 WINDOWS = os.name == "nt"
 
 CMAKE = shutil.which("cmake3") or shutil.which("cmake")
-MAKE = shutil.which("make")
 
 ################################################################################
 # Global variables for controlling the build variant
@@ -40,49 +38,48 @@
 
 # Default value is set to TRUE\1 to keep the settings same as the current ones.
 # However going forward the recommended way to is to set this to False\0
-ONNX_ML = not bool(os.getenv("ONNX_ML") == "0")
-ONNX_VERIFY_PROTO3 = bool(os.getenv("ONNX_VERIFY_PROTO3") == "1")
+ONNX_ML = os.getenv("ONNX_ML") != "0"
+ONNX_VERIFY_PROTO3 = os.getenv("ONNX_VERIFY_PROTO3") == "1"
 ONNX_NAMESPACE = os.getenv("ONNX_NAMESPACE", "onnx")
-ONNX_BUILD_TESTS = bool(os.getenv("ONNX_BUILD_TESTS") == "1")
-ONNX_DISABLE_EXCEPTIONS = bool(os.getenv("ONNX_DISABLE_EXCEPTIONS") == "1")
-ONNX_DISABLE_STATIC_REGISTRATION = bool(
-    os.getenv("ONNX_DISABLE_STATIC_REGISTRATION") == "1"
-)
+ONNX_BUILD_TESTS = os.getenv("ONNX_BUILD_TESTS") == "1"
+ONNX_DISABLE_EXCEPTIONS = os.getenv("ONNX_DISABLE_EXCEPTIONS") == "1"
+ONNX_DISABLE_STATIC_REGISTRATION = os.getenv("ONNX_DISABLE_STATIC_REGISTRATION") == "1"
+ONNX_PREVIEW_BUILD = os.getenv("ONNX_PREVIEW_BUILD") == "1"
 
-USE_MSVC_STATIC_RUNTIME = bool(os.getenv("USE_MSVC_STATIC_RUNTIME", "0") == "1")
-DEBUG = bool(os.getenv("DEBUG", "0") == "1")
-COVERAGE = bool(os.getenv("COVERAGE", "0") == "1")
+USE_MSVC_STATIC_RUNTIME = os.getenv("USE_MSVC_STATIC_RUNTIME", "0") == "1"
+DEBUG = os.getenv("DEBUG", "0") == "1"
+COVERAGE = os.getenv("COVERAGE", "0") == "1"
 
 # Customize the wheel plat-name, usually needed for MacOS builds.
 # See usage in .github/workflows/release_mac.yml
 ONNX_WHEEL_PLATFORM_NAME = os.getenv("ONNX_WHEEL_PLATFORM_NAME")
 
+################################################################################
+# Pre Check
+################################################################################
+
+assert CMAKE, "Could not find cmake in PATH"
+
 ################################################################################
 # Version
 ################################################################################
 
 try:
-    git_version = (
+    _git_version = (
         subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=TOP_DIR)
         .decode("ascii")
         .strip()
     )
 except (OSError, subprocess.CalledProcessError):
-    git_version = ""
+    _git_version = ""
 
-with open(os.path.join(TOP_DIR, "VERSION_NUMBER")) as version_file:
-    VERSION_NUMBER = version_file.read().strip()
-    if os.getenv("ONNX_PREVIEW_BUILD") is not None:
+with open(os.path.join(TOP_DIR, "VERSION_NUMBER"), encoding="utf-8") as version_file:
+    _version = version_file.read().strip()
+    if ONNX_PREVIEW_BUILD:
         # Create the dev build for weekly releases
-        today_number = datetime.date.today().strftime("%Y%m%d")
-        VERSION_NUMBER += ".dev" + today_number
-    VERSION_INFO = {"version": VERSION_NUMBER, "git_version": git_version}
-
-################################################################################
-# Pre Check
-################################################################################
-
-assert CMAKE, "Could not find cmake executable!"
+        todays_date = datetime.date.today().strftime("%Y%m%d")
+        _version += ".dev" + todays_date
+    VERSION_INFO = {"version": _version, "git_version": _git_version}
 
 ################################################################################
 # Utilities
@@ -116,30 +113,23 @@ def get_ext_suffix():
 ################################################################################
 
 
-class ONNXCommand(setuptools.Command):
-    user_options: ClassVar[list] = []
-
-    def initialize_options(self):
-        pass
-
-    def finalize_options(self):
-        pass
-
+def create_version(directory: str):
+    """Create version.py based on VERSION_INFO."""
+    version_file_path = os.path.join(directory, "onnx", "version.py")
+    os.makedirs(os.path.dirname(version_file_path), exist_ok=True)
 
-class CreateVersion(ONNXCommand):
-    def run(self):
-        with open(os.path.join(SRC_DIR, "version.py"), "w") as f:
-            f.write(
-                textwrap.dedent(
-                    f"""\
-                    # This file is generated by setup.py. DO NOT EDIT!
+    with open(version_file_path, "w", encoding="utf-8") as f:
+        f.write(
+            textwrap.dedent(
+                f"""\
+                # This file is generated by setup.py. DO NOT EDIT!
 
 
-                    version = "{VERSION_INFO['version']}"
-                    git_version = "{VERSION_INFO['git_version']}"
-                    """
-                )
+                version = "{VERSION_INFO['version']}"
+                git_version = "{VERSION_INFO['git_version']}"
+                """
             )
+        )
 
 
 class CmakeBuild(setuptools.Command):
@@ -156,8 +146,6 @@ class CmakeBuild(setuptools.Command):
         ("jobs=", "j", "Specifies the number of jobs to use with make")
     ]
 
-    built = False
-
     def initialize_options(self):
         self.jobs = None
 
@@ -168,11 +156,7 @@ def finalize_options(self):
         self.jobs = multiprocessing.cpu_count() if self.jobs is None else int(self.jobs)
 
     def run(self):
-        if CmakeBuild.built:
-            return
-        CmakeBuild.built = True
-        if not os.path.exists(CMAKE_BUILD_DIR):
-            os.makedirs(CMAKE_BUILD_DIR)
+        os.makedirs(CMAKE_BUILD_DIR, exist_ok=True)
 
         with cd(CMAKE_BUILD_DIR):
             build_type = "Release"
@@ -261,56 +245,72 @@ def run(self):
 
 class BuildPy(setuptools.command.build_py.build_py):
     def run(self):
-        self.run_command("create_version")
-        self.run_command("cmake_build")
-
-        generated_python_files = glob.glob(
-            os.path.join(CMAKE_BUILD_DIR, "onnx", "*.py")
-        ) + glob.glob(os.path.join(CMAKE_BUILD_DIR, "onnx", "*.pyi"))
-
-        for src in generated_python_files:
-            dst = os.path.join(TOP_DIR, os.path.relpath(src, CMAKE_BUILD_DIR))
-            self.copy_file(src, dst)
-        # Review the command customizations to enable editable_mode
-        self.editable_mode = False
-        return setuptools.command.build_py.build_py.run(self)
+        if self.editable_mode:
+            dst_dir = TOP_DIR
+        else:
+            dst_dir = self.build_lib
+        create_version(dst_dir)
+        return super().run()
 
 
 class Develop(setuptools.command.develop.develop):
     def run(self):
-        self.run_command("build_py")
-        setuptools.command.develop.develop.run(self)
+        create_version(TOP_DIR)
+        return super().run()
 
 
 class BuildExt(setuptools.command.build_ext.build_ext):
     def run(self):
         self.run_command("cmake_build")
-        setuptools.command.build_ext.build_ext.run(self)
+        return super().run()
 
     def build_extensions(self):
+        # We override this method entirely because the actual building is done
+        # by cmake_build. Here we just copy the built extensions to the final
+        # destination.
+        build_lib = self.build_lib
+        extension_dst_dir = os.path.join(build_lib, "onnx")
+        os.makedirs(extension_dst_dir, exist_ok=True)
+
         for ext in self.extensions:
             fullname = self.get_ext_fullname(ext.name)
             filename = os.path.basename(self.get_ext_filename(fullname))
 
-            lib_path = CMAKE_BUILD_DIR
-            if WINDOWS:
-                debug_lib_dir = os.path.join(lib_path, "Debug")
-                release_lib_dir = os.path.join(lib_path, "Release")
+            if not WINDOWS:
+                lib_dir = CMAKE_BUILD_DIR
+            else:
+                # Windows compiled extensions are stored in Release/Debug subfolders
+                debug_lib_dir = os.path.join(CMAKE_BUILD_DIR, "Debug")
+                release_lib_dir = os.path.join(CMAKE_BUILD_DIR, "Release")
                 if os.path.exists(debug_lib_dir):
-                    lib_path = debug_lib_dir
+                    lib_dir = debug_lib_dir
                 elif os.path.exists(release_lib_dir):
-                    lib_path = release_lib_dir
-            src = os.path.join(lib_path, filename)
-            dst = os.path.join(os.path.realpath(self.build_lib), "onnx", filename)
+                    lib_dir = release_lib_dir
+            src = os.path.join(lib_dir, filename)
+            dst = os.path.join(extension_dst_dir, filename)
+            self.copy_file(src, dst)
+
+        # Copy over the generated python files to build/source dir depending on editable mode
+        if self.editable_mode:
+            dst_dir = TOP_DIR
+        else:
+            dst_dir = build_lib
+        generated_python_files = (
+            *glob.glob(os.path.join(CMAKE_BUILD_DIR, "onnx", "*.py")),
+            *glob.glob(os.path.join(CMAKE_BUILD_DIR, "onnx", "*.pyi")),
+        )
+        assert generated_python_files, "Bug: No generated python files found"
+        for src in generated_python_files:
+            dst = os.path.join(dst_dir, os.path.relpath(src, CMAKE_BUILD_DIR))
+            os.makedirs(os.path.dirname(dst), exist_ok=True)
             self.copy_file(src, dst)
 
 
 CMD_CLASS = {
-    "create_version": CreateVersion,
     "cmake_build": CmakeBuild,
     "build_py": BuildPy,
-    "develop": Develop,
     "build_ext": BuildExt,
+    "develop": Develop,
 }
 
 ################################################################################


Test patch: