Repository: meltano/meltano

Issue:
Run custom build code when creating wheel
Context: We currently have a `build.py` file which is not used. If we instruct Poetry to use it, it will be included in the sdist, and build the webapp (among other things) into the wheel when the wheel is either built from source directly or from the sdist.

The problem with `build.py` is that when Poetry uses `build.py` it builds a platform specific wheel, rather than a pure wheel. We currently build a pure wheel, and want to continue to do so.

The crux of this issue is to find some way to run custom code when a wheel is being constructed (either directly from source or from an sdist). This code should at least build the UI webapp.

Resolving this issue will unblock:
- #6876 
- #6566
- #6488

---

~~Created this draft issue to track work with `poetry` team on how to get the build.py process in a good place. Today this is not compatible with a generic 'pure' wheel that is not platform-specific.~~

Converted to a normal issue in light of this becoming a likely blocker for:
- #6876 

We need to be able to go beyond what Poetry's build-system allows us to do. We currently do so with a mix of a mostly-dead-code `Makefile`, and a build workflow in `.github/`.

After spending a few hours looking into this again today, I've reaffirmed my belief that this won't be possible using `build.py`. We don't have any practical way to hook into Poetry's decision to build a plat-wheel, since it runs `build.py` (and Setuptools) in a subprocess. We could (and in the past I have) use the extreme flexibility of Setuptools to build the wheel, and then kill the parent process (Poetry), but this is a terribly hacky approach that I didn't open a PR for since I was not confident it in.

That said, I do believe we can achieve the desired effect (running arbitrary code when the wheel is built) by using a custom build backend that is a wrapper around `poetry-core`. Poetry won't use the custom build backend when using `poetry build`, but `pip wheel . --no-deps` will.


Discussion:

Resolving patch:
diff --git a/build.py b/build.py
deleted file mode 100644
--- a/build.py
+++ /dev/null
@@ -1,33 +0,0 @@
-from __future__ import annotations
-
-import os
-import shutil
-import subprocess
-
-
-def build():
-    # Build static web app
-    subprocess.run(["yarn", "install", "--immutable"], cwd="src/webapp")
-    subprocess.run(["yarn", "build"], cwd="src/webapp")
-
-    # Copy static files
-    os.makedirs("src/meltano/api/templates", exist_ok=True)
-    shutil.copy(
-        "src/webapp/dist/index.html",
-        "src/meltano/api/templates/webapp.html",
-    )
-    shutil.copy(
-        "src/webapp/dist/index-embed.html",
-        "src/meltano/api/templates/embed.html",
-    )
-
-    for dst in {"css", "js"}:
-        shutil.rmtree(f"src/meltano/api/static/{dst}", ignore_errors=True)
-        shutil.copytree(
-            f"src/webapp/dist/static/{dst}",
-            f"src/meltano/api/static/{dst}",
-        )
-
-
-if __name__ == "__main__":
-    build()
diff --git a/scripts/build.py b/scripts/build.py
new file mode 100644
--- /dev/null
+++ b/scripts/build.py
@@ -0,0 +1,101 @@
+"""A PEP 517 build backend for Meltano.
+
+This uses the PEP 517 build backend provided by `poetry-core`, with a slight
+modification. The `WheelBuilder.build` method has been patched to allow us to
+make arbitrary changes to the source/sdist immediately prior to the
+construction of the wheel.
+
+This is used instead of Poetry's unofficial `build.py` approach because Poetry
+the assumes that if `build.py` is used, it should build a platform-specific
+wheel, rather than a platform-agnostic pure wheel.
+"""
+
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+from warnings import warn
+
+
+def build_webapp() -> None:
+    """Build the Meltano UI webapp."""
+    warning_prefix = "Failed to build Meltano UI webapp:"
+
+    try:
+        node_version_check = subprocess.run(
+            ("node", "--version"), text=True, capture_output=True
+        )
+        node_version_check.check_returncode()
+    except Exception:
+        warn(f"{warning_prefix} could not execute 'node --version'", Warning)
+        return
+
+    try:
+        subprocess.run(("yarn", "--version")).check_returncode()
+    except Exception:
+        warn(f"{warning_prefix} could not execute 'yarn --version'", Warning)
+        return
+
+    if not node_version_check.stdout.startswith("v16"):
+        warn(f"{warning_prefix} NodeJS v16 is required", Warning)
+        return
+
+    # Build static web app
+    subprocess.run(("yarn", "install", "--immutable"), cwd="src/webapp")
+    subprocess.run(("yarn", "build"), cwd="src/webapp")
+
+
+def include_webapp() -> None:
+    """Copy the built webapp into the source tree."""
+    os.makedirs("src/meltano/api/templates", exist_ok=True)
+    shutil.copy(
+        "src/webapp/dist/index.html",
+        "src/meltano/api/templates/webapp.html",
+    )
+    shutil.copy(
+        "src/webapp/dist/index-embed.html",
+        "src/meltano/api/templates/embed.html",
+    )
+    for dst in ("css", "js"):
+        shutil.rmtree(f"src/meltano/api/static/{dst}", ignore_errors=True)
+        shutil.copytree(
+            f"src/webapp/dist/static/{dst}",
+            f"src/meltano/api/static/{dst}",
+        )
+
+
+def include_schemas() -> None:
+    """Copy Meltano schemas into the source tree."""
+    shutil.rmtree("src/meltano/schema/", ignore_errors=True)
+    shutil.copytree("schema/", "src/meltano/schema/")
+    # TODO: Once Python 3.7 support is dropped, we should use
+    #       `shutil.copytree(..., dirs_exist_ok=True)` instead of deleting the
+    #       directory first.
+
+
+from poetry.core.masonry.builders.wheel import WheelBuilder  # noqa: E402
+
+original_wheel_build = WheelBuilder.build
+
+
+def custom_build_wheel(*args, **kwargs) -> str:
+    """Wheel building via poetry-core with customizations for Meltano.
+
+    Args:
+        args: Positional arguments to pass through to `WheelBuilder.build`.
+        kwargs: Keyword arguments to pass through to `WheelBuilder.build`.
+
+    Returns:
+        The value returned by `WheelBuilder.build(*args, **kwargs)`.
+    """
+    build_webapp()
+    include_webapp()
+    include_schemas()
+    return original_wheel_build(*args, **kwargs)
+
+
+WheelBuilder.build = custom_build_wheel
+
+# Provide the Poetry core PEP 517 API, which will now use the patched `WheelBuilder`.
+from poetry.core.masonry import api  # noqa: F401, E402


Test patch: