Package: runtime-builder
Runtime BuilderSometimes python developers want to include compiled output or other binary blob artifacts in their wheels. The source materials for this output is checked into git, but the blobs are not, since they're not source material and are at risk getting out of sync with the source distribution.When building the wheel, the blobs need to be generated. But it may also be nice to rebuild the blobs if the source have changed at runtime to automate a step of the edit/test cycle.This project lets you declare a single configuration describing how to build the artifacts, and build them at runtime during the development cycle or at wheel build time.ExampleThere are example projects intests/project_template_ensconsandtests/project_template_setuptools.tests/project_template_setuptools
├── README.md
├── proj
│   ├── bar.source
│   ├── foo.source
│   ├── run_test.py
│   └── runtime_build
├── pyproject.toml.template
├── setup.py
├── test_pip_install.sh
└── test_pip_install_editable.shThis examples usessetuptoolsandsetup.pyto build. Thesetup.pyandruntime_buildfiles work together to make available artifacts built from the.sourcefiles.project_template_enscons
├── README.md
├── SConstruct
├── proj
│   ├── bar.source
│   ├── foo.source
│   ├── run_test.py
│   └── runtime_build
├── pyproject.toml.template
├── test_pip_install.sh
└── test_pip_install_editable.shNote that this example usesensconsto build, assetuptoolsdoes not easily allow fine-grained control of the contents of thesdistandwheelfiles. In particular, this example takes pains to include the source filesruntime_build,foo.sourceandbar.sourcein the sdist but not the wheel.ConfigurationArtifacts to be build must be declared in aruntime_buildfile located in the destination module of the artifacts. Each submodule can have zero or one of these build files. In theproject_templateexample above,bar.sourceandfoo.sourceeach produce an output in theprojsubmodule, andruntime_builddefines what the artifacts are.This configuration file can be loaded at build time or at run time, so itcannotimport anything that isn't made available in thebuild-system.requiressection ofpyproject.toml. In particular, it cannot import anything from the project being built since it has to be built before it can be installed in the build package, leading to a bootstrap problem.Here is the example from above:frompathlibimportPathfromtypingimportCallableclassMultiplyBuild:def__init__(self,factor:int)->None:self.factor=factordef__call__(self,target_path:Path):source_path=target_path.with_suffix(".source")t=int(open(source_path).read())withopen(target_path,"w")asf:f.write(f"{t*self.factor}")defcount_build(target_path:Path)->None:source_path=target_path.with_suffix(".source")t=open(source_path).read()withopen(target_path,"w")asf:f.write(f"{len(t)}\n")BUILD_ARGUMENTS:dict[str,Callable[[Path],None]]={"foo.val":MultiplyBuild(50),"bar.count":count_build,}The key is to return adictobject calledBUILD_ARGUMENTSthat hasstrkeys corresponding to artifacts, andCallablevalues. TheCallableshould generate the given artifact.These examples could potentially build the artifacts every time they are referenced. If a build takes long enough, you may want to add code to only build when necessary to speed the edit/test cycle.Build-timeTheruntime_builderwheel is primarily a build-time dependency. If you want to take advantage of the run-time building, you need to install it in your runtime virtual environment. However, it generally should not be installed in the release version of your package.Add it to yourpyproject.tomlfile:[build-system]requires=["runtime_builder",...]In theSConstructfile is the following line:built_items = build_all_items_for_package("proj")This returns the list of artifacts as a list ofPathobjects. These files must be included in the final wheel.RuntimeTo use the dynamic build capabilities at runtime, use something like this boilerplate function:try:fromruntime_builderimportbuild_on_demandexceptImportError:# `runtime_builder` is an optional dependency for development onlydefbuild_on_demand(*args):passdefload_resource(resource_path:str,package:Package,)->bytes:build_on_demand(package,resource_path)withas_file(files(package).joinpath(resource_path))astarget_path:returntarget_path.read_bytes()This checks for presence ofruntime_builder, and invokes it if available before accessing the resource.