Package: dynamake
DynaMake 0.6.2 - Dynamic Make in PythonHOWTo install,pip install--userdynamake(orsudo pip install dynamakeif you havesudoprivileges).Then, create aDynaMake.pyfile. Here is a trivial one:fromdynamakeimport*@step(output='foo')asyncdefcopy_bar_to_foo()->None:require('bar')awaitshell('cp bar foo')Which is equivalent to theMakefile:foo:barcpbarfooThe Python version is more verbose, so if this was all there was to it,makewould have been preferable. However,
DynaMake allows one to specify scripts that are impossible inmake, justifying the additional syntax. See thetutorialfor the details.WHY“What the world needs is another build tool”—Way too many peopleSo, why yetanotherone?DynaMake’s raisons d’etre are:First class support for fully dynamic build graphs.Reproducible builds.Python implementation.DynaMake was created to address a concrete need for repeatable configurable processing in the context of scientific
computation pipelines, but should be applicable in wider problem domains.Dynamic Build GraphsThis is a fancy way of saying that the following are supported:Dynamic inputs: The full set of inputs of a build step may depend on a subset of its inputs.A classic example of dynamic inputs is compiling a C source file, which actually depends on all the included header
files. Therefore, for a safe reproducible build, the build tool first needs to ensure the C source file is up-to-date
(it might be generated by some tool such asYACC), then scan the file for
included header file names, recursively ensure each of these is up-to-date and scan them for further included header
files, and only once all this is done, we can tell the exact list of file dependencies for compiling the C source file
into an object file.Dynamic inputs are supported in various ways by most build tools - some of these ways being more convoluted than others.
DynaMake provides natural first-class support for such cases, inspired by theshakebuild
system. Thisvideois a nice short explanation of the approach.Multiple outputs: A build step may create multiple output files. A classic example of multiple outputs isYACCwhich generates both a header file and a source file from a single parser grammar file.Surprisingly, multiple outputs are not supported by the venerablemakeprogram. However many other more modern tools
do support this feature.Dynamic outputs: The set of outputs of a build step may depend on its inputs.An example of dynamic outputs is running a clustering step on some large data, which may produce some number of clusters
which is not known in advance. Each of these clusters needs to go through some further processing. In DynaMake, one can
specifyoutput='clusters/{*_id}.txt'to specify that the step will produce several output files with different
identifiers.Dynamic outputs are not supported by the vast majority of build tools. A notable exception issnakemakewhich provides support for some restricted cases.Pattern Steps: A build step may apply to a parameterized set of outputs rather than to specific outputs.Restricted forms of pattern steps have existed starting from the venerablemake.c.o:and%.o: %.cpattern
rules. However, in many tools (such asmake) these pattern steps are restricted to having a single parameter (in
this case, the base name of the output file). In the case ofmake, these rules are also very fragile (silently
ignored when make can’t create a dependency, leading to broken builds).DynaMake provides a more general form of pattern rules with an unlimited number of named parameters (e.g.output='results/{*method}/outputs/{*parameters}.data') which allows the build step to naturally use the captured
values of the parameters inside the step (e.g.awaitshell(f'{method}< {parameters} >{output()}').The downside of dynamic build graphs is that they make some build tool features really hard to implement. Therefore,
retrofitting them into an existing build tool causes some features to break. In the worst case this leads to silent
broken builds. For example:The ability to aggressively optimize the case when a build needs to do nothing at all, and in general reduce the build
system overhead.The ability to perform a dry run thataccuratelylistsallthe steps that will be needed to build an arbitrary
target.Having a purely declarative build language, which can be more easily learned than any programming language (even
Python :-) and may be processed as pure data by additional tools.Reproducible BuildsBy definition, all  build tools will correctly rebuild outputs if any of their dependencies change. However, most build
tools will not rebuild the outputs if the actions to create them were changed (e.g., adding/removing compilation flags).By default, DynaMake tracks the exact actions that were used in the past to generate every output and will rebuild the
output if this has changed in any way. This requires DynaMake to maintain state between builds inside a sub-directory
(by default,.dynamake, but you can override it using theDYNAMAKE_PERSISTENT_DIRenvironment variable).There are good reasons to avoid any such additional persistent state. DynaMake allows disabling this feature. Specifying
the--rebuild_changed_actionsFalsecommand line flag will instruct DynaMake to rely only on the modification times
of the input files. This of course results in less reliable rebuilds.PythonDynaMake is heavily inspired byshake. However,shakeis implemented in Haskell, which is unlikely to be
pre-installed on a typical machine, and installing it isn’t trivial (especially when one has nosudoprivileges).
Also,shakerules are also written in Haskell, which is very different from most popular programming languages.In contrast, Python is much more likely to already be installed on a typical machine, and installing DynaMake is trivial
usingpip install--userdynamake(orsudo pip install dynamakeif you havesudoprivileges). The build
rules are written in Python, which many more people are familiar with, and is simpler to pick up.WHY NOTDynaMake’s unique blend of features comes at some costs:It is a new, immature tool. As such, it lacks some features it could/should provide, is less efficient than it could
be, and you may encounter the occasional bug. Hopefully this will improve with time. If you want DynaMake-like
features with a proven track record, you should considershake.The provided goals, as described above, may be a poor fit for your use case.If your build graph and configuration are truly static, consider usingNinjawhich tries
to maximize the benefits of such a static build pipeline. It is almost the opposite of DynaMake in this respect.If your build graph is only “mostly static” (e.g., just needs a restricted form of dynamic inputs, such as included
header files), then you have (too) many other options to list here. Using the classicalmakeis a good default
choice.DynaMake is a low-level build tool, on par withmakeandninja.If you are looking for a tool that comes with a lot of built-in rules for dealing with specific computer languages
(say, C/C++), and will automatically deal with cross-platform issues, consider usingCMakeorXMakeinstead.WHAT NOT (YET)Since DynaMake is very new, there are many features that should be implemented, but haven’t been worked on yet:Allow forcing rebuilding (some) targets.Allow skipping generating intermediate files if otherwise no actions need to be done. This is very hard to do with a
dynamic build graph - probably impossible in the general case, but common cases might be possible(?)Generate a tree (actually a DAG) of step invocations. This can be collected from the persistent state files.Generate a visualization of the timeline of action executions showing start and end times, with resource consumption.
This would be similar to the profiling capabilities ofshake.Allow using checksums instead of timestamps to determine if actions can be skipped, either by default or on a per-file
basis.History0.6.0First release.0.6.1Patchtravis-cibuild and links inREADME.rst.0.6.2Improved project template.Improvedmypyconfiguration.