Package: alpha-build-core
AlphaBuildAlphaBuild is a simple monorepo build tool based on Make with out-of-the-box support for numerous tools in the
Python, Bash, Jupyter Notebooks, Markdown, YAML ecosystems and with a strong focus on extensibility.The way AlphaBuild works draws inspiration heavily from monorepo build tools such as Pants, Bazel, Buck.
It can run at once multiple linters, formatters, type checkers, hermetic packers, testing frameworks, virtual
environment managers etc.PlatformsUsageGoals - what tools we runTargets - what files we run the tools onAlphaBuild structureIDE IntegrationExample monorepo running AlphaBuildCommon admin actionsInstallationCI/CD SetupUpgradeChange goal definitionsAdd goalsUpdate PYTHONPATHSee/Change tools configThird party environmentsNested MakefilesGenerate requirements.txt for each sub-projectGenerate setup.py for each sub-projectMarkdown badgeOut-of-the-box tools by languageHigh-level comparison with Pants, Bazel, Pre-commit,
MakefilesLimitationsDetailed comparison: Make vs Pre-commit vs Tox/Nox vs Bazel vs PantsPlatformsAlphaBuild works on Linux distributions, MacOS, WSL and Windows with Git Bash.MakeandBashare AlphaBuild's only pre-requisites (Note: Bash with GNU utilities, not BSD utilities).Linux, WSLSince Linux is awesome and WSL follows its footsteps, AlphaBuild should just work there.MacOSAlphaBuild relies heavily on the GNU version offindandegrep, so, if you are running an OS which,
by default, uses BSD rather than GNU (pointing fingers to MacOS here, you may need tobrew installfindutilsandgrep, see:https://xenodium.com/gnu-find-on-macos/)Also, AlphaBuild may not work if you have ancient versions ofMake/Bash, so try upgrading them if some commands
don't seem to work. Macs typically come equipped with ancient versions ofBash.WindowsOn Windows,MakeandBashare not supported out-of-the-box, so it is recommended to use AlphaBuild on Windows within
Git Bash. Note that Git Bash does not come with Make pre-installed. Once you downloaded and installed Git Bash
runbuild-support/git-bash-integration/install_make.shrunning Git Bash as administrator. Alternatively, you may getMakefromconda.UsageUsually to format, lint, type-check, test, package, ... the code, one needs to run a bunch of commands in the terminal,
setting the right flags and the right parameters. This Make-based build system helps with running these commands with
a very simple interface:make <goal> <optional-targets>wheregoal = what tools we runtargets = over which files we run these tools.Goals - what tools we runGoals mean what command line tools to run. This build system can run one more tools at once as follows:Singleindividualtoole.g.make mypy,make flake8,make isortMultiple tools for aspecific languagee.g.make fmt-pyruns all Python formatters e.g.isort,black,docformatter,flynt,autoflakee.g.make fmt-shrunsshfmte.g.make lint-pyruns Python all linters and all formatters in "verification" mode, that isflake8+pylint+
check whether the code is already formatted withisort,black,docformatter,flynt,autoflakemake fmt-md,make lint-yml,test-sh,test-py... work similarlyMultiple tools formultiple languagese.g.make fmtruns all formatters for all supported languages (Python, Bash, Markdown, YAML, ...)e.g.make lintruns all linters for all supported languagese.g.make testruns all test suites for all supported languagesIt is possible to run multiple goals at once likemake lint test. In addition, it is very easy to change the meaning
of goals that run more than one command since they are very simply defined in Make based on other goals. For example,
one can remove theshfmtfrom bash linting simply by doing the below:# Beforelint-sh:shellcheckshfmt-check# where shellcheck and shfmt-check run the respective commands# Afterlint-sh:shellcheckPer-tool config files (e.g.mypy.ini,pyproject.toml) are typically places inbuild-support/<language>/tools-config/.Targets - what files we run the tools onWe have seen that Make gives us the power to run multiple terminal commands effortlessly. Using a Makefile like
described above is standard practice in many projects, typically running the different tools over all their files.
However, as projects grow, the need to run these tools at different granularities (e.g. in a specific directory,
over a given file, on the diff between two branches, since we last committed etc). This is where targets come into play.With default targetsThe default targets per-language are defined at the top of theMakefilein language specific variables
e.g.ONPY=py-project1/ py-project2/ script.pyandONSH=scripts/.make lintruns:all Python linters on all directories (in the$ONPY) that contain Python/stub files.all notebook linters on all directories (in$ONNB) that contain.ipynbfiles.all Bash linters (shellcheck) on all directories (in$ONSH) that contain Bash files.a Haskell linter (hlint) on all directories (in$ONHS) that contain Haskell files.a YAML linter (yamllint) on all directories (in$ONYML) that contain YAML files.make lint,make fmt -j1,make type-checkwork similarlyremember that the$(ONPY),$(ONSH), ... variables are defined at the top of the Makefile and represent the
default locations where AlphaBuild searches for files certain languages.With specific targetsTo specify manually the files/directories you want to run your tools on, AlphaBuild leverages the "on" syntax:file:make lint on=app_iqor/server.pyruns all Python linters on the file, same
asmake lint-py on=app_iqor/server.pydirectory:make lint on=lib_py_utilsruns a bunch of linters on the directory, in this case, same
asmake lint-py on=lib_py_utils/.files/directories:make lint on="lib_py_utils app_iqor/server.py"runs a bunch of linters on both targets.globs:make lint on=lib_*aliases:make fmt on=iqorwhere at the top of the Makefileiqor=app_iqor/iqor app_iqor/test_iqor, this is the
same asmake fmt on=app_iqor/iqor app_iqor/test_iqorbecauseiqoris an alias forapp_iqor/iqor app_iqor/test_iqor. Even though this example is simplistic, it is useful to alias combinations of
multiple files/directories. It is recommended to set aliases as constants in the Makefile even though environment
variables would also work.same formake fmt,make test,make type-check.With git revision targetsmake fmt -j1 since=masterruns all formatters on the diff between the current branch and master.make fmt -j1 since=HEAD~1runs all formatters on all files that changed since "2 commits ago".make lint since=--cachedruns all linters on all files that are "git added".all goals that support the "on" syntax also support the "since" syntaxMixed "on" and "since"One can use the "on" and "since" syntaxes at the same time. For example:make lint on=my_dir/ since=HEAD~2will run all linters on all files inmy_dir/that changed since "3 commits ago".ConstraintsDifferent languages may have different goals, for example Python can be packaged hermetically with Shiv, while Bash
obviously can't.The following goals must support the "on" and "since" syntax and ensure that they are only run if there are any targets
for the language they target:formatlinttype-checktestIf you want to learn more about the API of a specific goal, check the source code.AlphaBuild structureMakefile:AlphaBuild's entry point, this is where all components come together.3rdparty/:Files required to build environments of 3rd party dependencies (e.g. requirements.txt files,
package.json or lock files)build-support/:Makefile library inspired by Pants/Bazel to run linters, formatters, test frameworks, type
checkers, packers etc. on a variety of languages (Python, Jupyter Notebooks, Bash, Haskell, YAML, Markdown)The flags used per-tool (e.g. setting the paths to config files) can be found inbuild-support/alpha-build/config/<lang>.mkThe core part of AlphaBuild lives inbuild-support/alpha-build/core/, this comprises the build-system backboneresolver.mkand recipes to run lots of readily-available tools. This should be the same for all monorepos that
use AlphaBuild.By convention, repo-specific custom goals go inbuild-support/alpha-build/extensions/following the examples incore.build-support/<other-programming-lang-than-make>/contain things like config files for each tool and other files
required for your custom AlphaBuild goals.IDE IntegrationPyCharm / IntelliJWindows: On Windows, it is advised to set Git Bash as the default "Terminal". In "settings" search for "terminal",
go to "Tools â†’ Terminal" and set "Shell Path" to something likeC:\Program Files\Git\bin\bash.exe.PYTHONPATH: If you are writing Python, please mark the directories for each project as "Sources Roots", such that
PyCharm discovers your imports.AlphaBuild hotkeys: It is easy to use AlphaBuild with hotkeys. For example, mappingAlt+Ftomake fmt -j1 on=<current-file>andAlt+Ltomake lint on=<current-file>. To set this up, follow the example inbuild-support/ide-integration/windows/to set up an external tool. Unix systems would be similar. Next, just map the
new external tools to your hotkeys.Common admin actionsInstallationUse thiscookiecuttertemplate:https://github.com/cristianmatache/cookiecutter-alpha-build-polyrepo-py.
Feel free to extend the project such that it becomes a monorepo.UpgradeTo upgrade an existing installation if new tools are added or changes are made to the target resolution infrastructure,
one would simply need to replace thebuild-support/alpha-build/coredirectory. To do that please run:pipinstallalpha-build-core--targettmp/
tar-xvftmp/alpha_build_core.tar.gz
rm-rftmp/CI/CD setupSince all CI/CD pipelines essentially rely on running some scripts in a certain order, AlphaBuild can be called
directly from any CI/CD pipeline regardless of CI/CD technology provider. AlphaBuild helps with ensuring that both the
CI pipelines and developers run the exact same commands. Since, one can easily select targets within the repo, setting
pipelines on a per sub-project basis, is effortless with AlphaBuild.Change goal definitionsLet's say, for example, you don't want to runpylintas part of your python linting. You would simply go to theMakefileand change the definition of thelint-pygoal to not includepylint.Add goalsThe goals that are available out of the box are found inbuild-support/alpha-build/core/<language>/.
You can extend/replace the core goals for new languages and/or tools by writing.mkcode inbuild-support/alpha-build/extensions/<language>/following the examples inbuild-support/alpha-build/core/.
For example,https://github.com/cristianmatache/workspaceextends AlphaBuild with Prometheus and Alertmanager goals.Update PYTHONPATHThe PYTHONPATH is set at the top of theMakefile. For example, to add new directories to
the PYTHONPATH (i.e. to mark them as sources roots) setPY_SOURCES_ROOTSat the top of the Makefile.See/Change tools configLet's say you want to change the waymypyis configured to exclude some directory from checking. Then head tobuild-support/alpha-build/config/python.mkcheck what is the path to themypyconfig file, go there and update it.
All other tools work similarly.Third party environmentsExact reproduction of the default environment:The recipes to fully replicate the default environment
(mostly usingpip,condaandnpm) are found inbuild-support/alpha-build/core/<langugage>/setup.mk, where they
use dependency files and lock files that can be found in3rdparty/. In practice, runmake env-default-replicateinside a conda environment. Also make sure you also havenpminstalled becausemarkdownlintandbatsbash
testing framework come fromnpm(if you don't need them no need to worry aboutnpmjust exclude themarkdownenvironment rule from the pre-requisites ofenv-default-replicate)Create/Upgrade/Edit default environment:If you want to edit the default environment, for example to add,
remove, constrain packages edit therequirements.txtnot theconstraints.txtfile (in3rdparty/).
Theconstraints.txtis only used for reproducibility. If you just want to upgrade your third party dependencies
there is no need to temper with therequirements.txtfiles. Then runmake env-default-upgradeand check the lock
files back into git.Add a new environment:To add a new environment, first add the dependency files (e.g.requirements.txt) in3rdparty/<new-env-name>, add a new goal inbuild-support/alpha-build/extensions. For environment management over
time, we strongly encourage maintaining the approach split between creation/upgrade/edit and exact reproduction of
environments.Nested MakefilesSupposing you want to use a different config file forblackfor a project in your monorepo. You would have 2 options:
change the config file globally frombuild-support/alpha-build/config/python.mk(this would affect other projects) or create
anotherMakefilein your specific project if you just want different settings for your little project (whether this
is a good or a bad idea is more of a philosophical debate, I would argue that globally consistent config files are
preferable, but I acknowledge that this may be needed sometimes).
So, to have nestedMakefiles that work with different config files:# root/Makefileblack.%:# Add this goal to be able to delegate to inner Makefile-s$(MAKE)-C$(subst.,/,$*)custom-blackfmt-py:blackblack.my-proj# Add your custom "black" goals here# root/my-proj/Makefileinclude build-support/alpha-build/core/python/format.mkcustom-black:$(evaltargets:=$(onpy))$(MAKE)blacktargets="$(on)"BLACK_FLAGS="-S --config my-proj/pyproject.toml"This waymake fmt-pyat the root would call the regularblackrule but will also delegate to the inner Makefilecustom-blackrule to run the same tool differently. Don't forget to excludemy-proj/from black in the outer
Makefile, otherwise black would be run twice (once from the outside Makefile and again from the inner one).Generate requirements.txt for each sub-projectRunmake reqs-py.Generate setup.py for each sub-projectRunbuild-support/python/packaging/generate_pip_install_files.pyMarkdown badgeIf you like AlphaBuild, wear the Markdown badge on your repo:[![powered_by_alpha_build](https://img.shields.io/badge/Powered%20by%20-AlphaBuild-lightblue?style=flat&logo=CMake&logoColor=lightblue)
](https://github.com/cristianmatache/alpha-build)Out-of-the-box tools by languageAlphaBuild has recipes to run the following tools. However, if you don't use some of them you don't need to have them
installed in your environments. For example, let's say you don't usebanditthen you don't need to havebanditinstalled in your environment provided that you are not using thebanditgoal per-se or as part of a composite goal
likelint-pyorlint.Click to expand and see the supported tools!- Python:
  - Setup: `pip` / `conda`
  - Type-check: `mypy`
  - Test: `pytest` (along with `doctest` and other plugins)
  - Format + Lint: `black`, `docformatter`, `isort`, `autoflake`, `flynt`, `pre-commit`, `pyupgrade`
  - Lint only: `flake8`, `pylint`, `bandit`, `pydocstyle` (along with plugins like `darglint`, `flake8-bugbear`)
  - Package: `pipreqs`, `shiv`
- Jupyter:
  - Setup: `pip`
  - Format + Lint: `jupyterblack`, `nbstripout`
  - Lint only: `flake8-nb`
- Bash:
  - Setup: `npm` and `conda`
  - Test: `bats` (bash testing: `bats-core`, `bats-assert`, `bats-support`)
  - Format + Lint: `shfmt`
  - Lint only: `shellcheck`
- Haskell:
  - Lint: `hlint`
- YAML:
  - Setup: `pip`
  - Lint: `yamllint`, `prettier`Markdown:Setup:npmFormat + Lint:markdownlint,prettierHTML + CSS:Setup:npmFormat + Lint:prettierJavaScript:Setup:npmFormat + Lint:prettierTypeScript:Setup:npmFormat + Lint:prettierreStructuredText Text (.rst)Lint:rstcheckSwift:Format:SwiftLint,swift-formatLint:SwiftLint,swift-formatKotlin:Format:ktlintLint:ktlintIt is very easy to extend this list with another tool, just following the existing examples.Example monorepo running AlphaBuildTo see AlphaBuild at work in a real-world example checkhttps://github.com/cristianmatache/workspaceout.
Workspace extends AlphaBuild with support for Prometheus, Alertmanager and Grafana.High-level comparison with Pants, Bazel, Pre-commit and traditional MakefilesModern build tools like Pants or Bazel work similarly to AlphaBuild in terms of goals and targets, but they also add
a caching layer on previous results of running the goals. While they come equipped with heavy machinery to support
enormous scale projects, they also come with some restrictions and specialized maintenance and contribution requirements.For example, Pants which, in my opinion, is the most suitable modern build tool for Python doesn't allow building
environments with arbitrary package managers (e.g. conda, mamba), does not work on Windows, prohibits inconsistent
environments (which is good but sometimes simply impossible in practice), does not yet support multiple environments.
Bazel, requires maintaining the dependencies between Python files twice, once as "imports" in the Python files
(the normal thing to do) and twice in some specificBUILDfiles that must be placed in each directory (by contrast
Pants features autodiscovery). Maintaining the same dependencies in two places is quite draining. Of course, these tools
come with benefits like (remote) caching, incrementality and out-of-the-box support for hermetic packaging (e.g. PEXes),
remote execution etc. Moreover, playing with some new command line tools, or new programming languages / types of files
(e.g. Jupyter Notebooks, Markdown, YAML) may be challenging with these frameworks. The Pants community is very welcoming
and supportive towards incorporating new tools, so it would be good to give Pants a try first. However, if any of the
mentioned shortcomings is a hard requirement, Make seems like a good and robust alternative in the meanwhile which
withstood the test of time in so many settings. AlphaBuild's strengths are its flexibility, simplicity, transparency and
tooling richness. One can quickly hack/add a new tool, see the commands that run under the hood and does not need to
worry about BUILD files or the config language.Since AlphaBuild is essentially a script manager (Python, Bash, Perl, anything) enhanced with advanced
target/file/directory selection, AlphaBuild would allow an incremental adoption of large-scale build tools like Pants.
For example, in the main Makefile, one could do:# Makefilelint-with-pants:$(evalon:=::)# Default value if the user does not specify "on" in the terminal command like: make goal on=../pantslint$(on)lint:lint-mdlint-nblint-ymllint-with-pantssuch that running a command like the below would delegate most of the work to Pants while using AlphaBuild's core or
custom capabilities not yet available in Pants (e.g. linting notebooks, markdown or YAML files).makelinton=my-dir/Barepre-commitand typical usages of Make work exceptionally well on small projects, but they don't really scale
well to multi-projects monorepos. The build system proposed here, already incorporatespre-commitand is obviously
compatible with any existing Makefiles. This approach simply takes the idea of advanced target selection and ports it
over to classical techniques like pre-commit and Make.LimitationsSince AlphaBuild is essentially a small-repo tool (Python Makefile) adapted to work on larger codebases (through target
selection), there is a point from where it will no longer be able to scale up. Fortunately, that point is quite far away
from medium-sized repos/teams.In addition, AlphaBuild requires that the commands it builds are shorter thangetconf ARG_MAXcharacters.Detailed comparison: AlphaBuild vs Make vs Pre-commit vs Tox/Nox vs Bazel vs PantsMost small/medium popular Python open source projects use Make, Pre-commit and/or Tox/Nox with a crushing majority
for formatting, linting and/or testing and/or publishing. For the same purposes, fewer projects simply use a bunch
of Bash scripts or monorepo-style build tools like Bazel (or Pants, Buck, Please).
Make, pre-commit, nox/tox work pretty well together in the same repo and are often used so.Note 1:Every time we talk about Pants we talk about Pants V2, which is a fundamentally different product from
Pants V1. Even Twitter gave up on Pants V1 and is moving to Bazel (seehttps://twitter.com/jin_/status/1255133781876330497).Note 2:IMHO Pants (V2) is the best (but not yet perfect) build tool for large monorepos out there and has
great potential.Pros and ConsClick to expand and see the pros and cons of each tool!Tox/Nox:Pros:Good to run the same commands in multiple environmentsCross-platformCons:Does not scale to large reposPython-onlyNotes: can be called from within Make, but can also call Make and pre-commit commands
(see:https://tox.wiki/en/latest/config.html#conf-allowlist_externals)Pre-commit:Pros:Great to manage git hooks (as the name implies)Can run over all files or over a specific set of files but not directories
(slightly more advanced target selection)Sort of incrementalCons:Does not work on Windows (docs say it doesn't but in fact it does partly) [EDIT this is no longer the case.]Typically, very much geared towards format and lint.Notes: can be called from within MakeMake:Pros:Very flexible -> can essentially cover any tools/language (AlphaBuild is just Make with several built-in tools
and target selection) and any way of fetching dependencies, multiple 3rd party environments etc.Very transparent -> easy to supportCross-platformSupports nested MakefilesCons:More scalable than pre-commit/tox/nox but not as scalable or hermetic as Bazel/Pants/Buck/Please
(seehttps://github.com/thought-machine/please#why-please-and-not-makefor details).Notes: can run pre-commit, tox/nox but can also be run from tox/nox (not from pre-commit though)Bazel:Pros:Great for large scale projects (incremental, DAG, remote caching/execution)HermeticCons:Need to maintain special BUILD files in every directory in which all dependencies are written again
(once imported in the code, twice in the BUILD files)Low tool coverage for Python/Jupyter ecosystemSupport for 3rd party Python environments was not great (not sure if it is still the case)Has more sophisticated support needs (dedicated engineers and/or tuned CI infra)Notes: can be called from within MakePants V2:Pros:Great for large scale projects (incremental, DAG, remote caching/execution)Dependencies are auto-discovered in BUILD filesHermeticCons:Environment support: no conda, no multiple environments, no arbitrary ways to create environments, no inconsistent
envsNo support for Windows (only for WSL)Tool/language support could be better (Pants's support for Python is better than Bazel's though)Has more sophisticated support needs (e.g. dedicated engineers and/or tuned CI infra)Does not readily support the equivalent of nested MakefilesNotes:Can be called from within MakeStill need to have BUILD files in every directory (much easier to work with than in Bazel)Users by toolClick to expand and see who uses each tool!Nox, Tox, Pre-commitThese tools are simply extremely popular.Bazelraypytorch (Make for Python, Bash and CMake; in parallel with Bazel for other languages)selenium (with Rake i.e. ruby make) -> example BUILD filehttps://github.com/SeleniumHQ/selenium/blob/trunk/py/BUILD.bazeltensorflowkerasprotobufjax (with pre-commit)Pantshttps://www.pantsbuild.org/page/who-uses-pantsMakeDatapandas (in parallel with pre-commit)vaexpydanticpandera (in parallel with pre-commit, Make also runs nox)MLpytorch (Make for Python, Bash and CMake; in parallel with Bazel for other languages)xgboostDetermined AI (example of a more scalable Make-based infrastructure with nested Makefiles)VisualizationseabornDistributedcelery (in parallel with tox and pre-commit)Stream processingfaust (Make also runs pre-commit)Webrequestsgunicorn (in parallel with tox)sentry-python (Make also runs tox)AsyncIO (Aio-libs)aiohttp (Make also runs pre-commit)yarlaiomysqlaioredis-py (Make also runs pre-commit)aiopgDevOpsansible (in parallel with tox)pytest-testinfra (in parallel with tox)Documentationsphinx: the home repo (in parallel with tox)everything that uses sphinx for documentationPackagingpoetry (in parallel with tox and pre-commit)pyyaml (in parallel with tox)colorama (in parallel with tox)wrapt (Make also runs tox)pydata bottleneckfacebook researchmephistodemucsdiffqdoraapachesuperset (in parallel with tox and pre-commit)IBMlale (in parallel with pre-commit)compliance-trestle (Make also runs pre-commit)redditbaseplate.pybaseplate.py-upgradercqlmapperOther companies: AWS, Lyft, Microsoft, GoCardless, HewlettPackard