Package: buildtool
OverviewThis is amakealternative with a simpler syntax and some useful features.ExampleHello, World!To create a buildtool project, place aWORKSPACEfile at the root of your project. For now, it can be empty. Rungit initto initialize a git repo in your project root.Then you can define build rules inBUILDfiles.  Imagine a simple project with one source file:src/main.c, that can be compiled withgcc main.c -c -o ../build/a.outto produce the output filebuild/a.out. Our project structure looks like:- WORKSPACE
 - src/
    - main.c
    - BUILDWe have placed aBUILDfile in thesrc/directory. We will declare a rule calledmainthat buildsmain.cand producesa.outin thisBUILDfile.# src/BUILDdefimpl(ctx):ctx.sh("mkdir -p ../build")ctx.sh("gcc main.c -c -o ../build/a.out")callback(name="main",deps=["main.c"],impl=impl,out="../build/a.out",)Let's see what this file does. Thecallbacktellsbuildtoolthat we are declaring a rule withname="main". Thedepsof a rule are the files that the rule requires to run. Theoutis a file (or list of files) that a rule produces from its deps. Finally, the implementationimplof a rule describes the actions that a rule performs in order to generate its output from its inputs.The paths passed into the callback are relative to the folder containing theBUILDfile.Animplfunction takes in a single argument: the build contextctx. When defining a rule's implementation, do not use functions likeos.systemdirectly - aside from invoking methods in thectx, allimplfunctions must be side-effect free. Here,ctx.sh(...)runs a shell command in the directory containing theBUILDfolder.Now we can run this rule. Runbuildtool maininside the project. The output filebuild/a.outshould have appeared.So far, we haven't done anything that couldn't be done with a simple Makefile.However, now try modifying the build rule sodeps=[], and rerunbuildtool main(or justbt mainfor short). We get an errorerror: no such file or directory: 'main.c'This is despite the fact thatsrc/main.cis clearly still in our repo. Buildtool runs all builds in a separate "sandbox" directory with only the explicitly stated dependencies made available. This means that it is (almost) impossible to make a mistake when specifying rule dependencies, since then the build will fail when run in the sandbox directory.Generic RulesNow imagine that we have a second source fileanother.cinsrc/, and want to compile it tobuild/b.out. One way would be to copy/paste the existingmainbuild rule and create a second rule to generateb.out. Instead, however, we can define a generic build rule, anddeclareit twice - once for each of our two targets.This can be done as follows:# src/BUILDdefdeclare(*,name:str,src:str,out:str):defimpl(ctx):ctx.sh("mkdir -p ../build")ctx.sh(f"gcc{src}-c -o{out}")callback(name=name,deps=[src],impl=impl,out=out,)declare(name="main",src="main.c",out="../build/a.out")declare(name="another",src="another.c",out="../build/b.out")Thedeclarefunction is just a standard Python function, and whensrc/BUILDis evaluated,callback()is called twice bydeclare()to declare each of our rules.Loading FilesIn larger repos, it may make sense to move these generic rules into a separate file. Let us do so here, creating arules.pyfile in the samesrc/directory, so our file hierarchy now looks like:- WORKSPACE
- src/
    - main.c
    - another.c
    - BUILD
    - rules.pyWe movedeclareintorules.py, so# src/rules.pydefdeclare(*,name:str,src:str,out:str):defimpl(ctx):ctx.sh("mkdir -p ../build")ctx.sh(f"gcc{src}-c -o{out}")returncallback(name=name,deps=[src],impl=impl,out=out,)To import it fromrules.py, we can use theload()function, as follows:# src/BUILDrules=load("rules.py")rules.declare(name="main",src="main.c",out="../build/a.out")rules.declare(name="another",src="another.c",out="../build/b.out")The path passed toload()is relative to the loading file.*.pyfiles can also load other*.pyfiles, so long as no cycles exist. It is not possible to load aBUILDfile from another file. Furthermore, it is only possible to declare a rule from aBUILDfile. Now we can runbt mainorbt anotherto generatea.outandb.out, respectively.Rule DependenciesIn addition to depending on other files, rules can depend on other rules. Unlike depending on files, depending on other rules will not change the files made available when running builds in the sandbox directory. Instead, if ruleAdepends on ruleB, then whenever we build ruleA, ruleBis guaranteed to also be built.For instance, we may wish to have a build target to build bothmainandanothertogether. This can be done as follows:# src/BUILDrules=load("rules.py")rules.declare(name="main",src="main.c",out="../build/a.out")rules.declare(name="another",src="another.c",out="../build/b.out")callback(name="all",deps=[":main",":another"],)By adding a:in front of the names of a dependency, we signify that it is the name of a rule, not the name of a file. When running builds from the command-line, we can also use this syntax to disambiguate between a rule and a file with the same name (e.g.bt :all), but it is not required if the target can be resolved unambiguously.Now, runningbt allwill build botha.outandb.out.Ifnameis passed tocallback(), it will return:<name>. This lets us avoid repeating rule names, as follows:# src/BUILDrules=load("rules.py")callback(name="all",deps=[rules.declare(name="main",src="main.c",out="../build/a.out"),rules.declare(name="another",src="another.c",out="../build/b.out")],)However, it is good practice to avoid "nesting" rules in this fashion.Paths and GlobbingRather than writing a separate rule for each.cfile insrc/, we may wish to automatically declare rules to build them. This can be done using thefind()function, which lets us glob for files, as follows:# src/BUILDfromos.pathimportbasenamerules=load("rules.py")all_rules=[]forsrcinfind("*.c"):name=basename(src)[:-2]all_rules.append(rules.declare(name=name,src=src,out=f"../build/{name}.out"))callback(name="all",deps=all_rules,)The path passed tofind()is relative to the directory containing theBUILDfile. We can also pass in paths relative to the root of the project, by prefixing them with//. So instead of writingfind("*.c"), we could have equivalently writtenfind("//src/*.c"). This syntax for paths relative to the project root can be used elsewhere where paths are required, such as an element ofdeps, the value ofout, or as an argument toload().If we runbt main, we get the following error:subprocess.CalledProcessError: Command '['gcc //src/main.c -o ../build/main.out']' returned non-zero exit status 1.We see thatfind()has returned a path relative to the project root, which cannot be directly passed to the shell. One fix would be to again useos.path.basenameinrules.pyto extract the filenamemain.c. However, this will cause problems if we later try to use our rule to compile a file in a subfolder. Instead, there exists a methodctx.relative()that takes in a path of any format and outputs a path relative to the working directory in an implementation.We can use this method to modifyrules.pyas follows:# src/rules.pydefdeclare(*,name:str,src:str,out:str):defimpl(ctx):ctx.sh("mkdir -p ../build")ctx.sh(f"gcc{ctx.relative(src)}-c -o{ctx.relative(out)}")returncallback(name=name,deps=[src],impl=impl,out=out,)For now, we will not worry about updating themkdircall to support subdirectories. Nowbt allshould work correctly.Dynamic DependenciesSometimes, we do not know all the dependencies of a rule in advance. For instance, imagine thatmain.cdepends onanother.c. If we runbt main, we get the errormain.c:2:10: fatal error: 'another.c' file not foundbecause only explicitly stated dependencies are available when running a build.One solution would be to updatedeclare()to take in a list of dependencies and manually specify thatmain.cdepends onanother.c. Alternatively, we can add a dependency dynamically when running the build.First, we need to know how to detect dependencies. If we rungcc main.c -MM, we obtain:$gccmain.c-MM
main.o:main.canother.cThis is in a format acceptable for Makefiles, but we need to process it to extract the raw file names. We can do so by modifyingrules.pyas follows:# src/rules.pydefdeclare(*,name:str,src:str,out:str):defimpl(ctx):ctx.sh("mkdir -p ../build")raw_deps=ctx.input(sh=f"gcc{ctx.relative(src)}-MM")deps=raw_deps.strip().split(" ")[1:]ctx.add_deps(deps)ctx.sh(f"gcc{ctx.relative(src)}-c -o{ctx.relative(out)}")returncallback(name=name,impl=impl,out=out,)First, we usectx.input(sh=...)to run a shell command and read back the stdout. After parsing the output to determine what files to depend on, we then usectx.add_deps()to add them asdynamic dependencies, replacing thedeps=[src]previously passed intocallback(). Finally, we run the standard compile, as before. Notice that the initialctx.input()call depended onsrc, but it was only added as a dependencyafterwards. This is allowed with dynamic dependencies, so long as after theimpl()completes, all the dependencies ever used have been added.Now,bt allsuccessfully builds the target files.WorkspacesWe now are able to build a simple project. When managing large projects, it is useful to also automate setup of the build environment, so a user can clone the repo, runbuildtool, and obtain the built output without any manual configuration. This is the role of theWORKSPACEfile.In aWORKSPACEfile, there is a new import available frombuildtool: theconfig. A simpleWORKSPACEfile may look like this:# WORKSPACEconfig.register_default_build_rule(":all")config.register_output_directory("build")config.require_buildtool_version("0.1.25")The default build rule is the rule that is invoked when runningbtin a project directory with no rule specified. The output directory is a directory that is cleaned when runningbt --clean, in order to remove previous build artifacts (multiple output directories can be registered). Finally, a minimumbuildtoolversion can be required, so that if old versions are used to build the project, a clear error message will be printed instructing the user to update.In addition, we can declaresetup rulesin theWORKSPACEfile. Unlike build rules, setup rules are not run in sandboxed directories, so their dependencies are not automatically enforced. While they must specify their outputs, since they run in the main project directory, they are not verified either. Unlike build rules, setup rules cannot usectx.add_deps(),  but must specify their dependencies statically.For instance, imagine thatgccis not present in the/usr/bin/directory, but is instead located somewhere else in thePATH. ThePATHis normalized to/usr/bin/in build rules, so our previous rule would not work since the shell would not be able to findgcc. Instead, we will use a setup rule to detectgccand add a symlink from//env/bin/gccto wherever it is located on the machine. Then we will use this symlink in our build rules to compile our*.cfiles.We modify ourWORKSPACEfile as follows:# WORKSPACEdefdeclare_gcc_symlink():defimpl(ctx):target=ctx.input(sh=f"which gcc").strip()ctx.sh("mkdir -p env/bin")ctx.sh("rm -f env/bin/gcc")ctx.sh(f"ln -s{target}env/bin/gcc")returncallback(name="gcc",impl=impl,out="env/bin/gcc",)callback(name="init",deps=[declare_gcc_symlink()])config.register_default_setup_rule(":init")config.register_default_build_rule(":all")config.register_output_directory("build")config.require_buildtool_version("0.1.25")Notice that theimpl()ofdeclare_gcc_symlink()clears any past outputs before rerunning, since it runs in the project directory directly, not in a sandbox. In addition, notice that we have registered adefault_setup_rulein our config. If such a rule is registered, buildtool will ensure that it is built before building any subsequent targets.To run thegccsetup rule separately, runbt setup:gcc. Unlike build rules, we cannot runbt env/bin/gccto regenerate the file - we can only run setup rules from the command line by their name. Thus, all setup rules are required to have a name, though they candependon source files or on files built by other setup rules.Next, we will modifyrules.pyto use//env/bin/gcc, instead of/usr/bin/gcc. Rather than hardcoding this new path intoctx.sh, we will modify thePATHused byctx.sh()to look in//env/binand then/bin, but not/usr/bin. This can be done as follows:# src/rules.pyENV=dict(PATH=["@//env/bin/","/bin"])defdeclare(*,name:str,src:str,out:str):defimpl(ctx):ctx.sh("mkdir -p ../build",env=ENV)raw_deps=ctx.input(sh=f"gcc{ctx.relative(src)}-MM",env=ENV)deps=raw_deps.strip().split(" ")[1:]ctx.add_deps(deps)ctx.sh(f"gcc{ctx.relative(src)}-c -o{ctx.relative(out)}",env=ENV)returncallback(name=name,impl=impl,out=out,)When we prefix a path with@//, it means that the path is relative to the project root directory, even if the build is being run in a sandbox. In contrast, if a path is prefixed with//, then it is treated as relative to the sandbox root directory when running a sandboxed builds. Paths can only be prefixed by@//in environment variables, not anywhere else.Notice that we have not defined thePATHinENVto be a string, but rather as a list of paths, using buildtool syntax. The buildtool will automatically resolve these paths to absolute paths and concatenate them together to form a string that will be passed into the shell environment. This is done so that absolute paths are never handled directly in build rules - they are to be avoided since they cause issues with caching. If an absolute path is needed as part of a shell command, it can be added to the environment and then accessed using shell syntax.We can now runbt allto regenerate the output. Notice that anenv/folder has been created, containing thegccsymlink. Conventionally, we do not register theenv/folder (or other targets built by setup rules) as an output directory, since it is unlikely to be the user's intention to clear it when runningbt --clear.