Package: lbuild
lbuild: generic, modular code generation in Python 3The Library Builder (pronouncedlbuild) is a BSD licensedPython 3 toolfor describing repositories containing modules which can copy or generate a set
of files based on the user provided data and options.lbuildallows splitting up complex code generation projects into smaller
modules with configurable options, and provides for their transparent
discovery, documentation and dependency management.
Each module is written in Python 3 and declares its options and how to generate
its content via theJinja2 templating engineor a file/folder copy.You caninstalllbuildvia PyPi:pip install lbuildProjects usinglbuild:modm generates a HAL for thousands of embedded devicesusinglbuildand a data-driven code generation pipeline.Taproot: a friendly control library and framework for RoboMaster robotsuseslbuild.OUTPOST - Open modUlar sofTware PlatfOrm for SpacecrafTuseslbuildto assemble an execution platform targeted at embedded systems running mission
critical software.The dedicated maintainer oflbuildis@salkinium.OverviewConsider this repository:$ lbuild discover
Parser(lbuild)
╰── Repository(repo @ ../repo)
    ├── Option(option) = value in [value, special]
    ├── Module(repo:module)
    │   ├── Option(option) = yes in [yes, no]
    │   ├── Module(repo:module:submodule)
    │   │   ╰── Option(option) = REQUIRED in [1, 2, 3, 4, 5]
    │   ╰── Module(repo:module:submodule2)
    ╰── Module(modm:module2)lbuildis called by the user with a configuration file which contains the
repositories to scan, the modules to include and the options to configure
them with:<library><repositories><repository><path>../repo/repo.lb</path></repository></repositories><options><optionname="repo:option">special</option><optionname="repo:module:option">3</option></options><modules><module>repo:module</module></modules></library>Therepo.lbfile is compiled bylbuildand the two functionsinit,prepareare called:definit(repo):repo.name="repo"repo.add_option(EnumerationOption(name="option",enumeration=["value","special"],default="value"))defprepare(repo,options):repo.find_modules_recursive("src")This gives the repository a name and declares a string option. The prepare step
adds all module files in thesrc/folder.Eachmodule.lbfile is then compiled bylbuild, and the three functionsinit,prepareandbuildare called:definit(module):module.name=":module"defprepare(module,options):ifoptions["repo:option"]=="special":module.add_option(EnumerationOption(name="option",enumeration=[1,2,3,4,5]))returnTruereturnFalsedefbuild(env):env.outbasepath="repo/module"env.copy("static.hpp")fornumberinrange(env["repo:module:option"]):env.template("template.cpp.in","template_{}.cpp".format(number+1))The init step sets the module's name and its parent name. The prepare step
then adds aEnumerationOptionand makes the module available, if the repository option
is set to"special". Finally in the build step, a number of files are generated
based on the option's content.The files are generated at the call-site oflbuild buildwhich would then
look something like this:$ ls
main.cpp        project.xml
 $ lbuild build
 $ tree
.
├── main.cpp
├── repo
│   ├── module
│   │   ├── static.hpp
│   │   ├── template_1.cpp
│   │   ├── template_2.cpp
│   │   └── template_3.cppDocumentationThe above example shows a minimal feature set, butlbuildhas a few more
tricks up its sleeves. Let's have a look at the API in more detail with examples
fromthe modm repository.Command Line InterfaceBefore you can build a project you need to provide a configuration.lbuildaims to make discovery easy from the command line:$ lbuild --repository ../modm/repo.lb discover
Parser(lbuild)
╰── Repository(modm @ ../modm)   modm: a barebone embedded library generator
    ╰── Option(target) = REQUIRED in [at90can128, at90can32, at90can64, ...This gives you an overview of the repositories and their options. In this case
themodm:targetrepository option is required, so let's check that out:$ lbuild -r ../modm/repo.lb discover-options
modm:target = REQUIRED in [at90can128, at90can32, at90can64, at90pwm1, at90pwm161, at90pwm2,
                           ... a really long list ...
                           stm32l4s9vit, stm32l4s9zij, stm32l4s9zit, stm32l4s9ziy]

  Meta-HAL target deviceYou can then choose this repository option and discover the available modules
for this specific repository option:$ lbuild -r ../modm/repo.lb --option modm:target=stm32f407vgt discover
Parser(lbuild)
╰── Repository(modm @ ../modm)   modm: a barebone embedded library generator
    ├── Option(target) = stm32f407vgt in [at90can128, at90can32, at90can64, ...]
    ├── Configuration(modm:disco-f407vg)
    ├── Module(modm:board)   Board Support Packages
    │   ╰── Module(modm:board:disco-f469ni)   STM32F469IDISCOVERY
    ├── Module(modm:build)   Build System Generators
    │   ├── PathOption(build.path) = build/parent-folder in [String]
    │   ├── Option(project.name) = parent-folder in [String]
    │   ╰── Module(modm:build:scons)  SCons Build Script Generator
    │       ├── Option(info.build) = no in [yes, no]
    │       ╰── Option(info.git) = Disabled in [Disabled, Info, Info+Status]
    ├── Module(modm:platform)   Platform HAL
    │   ├── Module(modm:platform:can)   Controller Area Network (CAN)
    │   │   ╰── Module(modm:platform:can:1)   Instance 1
    │   │       ├── Option(buffer.rx) = 32 in [1 .. 32 .. 65534]
    │   │       ╰── Option(buffer.tx) = 32 in [1 .. 32 .. 65534]
    │   ├── Module(modm:platform:core)   ARM Cortex-M Core
    │   │   ├── Option(allocator) = newlib in [block, newlib, tlsf]
    │   │   ├── Option(main_stack_size) = 3072 in [256 .. 3072 .. 65536]
    │   │   ╰── Option(vector_table_location) = rom in [ram, rom]You can now discover all module options in more detail:$ lbuild -r ../modm/repo.lb -D modm:target=stm32f407vgt discover-options
modm:target = stm32f407vgt in [at90can128, at90can32, at90can64, ...]

  Meta-HAL target device

modm:build:build.path = build/parent-folder in [String]

  Path to the build folder

modm:build:project.name = parent-folder in [String]

  Project name for executableOr check out specific module and option descriptions:$ lbuild -r ../modm/repo.lb -D modm:target=stm32f407vgt discover -n :build
>> modm:build

# Build System Generators

This parent module defines a common set of functionality that is independent of
the specific build system generator implementation.

>>>> modm:build:project.name  [StringOption]

# Project Name

The project name defaults to the folder name you're calling lbuild from.

Value: parent-folder
Inputs: [String]

>>>> modm:build:build.path  [StringOption]

# Build Path

The build path is defaulted to `build/{modm:build:project.name}`.

Value: build/parent-folder
Inputs: [String]The complete lbuild command line interface is available withlbuild -h.ConfigurationEven thoughlbuildcan be configured sorely via the command line, it is
strongly recommended to create a configuration file (default isproject.xml)
whichlbuildwill search for in the current working directory.<library><repositories><!-- Declare all your repository locations relative to this file here --><repository><path>path/to/repo.lb</path></repository><!-- You can also use environment variables in all nodes --><repository><path>${PROJECTHOME}/repo2.lb</path></repository><!-- You can also search for repository files --><glob>ext/**/repo.lb</glob></repositories><!-- You can also inherit from another configfile. The options you specifyhere will be overwritten. --><extends>path/to/config.xml</extends><!-- A repository may provide aliases for configurations, so that you canuse a string as well, instead of a path. This saves you from knowingexactly where the configuration file is stored in the repo.See also `repo.add_configuration(...)`. --><extends>repo:name_of_config</extends><!-- A configuration alias may also be versioned --><extends>repo:name_of_config:specific_version</extends><!-- You can declare the *where* the output should be generated, default is cwd --><outpath>generated/folder</outpath><options><!-- Options are treated as key-value pairs --><optionname="repo:repo_option_name">value</option><!-- An option set is the only one allowing multiple values --><optionname="repo:module:module_option_name">set,options,may,contain,commas</option></options><modules><!-- You only need to declare the modules you are actively using.The dependencies are automatically resolved by lbuild. --><module>repo:module</module><module>repo:other_module:submodule</module></modules></library>On startup,lbuildwill search the current working directory upwards for one or morelbuild.xmlfiles, which if found, are used as the base configuration, inherited
by all other configurations. This is very useful when several projects all
require the same repositories, and you don't want to specify each repository
path for each project.<library><repositories><repository><path>path/to/common/repo.lb</path></repository></repositories><modules><module>repo:module-required-by-all</module></modules></library>In the simplest case your project just<extends>this base config.<library><extends>repo:config-name</extends></library>Fileslbuildproperly imports the declared repository and modules files, so you can
use everything that Python has to offer.
In addition toimporting your required modules,lbuildprovides these
global functions and classes for use in all files:localpath(path): remaps paths relative to the currently executing file.
All paths are already interpreted relative to this file, but you can use this
to be explicit.repopath(path): remaps paths relative to the repository file. You should use
this to reference paths that are not related to your module.listify(obj): turns obj into a list, mapsNoneto empty list.listrify(obj): turns obj into a list of strings, mapsNoneto empty list.uniquify(obj): turns obj into a unique list, mapsNoneto empty list.FileReader(path): reads the contents of a file and turns it into a string.{*}Option(...): classes for describing options,see Options.{*}Query(...): classes for sharing code and data,see Queries.{*}Collector(...): classes for describing metadata sinks,see Collectors.Alias(...): links to other nodes,see Aliases.Configuration(...): links to a configuration inside the repository.Repositorieslbuildcalls these three functions for any repository file:init(repo): provides name, documentation and other global functionality.prepare(repo, options): adds all module files for this repository.build(env)(optional):onlycalled if at least one module within the
repository is built. It is meant for actions that must be performed foranymodule, like generating a global header file, or adding to the include path.# You can use everything Python has to offerimportantigravitydefinit(repo):# You must give your repository a name, and it must be unique within the# scope of your project as it is used for namespacing all modulesrepo.name="name"# You can set a repository description here, either as an inline stringrepo.description="Repository Description"# or as a multi-line stringrepo.description="""Multiline description.Use whatever markup you want, lbuild treats it all as text."""# or read it from a separate file altogetherrepo.description=FileReader("module.md")# lbuild displays the descriptions as-is, without any modification, however,# you can set a custom format handler to change this for your repo.# NOTE: Custom format handlers are applied to all modules and options.defformat_description(node,description):# in modm there's unit test metadata in HTML comments, let's remove themdescription=re.sub(r"\n?<!--.*?-->\n?","",description,flags=re.S)# forward this to the default formatterreturnnode.format_description(node,description)repo.format_description=format_description# You can also format the short descriptions for the discover viewsdefformat_short_description(node,description):# Remove the leading # from the Markdown headersreturnnode.format_short_description(node,description.replace("#",""))repo.format_short_description=format_short_description# Add ignore patterns for all repository modules# ignore patterns follow fnmatch rulesrepo.add_ignore_patterns("*/*.lb","*/board.xml")# Add Jinja2 filters for all repository modules# NOTE: the filter is namespaced with the repository! {{ "A" | repo.number }} -> 65repo.add_filter("repo.number",lambdachar:ord(char))# Add an alias for a internal configuration# NOTE: the configuration is namespaced with the repository! <extends>repo:config</extends>repo.add_configuration(Configuration(name="config",description="Special Config",path="path/to/config.xml")# You can also add configuration versionsrepo.add_configuration(Configuration(name="config2",description="Versioned Config",path={"v1":"path/to/config_v1.xml","v2":"path/to/config_v2.xml"})# See Options for more option typesrepo.add_option(StringOption(name="option",default="value"))defprepare(repo,options):# Access repository options via the `options` resolverifoptions["repo:option"]=="value":# Adds module files directly, or via globbing, all paths relative to this filerepo.add_modules("folder/module.lb",repo.glob("*/*/module.lb"))# Searches recursively starting at basepath, adding any file that# fnmatch(`modulefile`), while ignoring fnmatch(`ignore`) patternsrepo.add_modules_recursive(basepath=".",modulefile="*.lb",ignore="*/ignore/patterns/*")# The build step is optionaldefbuild(env):# Add the generated src/ folder to the header search path collectorenv.collect("::include_path","src/")# See module.build(env) for complete feature description.Moduleslbuildcalls these five functions for any module file:init(module): provides module name, parent and documentation.prepare(module, options): enables modules, adds options and submodules by
taking the repository options into consideration.validate(env)(optional): validate your inputs before building anything.build(env): generate your library and add metadata to build log.post_build(env, buildlog)(optional): access the build log after the build
step completed.Module files are provided with these additional global classes:Module: Base class for generated modules.ValidateException: Exception to be raised when thevalidate(env)step fails.Note that in contrast to a repository, modules must return a boolean from theprepare(module, options)function, which indicates that the module is available
for the repository option configuration. This allows for modules to "share" a
name, but have completely different implementations.Thevalidate(env)step is used to validate the input for the build step,
allowing for computations that can fail to raise aValidateException("reason").lbuildwill collect these exceptions for all modules and display them
together before aborting the build. This step is performed before each build,
and you cannot generate any files in this step, only read the repository's state.
You can manually call this step via thelbuild validatecommand.Thebuild(env)step is where the actual file generation happens. Here you can
copy and generate files and folders from Jinja2 templates with the substitutions
of you choice and the configuration of the modules. Each file operation is
appended to a global build log, which you can also explicitly add metadata to.Thepost_build(env)step is meant for modules that need to generate
files which receive information from all built modules. The typically use-case
here is generating scripts for build systems, which need to know about what
files were generated and all module's metadata.definit(module):# give your module a hierarchical name, the repository name is implicitmodule.name="repo:name"module.name=":name"# same as this# You can set a module description heremodule.description="Description"module.description="""Multiline"""module.description=FileReader("module.md")# modules can have their own formatters, works the same as for repositoriesmodule.format_description=custom_format_descriptionmodule.format_short_description=custom_format_short_description# Add Jinja2 filters for this modules and all submodules# NOTE: the filter is namespace with the repository! {{ 65 | repo.character }} -> "A"module.add_filter("repo.character",lambdanumber:chr(number))defprepare(module,options):# Access repository options via the `options` resolverifoptions["repo:option"]=="value":# Returning False from this step disables this modulereturnFalse# modules can depend on other modulesmodule.depends("repo:module1",":module2",":module3:submodule",...)# You can add more submodules in filesmodule.add_submodule("folder/submodule.lb")# You can generate more modules here. This is useful if you have a lot of# very similar modules (like instances of hardware peripherals) that you# don't want to create a module file for each for.classInstance(Module):def__init__(self,instance):self.instance=instancedefinit(self,module):module.name=str(self.instance)defprepare(self,module,options):passdefvalidate(self,env):# optionalpassdefbuild(self,env):passdefpost_build(self,env):# optionalpass# You can statically create and add these submodulesforindexinrange(0,5):module.add_submodule(Instance(index))# or make the creation dependent on a repository optionforindexinoptions["repo:instances"]:module.add_submodule(Instance(index))# See Options for more option typesmodule.add_option(StringOption(name="option",default="world"))defcommon_operation(args):"""You can share any function with other modules.This is useful to not have to duplicate code across module.lb files."""returnargs# See Queries for more query typesmodule.add_query(Query(name="shared_function",function=common_operation))# You can collect information from active modules, to use any post_build step# See Collectors for more collector typesmodule.add_collector(PathCollector(name="include_path",description="Global header search paths"))# Make this module availablereturnTrue# store data computed in validate step for build step.build_data=None# The validation step is optionaldefvalidate(env):# Perform your input validations here# Access all optionsrepo_option=env["repo:option"]defaulted_option=env.get("repo:module:option",default="hello")# Use proper logging instead of print() please# env.log.warning(...) and env.log.error(...) also availableenv.log.debug("Repo option: '{}'".format(repo_option))# You can query for optionsifenv.has_option("repo:module:option")orenv.has_module("repo:module"):env.log.info("Module option: '{}'".format(env["repo:module:option"]))# Call shared functions from other modules with argumentsshared_function=env.query("repo:module:shared_function")result=shared_function("argument")# Or just precomputed properties without argumentsdata=env.query("repo:module:shared_property")# You may also use incomplete queries, see Name Resolutionenv.has_module(":module")# instead of repo:moduleenv.has_option("::option")# repo:module:option# And use fnmatch queries# matches any module starting with `mod` and option starting with `name`.env.has_option(":mod*:name*")env.has_query("::shared_*")env.has_collector("::collector")# Raise a ValidateException if something is wrongifdefaulted_option+repo_option!="hello world":raiseValidateException("Options are invalid because ...")# If you do heavy computations here for validation, you can store the# data in a global variable and reuse this for the build stepglobalbuild_databuild_data=defaulted_option*2# The build step can do everything the validation step can# But now you can finally generate filesdefbuild(env):# Set the output base path, this is relative to the lbuild invocation pathenv.outbasepath="repo/module"# Copy single filesenv.copy("file.hpp")# Copy single files while renaming themenv.copy("file.hpp","cool_filename.hpp")# Relative paths are preserved!!!env.copy("../file.hpp")# copies to repo/file.hppenv.copy("../file.hpp",dest="file.hpp")# copies to repo/module/file.hpp# You can also copy entire foldersenv.copy("folder/",dest="renamed/")# and ignore specific RELATIVE files/foldersenv.copy("folder/",ignore=env.ignore_files("*.txt","this_specific_file.hpp"))# or ignore specific ABSOLUTE pathsenv.copy("folder/",ignore=env.ignore_paths("*/folder/*.txt"))# You can also copy files out of a .zip or .tar archiveenv.extract("archive.zip")# everything inside the archiveenv.extract("archive.zip",dest="renamed/")# extract into folder# You can extract only parts of the archive, like a single fileenv.extract("archive.zip",src="filename.hpp",dest="renamed.hpp")# or an a single folder somewhere in the archiveenv.extract("archive.zip",src="folder/subfolder",dest="renamed/folder")# of course, you can ignore files and folders inside the archive tooenv.extract("archive.zip",src="folder",dest="renamed",ignore=env.ignore_files("*.txt"))# Set the global Jinja2 substitutions dictionaryenv.substitutions={"hello":"world","instances":map(str,env["repo:instances"]),"build_data":build_data,# from validation step}# and generate a file from a templateenv.template("template.hpp.in")# any `.in` postfix is automatically removed, unless you rename itforinstanceinenv["repo:instances"]:env.template("template.hpp.in","template_{}.hpp".format(instance))# You can explicitly add Jinja2 substitutions and filtersenv.template("template.hpp.in",substitutions={"more":"subs"},filters={"stringify":lambdai:str(i)})# Note: these filters are NOT namespaced with the repository name!# submodules are build first, so you can access the generated filesheaders=env.get_generated_local_files(lambdafile:file.endswith(".hpp"))# and use this information for a new template.env.template("module_header.hpp.in",substitutions={"headers":headers})# Add values to a collector, all these are type checkedenv.collect("::include_path","repo/must_be_valid_path/","repo/folder2/")# The post build step can do everything the build step can,# but you can't add to the metadata anymore:# - env.collect() unavailable# You have access to the entire buildlog up to this pointdefpost_build(env):# The absolute path to the lbuild output directoryoutpath=env.buildlog.outpath# All modules that were builtmodules=env.buildlog.modules# All file generation operations that were doneoperations=env.buildlog.operations# All operations per moduleoperations=env.buildlog.operations_per_module("repo:module")# iterate over all operations directlyforoperationinbuildlog:# Get the module name that generated this fileenv.log.info("Module({}) generated the '{}' file".format(operation.module,operation.filename))# You can also get the filename relative to a subfolder in outpathenv.relative_output(operation.filename,"subfolder/")# or as an absolute pathenv.real_output(operation.filename,"subfolder/")# get all include paths from all active modulesinclude_paths=env.collector_values("::include_path")Optionslbuildoptions are mappings from strings to Python objects.
Each option must have a unique name within their parent repository or module.
If you do not provide a default value, the option is marked asREQUIREDand
the project cannot be built without it.defprepare(module,options):# Add option to moduleoption=Option(...)module.add_option(option)defbuild(env):# Check if options existexists=env.has_option(":module:option")# Access option value or use default if option doesn't existvalue=env.get(":module:option",default="value")# Access option values, this may raise an exception if option doesn't existvalue=env[":module:option"]If your option requires a unique set of input values, you can telllbuildto
wrap the option into a set usingmodule.add_set_option():defprepare(module,options):# Add an option, but allow a set of unique values as input and outputmodule.add_set_option(option)defbuild(env):# a unique set of option values is returned hereforvalueinenv[":module:option"]:print(value)Option sets are declared as comma-separated strings, so that inheriting
configurations or passing option values via CLI can overwrite these sets.
AStringOptioncannot be wrapped into a set for this reasons, however, it's
easy to split your string value in Python exactly how you want.<!-- All comma separated values are validated by the option --><optionname=":module:set-option">value,1,obj</option>If you want to preserve duplicates to count the number of inputs, use a list
optionmodule.add_list_option():defprepare(module,options):# Add an option, but allow a list of values as input and outputmodule.add_list_option(option)defbuild(env):# a list of option values is returned herevalue_count=env[":module:option"].count("value")Options can have a dependency handler which is called when the project
configuration is merged into the module options. It will give you the chosen
input value and you can return a number of module dependencies.defadd_option_dependencies(value):ifspecial_condition(value):# return single dependencyreturn"repo:module"ifother_special_condition(value):# return multiple dependenciesreturn[":module1",":module2"]# No additional dependenciesreturnNoneStringOptionThis is the most generic option, allowing to input any string.
You may, however, provide your own validator that may raise aValueErrorif the input string does not match your expectations.
You may also pass a transformation function to convert the option value.
The string is passed unmodified from the configuration to the module and the
dependency handler.defvalidate_string(string):if"please"notinstring:raiseValueError("Input does not contain the magic word!")deftransform_string(string):returnstring.lower()option=StringOption(name="option-name",description="inline",# or FileReader("file.md")default="default string",validate=validate_string,transform=transform_string,dependencies=add_option_dependencies)PathOptionThis option operates on strings, but additionally validates them to be
syntactically valid paths, so the filesystem accepts these strings
as valid arguments to path operations. This option does not check if the path
exists, or if it can be created, just if the string is properly formatted.Since an empty string is not a valid path, but it can be useful to allow an
empty string as an input value to encode a special case (like a "disable" value),
you may setempty_ok=Trueto tell the path validation to ignore empty strings.By default, the path input is not modified and must be correctly interpreted in
the context of the module that uses it (usually relocated to the output path).
However, if you want to input an existing path you should setabsolute=True,
so thatlbuildcan relocate therelative pathdeclared in the config files
to an absolute path, which is indepented of the CWD.
This is particularly useful if you declare paths in config files that are not
located at the project root, like options inherited from multiplelbuild.xml.option=PathOption(name="option-name",description="path",default="path/to/folder/or/file",empty_ok=False,# is an empty path considered valid?absolute=False,# is the path relative to the config file?validate=validate_path,dependencies=add_option_dependencies)BooleanOptionThis option maps strings fromtrue,yes,1,enabletobool(True)andfalse,no,0,disabletobool(False). You can extend this list with a
custom transform handler. The dependency handler is passed thisboolvalue.deftransform_boolean(string):ifstring=='y':returnTrue;ifstring=='n':returnFalse;returnstring# hand over to built-in conversionoption=BooleanOption(name="option-name",description="boolean",default=True,transform=transform_boolean,dependencies=add_option_dependencies)NumericOptionThis option allows a number from [-Inf, +Inf]. You can limit this to the
range [minimum, maximum].
The values can be specified directly as numeric value or as a string, which is
interpreted using theeval()function, so that you can describe values as more
intuitive formulas when necessary.
You can also suffix numbers with the SI multipliersK,M,G,T,Ki,Mi,Gi, andTito simplify formulas even further.
Note that you should use strings to specify precise floating point values such
as "1/3". The validation and dependency handlers are passed a numeric value.option=NumericOption(name="option-name",description="numeric",minimum=0,maximum="5Mi*2",default="1K",validate=validate_number,dependencies=add_option_dependencies)EnumerationOptionThis option maps a string to any generic Python object.
You can provide a list, set, tuple or range, the only limitation is that
the objects must be convertible to a string for unique identification.
If this is not possible, you can provide a dictionary with a manual mapping
from string to object. The dependency handler is passed the string value.option=EnumerationOption(name="option-name",description="enumeration",# must be implicitly convertible to string!enumeration=["value",1,obj],# or use a dictionary explicitlyenumeration={"value":"value","1":1,"obj":obj},default="1",dependencies=add_option_dependencies)QueriesIt is sometimes necessary to share code and data betweenlbuildmodules,
which can be difficult when they are split across files and repositories.
Queries allow you to share functions and computed properties with other modules
using the global name resolution system.defprepare(module,options):# Add queries to modulequery=Query(...)module.add_query(query)defbuild(env):exists=env.has_query(":module:query")# Access query value or use default if query doesn't existdata=env.query(":module:query",default="value")Note that queries must be stateless (aka. a pure function), since module build
order is not guaranteed. You must enforce this property yourself.You can discover all the available queries in your repository usinglbuild discover --developer.QueryThis wraps any callable object into a query. By default the name is taken from
the object's name, however, you may overwrite this.
Note that when using a lambda function, you must provide a name.
The description is taken from the objects docstring.defshared_function(args):"""Describe what this query does.:param args: what does it need?:returns: what does it return?"""returnargsquery=Query(function=shared_function)query=Query(name="different_name",function=shared_function)EnvironmentQueryThis query's result is computed only once on demand and then cached.The data must be returned from a factory function that gets passed the
environment of the first module to access this query.
The return value is then cached for all further accesses.
This allows you to lazily compute your shared properties only once and only if
accessed by any module.deffactory(env):"""Describe what this query is about, but don't document the `env` argument.:returns: an immutable object"""# You can read the build environment, but cannot modify it herevalue=env["repo:module:option"]# This return data is cached, so this function is only called once.return{"key":value}query=EnvironmentQuery(name="name",factory=factory)CollectorsThe post-build step has access to the build log containing the list of modules
that were built and what files they generated.
However, these modules also need to pass additional data to the post-build steps,
so that this information can be computed locally.lbuildallows each module to declare what metadata it wants using a collector,
which is given a name, description and optional limitations depending on type.
In the build step, each module may add values to this collector, which the
post-build steps then can access.defprepare(module,options):# Add a collector to modulecollector=Collector(...)module.add_collector(collector)defbuild(env):exists=env.has_collector(":module:collector")# Add values to this collectorenv.collect(":module:collector","value1","value2","value3")defpost_build(env):# Get all unique values from all operationsunique_values=env.collector_values(":module:collector")# get all values from all operations, even duplicates!all_values=env.collector_values(":module:collector",unique=False)Note that the ordering of values is preserved only relative to the order they
were added within a module and only if accessing them non-uniquely!
The above example will preserve the order ofvalue1,value2andvalue3,
only if the values are accessed not uniquely and only relative to each other.When you add values to a collector, the current operation is recorded, consisting
out of the current module, but you may also explicitly set this to a set of
file operations:defbuild(env):operation=env.copy("file.hpp")# Add values to this collector for the file operationenv.collect(":module:collector","values",operations=operation)# The return value from a file operation is actually a set of operationsoperations=env.copy("folder1/")# So you can extend this set for multiple file operationsoperations|=env.copy("folder2/")# And then filter this set of operationsoperations=filter(lambdaop:op.filename.endswith(".txt"),operations)# Only add this metadata to .txt filesenv.collect(":module:collector","txt-file-values",operations=operations)# A file operation object has these properties:operation.module# full module name, this is always availableoperation.repository# repository name, always availableoperation.has_filename# Some operations are specific to filesoperation.filename# The generated filename relative to outpathdefpost_build(env):# You can use these operation properties to filter the collector valuestxt_filter=lambdaop:op.repository=="repo"andop.filename.endswith(".txt")unique_txt_values=env.collector_values(":module:collector",filterfunc=txt_filter)# May contain duplicate values!all_txt_values=env.collector_values(":module:collector",filterfunc=txt_filter,unique=False)If you have very special requirements for the ordering values (for example
when collecting compile flags), consider iterating over the collectors items
manually, and possibly de-duplicating and reordering the values yourself.defpost_build(env):# Get the collector, may return None if collector does not exist!collector=env.collector(":module:collector")ifcollectorisnotNone:foroperation,valuesincollector.items():# values is a list and may contain duplicatesprint(operation.module,values)ifoperation.has_filename:# not all operations have filenames!print(operation.filename)Note that collector values that were added by a module without explicit
operations do not have filename, only module names!Collectors are implemented using the same type-safe mechanisms asOptions, the only differences are the lack of dependency handlers
and default values, since you can add default values in the modules build step.You may add collector values via the project configuration. However, since these
collector values cannot be overwritten by inheriting configurations use this with care.<library><collectors><collectname="repo:collector_name">value</collect><collectname="repo:collector_name">value2</collect></collectors></library>You can discover all the available collectors in your repository usinglbuild discover --developer.CallableCollectorThis collector allows you to collect callable objects, that the post-build step
can execute. This can be useful for providing specializations to the post-build
module without it needing to know how they work.collector=CallableCollector(name="collector-name",description="callable")StringCollectorSeeStringOptionfor documentation.collector=StringCollector(name="collector-name",description="string",validate=validate_function)PathCollectorSeePathOptionfor documentation.collector=PathCollector(name="collector-name",description="path",empty_ok=False,absolute=False)BooleanCollectorSeeBooleanOptionfor documentation.collector=BooleanCollector(name="collector-name",description="boolean")NumericCollectorSeeNumericOptionfor documentation.collector=NumericCollector(name="collector-name",description="numeric",minimum=0,maximum=100)EnumerationCollectorSeeEnumerationOptionfor documentation.collector=EnumerationCollector(name="collector-name",description="enumeration",enumeration=enumeration)Aliaseslbuildaliases are mappings from one lbuild node to another. They are useful
for gracefully dealing with renaming or moving nodes in your lbuild module tree.
Aliases will print a warning when accessed showing the alias description. Each
alias must have a unique name within their parent repository or module.Aliases can be used for any type of node that you want forwarded. You can also
add aliases that do not have a destination and will raise an exception with the
alias description. This allows you to remove lbuild nodes while providing
details for a workaround.defprepare(module,options):# Move option in this modulemodule.add_module(Option(name="option"))# Forward the old option to the new optionmodule.add_alias(Alias(name="option-alias",destination="option",description="Renamed for clarity."))# Instead of silently failing, you can provide a detailed description# about why the node was removed and what the workaround is.module.add_alias(Alias(name="removed-alias",description="Removed. Workaround: ..."))# You alias any type to any other node.module.add_alias(Alias(name="submodule-alias",destination=":other-module:submodule"description="Removed. Workaround: ..."))defbuild(env):# Will show a warning (once) that the alias has been movedexists=env.has_option(":module:option-alias")# Accesses :module:option insteadvalue=env[":module:option-alias"]# This will raise an exception with the alias descriptionvalue=env[":module:removed-alias"]# will check for :other-module:submodule insteadvalue=env.has_module[":module:submodule-alias"]Jinja2 Configurationlbuilduses theJinja2 template enginewith the following global
configuration:Line statements start with%% statement.Line comments start with%# comment.Undefined variables throw an exception instead of being ignored.Global extensions:jinja2.ext.do.Global substitutions are:time:strftime("%d %b %Y, %H:%M:%S")options: an option resolver in the context of the current module.Name Resolutionlbuildmanages repositories, modules and options in a tree structure and
serializes identification into unique string using:as hierarchy delimiters.
Any identifier provided via the command line, configuration, repository or
module files use the same resolver, which allows usingpartially-qualifiedidentifiers. In addition, globbing for multiple identifiers using fnmatch
semantics is supported.The following rules for resolving identifiers apply:A fully-qualified identifier specifies all parts:repo:module:option.A partially-qualified identifier adds fnmatch wildcarts:*:m.dule:opt*.*wildcarts for entire hierarchies can be ommitted:::optionA special wildcart is:**, which globs for everything below the current
hierarchy level:repo:**selects all inrepo,repo:module:**all inrepo:module, etc.Wildcarts are resolved in reverse hierarchical order. Therefore,::optionmay be unique within the context of:module, but not within the entire
project.For accessing direct children, you may specify their name without any
delimiters:optionwithin the context of:modulewill resolve to:module:option.Partial identifiers were introduced to reduce verbosity and aid refactoring,
it is therefore recommended to:Omit the repository name for accessing modules and options within the same
repository.Accessing a module's options with their name directly.Execution orderlbuildexecutes in this order:repository:init()Create repository optionsrepository:prepare(repo-options)Find all modules in repositoriesmodule:init()module:prepare(repo-options)Create module optionsResolve module dependenciesmodule:validate(env)submodules-first,optionalmodule:build(env)submodules-firstrepo:build(env):optionalmodule:post_build(env): submodules-first,optional