Package: pyrate-build
pyratepyrateis a small python based build file generator targetingninja(s).It allows to describe the build process of small projects in a very simple way
using a python based configuration file.
This description is then turned into ninja build files, that enable a very
quick turnaround of project builds.Quick HOWTOThe following presents the necessary steps to quickly test the waters with this tool (assuming
ninja is already installed). These commands will installpyrate, generate the ninja
build file, build and execute a small executable:pipinstallpyrate-buildecho-e'#include <cstdio>\nint main() { printf("Ahoy World!"); return 0; }'>test.cppecho-e"executable('test', 'test.cpp')">build.pypyrateninja./testMore examples can be found in thegithubrepository.Installationpyrateis very easy to deploy - there are no particular installation steps to use it
once the single scriptpyrate.pyis available somewhere.
It can even board the project directory of your project and simply get called from there.
The only dependency to generate the ninja build files is having a working python installation.pyrateshould work out of the box with all python versions between 2.4 and 3.4.
To actually build the project, ninja has to be installed as well.The latest release version ofpyratecan be installed from the Python Package Index with:pipinstallpyrate-buildThe latest development version can be retrieved from the github repository:gitclonehttps://github.com/pyrate-build/pyrate-buildUsageThe quickest way to executepyrateis:pyrateWithout any parameters,pyratewill use the build configuration script (pyrate script) namedbuild.pyand create a ninja build file calledbuild.ninja.
If another pyrate script should be used, this can be specified as a positional argument.
The name of the created ninja build file can be customized using the option-oor--output.
The quick invocation shown above is therefore equivalent to the following invocation:pyrate--outputmybuild.ninjamybuild.pyWhen the script is started, it first changes the current directory to the directory
containing the build configuration script, so all path names are relative to it.pyratepath/to/mybuild.pywill therefore create the ninja build file path/to/build.ninjaIfpyrateis placed in a directory listed in the PATH environment variable (as automatically
done bypip installpyrate-build), the build configure script can be made executable to
invokepyrateautomatically by starting the build config script with:#!/usr/bin/env pyrateThere is some experimental support for the generation of plain makefiles,
which can be switched on with-Mor--makefile.Build File Configuration SyntaxThe build configuration forpyrateis written in python - so the full power
of python can be used to construct and describe the build process.
Several classes, functions and variables are available to ease and customize
the configuration of the build process.Specifying build inputIn general, a build input list that can be used to construct a build target takes the form:[<item1>, <item2>,...]Each item can be one of the following:astringis interpreted as a file name that is processed according to the rules specified by the packages in thetooldictionaryabuild targetas returned by the functions described inDefining build targetsor explicitly definedanexternal dependencyas returned by the functions described inExternal dependenciesor explicitly definedor any other kind ofBuildSource(explained later)Instead of a list, it is also possible to specify a space separated string of file names.
Such a string is automatically split, so the following two build input lists behave identically:"<file1> <file2>..."['<file1>','<file2>',...]Besides specifying file names by hand, there are many ways to get a list of files.
Common methods include calling the python functionos.listdiror using the helper
functionmatchprovided bypyrate:match(selector, dir_name ='.',recurse = False)This functions allows to select files from a directory using a string consisting
of black / white listing path name patterns.
The selector'*.cpp-test*.cpptest3.cpp *.h'for example selects all files ending with
‘.h’ and ‘.cpp’, with the exception of those ‘.cpp’ files that start with ‘test’ and are not
called ‘test3.cpp’.match_lib(dn ='.',recurse = False, lib_types = ['shared','static'])This functions allows to add all library files from a directory to the build process. By default,
both shared and static libraries will be picked up, but this can be changed with thelib_typesoption.Defining build targetsThere are four global helper functions to define object files, executables and libraries based
on a list of build inputs (which can be files, other targets or externals - as shown inSpecifying build input):executable(name, input_list, compiler_opts = None, linker_opts = None)shared_library(name, input_list = None, compiler_opts = None, linker_opts = None)static_library(name, input_list = None, compiler_opts = None, linker_opts = None)object_file(name, input_list, compiler_opts = None)Each function returns a build target object, that can be used as input / dependency of another function.
If multiple executables / libraries or object files with the same name but different inputs / options
are defined,pyratewill ensure that the output will have a unique name
(by appending a hash based suffix as needed). More details about this is available inTarget Collision Avoidance.If no input_list is given toshared_libraryorstatic_library, a BuildSource will be created,
that represents the specified library. Existing libraries can quickly be defined as dependencies this way,
but the name has to be a path to an existing file!These functions exist as global functions and as member functions of a so-called build context,
that describes how these functions are processed. The global functions are just executing
within the default build context.By default, all build targets that are defined by the above functions (or direct API calls) are built.
In order to select only certain default targets, the global variabledefault_targetscan be used:default_targets =[<target>,...](list of targets),<target>(single target) orNone(all targets are built)External dependenciesThe build environment / dependencies on external packages can be expressed using the
following functions / variables:find_external(name,...)use_external(name,...)The functionfind_externalsearches for some external dependency (built-in, pkg-config package
or self-defined) with the given name and returns either None or a representation of the dependency.use_externalwill first callfind_externaland add the external to the implicit input list
of the context if it exists.
The function takes additional positional and keyword arguments that depend on the external package.
A common argument for this function is a version selector, that is supplied through a global variable:versionThe comparison between this variable and a version specifier (eg.version >= 4.1)
will return a function that can be used to check the expression and is used by the external package finder.
A version specifier can be a string ('0.1.2') or tuple ((0, 1, 2)) with an arbitrary number
of delimeters, or a floating point number (1.2).
This allows for example to writefind_external('clang',version >= 3.5)to discover a clang installation with version 3.5 or later.Sincefind_externalalso integrates withpkg-config, a large number of external packages is
available - in addition to a handful of builtin external packages with special implementation features.
It is also possible to add new packages that are recognized.
A list of the builtin packages is presented inExternals.In order to simplify the creation of external packages that already provide a build configuration tool
to query version, linker or compiler options,pyrateprovides the function:create_external(name, build_helper,...)It requires the user to define a name for the external package and to supply the build configuration tool.
The values of additional parameters are interpreted as command line options for the build configuration tool.
The name of these additional parameters specify the name of the
rule that gets supplied with the flags given by the output of the build configuration tool.
However there are four parameters that have a special meaning:version_query- similar to the other parameters, the value of this parameter is used as build
configuration tool option to determine the current version of the external package.
As a consequence of providing this option, the resulting external package will support the parameterversion.version_parser- this parameter allows to supply a function that parses the version string
provided by the build configuration tool and is only used ifversion_queryis given.version- specifies required version (eg.version = version >= 11.5) and can only be used ifversion_queryis givenlink = optsis equivalent to specifyinglink_shared = opts,link_static = optsandlink_exe = optsThe following example recreates the builtin external package for wxWidgets and returns a representation
of the external package if a matching version is found:my_wxwidgets=create_external('wxwidgets',build_helper='wx-config',version_query='--version',link='--libs',compile_cpp='--cxxflags',version=version>=2.8)Configuration of the build environmentIt is possible to query the current version ofpyratevia the variable:pyrate_versionthis allows to ensure a compatible version ofpyratewith something along the lines of:assert(pyrate_version>'0.1.8')A build context allows for example to define implicit dependencies that are automatically
included in all generated object files, executables or libraries (viaimplicit_*options).
It is also possible to set base directories for the output generated by the build (viabasepath_*options).The default context used by the global functions presented inDefining build targetscan be set or accessed using the variable:default_context =Context(...)An instance of such a build context is created with:Context(...)- the most important parameters are:implicit_input,implicit_object_input,implicit_static_library_input,implicit_shared_library_inputandimplicit_executable_inputbasepath,basepath_object_file,basepath_static_library,basepath_shared_libray,basepath_executableThese parameters can also be changed on an existing context instance:default_context.basepath='build'A context also allows to access some additional settings - which are described in
more detail below. These settings are available as member functions of a context or
as global variables (that are provided by the default_context):toolsThis is a dictionary that contains links to external packages that provide the basic rules
and parameters that are used to process sources and generate targets.
This dictionary can be modified, but should not be overwritten.toolchainThis is a list ofToolchaininstances that is used to populate the tools dictionary
in reverse order. There are currently two toolchains available:gccandllvmThey can be accessed with the follwing two methods:find_toolchain(name,...)use_toolchain(name,...)These methods work in the same way as thefind_externalanduse_externalmethods.
The available toolchains and their options are presented inToolchains.
The following example would try to set the clang / clang++ compiler and llvm linker in the tool dictionaryuse_toolchain('llvm',version>=3.7,cpp_std='c++11',cpp_opts='-Wall')# is the same asllvm=find_toolchain('llvm',version>=3.7,cpp_std='c++11',cpp_opts='-Wall')ifllvm:toolchain.append(llvm)Target Collision AvoidanceAs explained inDefining build targets,pyratewill always ensure that targets with different inputs / options but
same name will generate different output files (by appending a hash based suffix as needed).
However it is possible to switch off the renaming of colliding targets for auniquetarget.
Beware: Having two different targets that switch off the renaming with the optionno_rename = Truewill abort the build file generation.
The following build configuration file:ex1=executable('example.bin','test.cpp',compiler_opts='-O1')ex2=executable('example.bin','test.cpp',compiler_opts='-O2')ex3=executable('example.bin','test.cpp',compiler_opts='-O3')ex4=executable('example.bin','test.cpp',compiler_opts='-O2',no_rename=True)print('hash(ex1) =%s'%ex1.get_hash())print('hash(ex2) =%s'%ex2.get_hash())print('hash(ex3) =%s'%ex3.get_hash())print('hash(ex4) =%s'%ex4.get_hash())will result (for example in an linux environment) in the generation ofthreeobject files namedtest_<hash1>.o,test_<hash2>.o,test_<hash3>.o, since there are only three different
settings used during the compilation oftest.cpp.
During the linking step, these object files will generatethreebinaries namedexample.bin,example_<hash4>.bin,example_<hash5>.bin.
Whereexample.binwas compiled with the compiler option ‘-O2’. To identify which
target belongs to which hash, the<target_obj>.get_hash()function can be used.However it isstronglyrecommended to always ensure collision free names for executables
and shared / static libraries.BuildSourceThe build source is the fundamental building block of pyrate. It is modeled by a classBuildSource,
which can be constructed with the following code:BuildSource(on_use_inputs=None,on_use_deps=None,on_use_variables=None)The three argumentson_use_inputs,on_use_depsandon_use_variablesspecify how a rule belonging
to a build target should react to having the BuildSource as input. Each argument can be a dictionary, where
the key specifies the rule (a rule name string orNoneto match any rule) and the value specifies foron_use_inputsa list of objects withnameattribute that is given as input arguments for the targeton_use_depsa list of objects withnameattribute that is specified as dependency of the targeton_use_variablesa dictionary with variables for the target. Probably the most important variable
isopts, which is used to supply options to rulesExamples for different build sources are:any string that is given as build input is converted into anInputFile` - a ``BuildSourcethat
forwards the specified file name to any rules (usingon_use_inputs)External- is a type ofBuildSourcethat specifieson_use_variablesamong other thingsall targets are BuildSources as well - so the result of ashared_librarycall can be used to
link another target against this libraymacro(expr)- creates a BuildSource that allows to define C/C++ preprocessor macros.Installing Targetsinstall(target_list, destination = None)This function will create install targets in the build file to install the given target / list of targets.
In addition to the install targets for the specific entries in thetarget_list, aninstalltarget
will be created that will contain all generated install targets.
The parameterdestinationallows to specify the installation path - if it is not given, the path
is taken from theplatform.install_pathsdictionary with the appropriate object target type.Subdirectoriesinclude(build_file_list, inherit = False, target_name = None)This function will read in the given build config file(s). If a directory is given
instead of a build config file,pyratewill enter the given directory and use the filebuild.pyif available. The parameterinheritallows to inheritbasepath_*andimplicit_*settings
from the current context. The parametertarget_nameallows to specify the name of the alias that
allows to build all included targets. By default, this target name is derived from the path given inbuild_file_list.Current implementation notice - the targets from the included file will be
adapted for proper paths and included in the build output of the main file. The goal is to allow
very loose coupling between the main project and the subsystem projects so each subsystem can
be independently processed without any changes.find_internal(name)This function allows to retrieve build targets that were created byexecutable,shared_library,static_libraryandobject_file. It will match against the user specified name, the installation
name (with platform specific extensions) and the build target name
(derived from the specifiedbasepathandbasepath_...and the installation name).
This is in particular useful when trying to specify dependencies one objects included from another file.ExternalsCurrently the following builtin externals are supported (listed with all possiblefind_externalarguments):gcc- GNU C compilerclang- LLVM C compilerg++,gpp- GNU C++ compilerclang++,clangpp- LLVM C++ compilergfortran- GNU Fortran compilerversion- specifies required version (eg.version >= 5.2)std- language standard version (eg.'c++14'or'latest').
A property with the same name allows to also set this value on an existing external (eg.tool['c'].std= 'c90').compiler- name of the executablecompiler_opts- options that are used during the compilation stageswig- The swig package also provides the member functionwrapperto describe the generation of automated interface codeversion- specifies required version (eg.version > '3.0.2')wrapper(target_language, library_name, interface_filename, libs =[<targets>...],context = None,...)-contextallows to specify a different build context, additional keyword parameters are forwarded to the shared_library
invokation that creates the wrapper librarylink-base- basic linker tools (usingldandar)link-gcc- calling linker via gcc (usinggccandgcc-ar)link-llvm- calling linker via llvm (usingclangandllvm-ar)link_static- path to the static linkerlink_static_opts- options for the static linkerlink_shared- path to the shared linkerlink_shared_opts- options for the shared linkerlink_exe- path to the executable linkerlink_exe_opts- options for the executable linkerpthread- posix thread librarystdlibcpp- GNU C++ librarylibcpp- LLVM C++ libraryroot- Library for large scale data analysis. This external also provides a member functiondictionaryto create ROOT I/O dictionary files.version- specifies required version (eg.version > '3.0.2')dictionary(name, header_list = None, include_list = None, opts = None, context = None,...)-nameis the name of the generated c++ dictionary file,header_listis the list of header files
with the C++ objects for which the serialization code will be generated,include_listis a list
of include directories that are needed by the dictionary generator to parse the given header files,optsare additional options given to rootcint,contextallows to specify a different build context, additional keyword parameters are forwarded to the shared_library
invokation that creates the wrapper libraryThe following list contains all builtin externals with a singlefind_externalparameterversion,
that specifies the required version (eg.version >= 2.6):fltk- FLTK GUI Libraryllvm- LLVM compiler infrastructure librariesodbc- Open Database Connectivity middlewarewx- wxWidgets GUI ToolkitMany more externals are available through the integration withpkg-config. The full list
of available packages on a system can be queried with:pkg-config--list-allAll packages listed in that overview can be accessed with thefind_externalfunction.ToolchainsThe following toolchains are currently available:gcc- the GNU compiler collection
This toolchain will activate thegccC compiler,g++C++ compiler and thegfortranFortran compiler. Linking will be done withlink-gccas driver.version- requested versionc_std,c_opts- control the std and flags of thegccexternalcpp_std,cpp_opts- control the std and flags of thegppexternalfortran_std,fortran_opts- control the std and flags of thegfortranexternallink_shared_opt,link_exe_opt- control the linker settingsllvm- the LLVM Compiler Infrastructure
This toolchain will activate theclangC compiler and theclang++C++ compiler.
Linking will be done with thelink-llvmpackage.version- requested versionc_std,c_opts- control the std and flags of theclangexternalcpp_std,cpp_opts- control the std and flags of theclang++externallink_shared_opt,link_exe_opt- control the linker settingsExampleThe basicpyratebuild configuration file for a simple C++ project with a single source file
producing a single executable looks like this:executable('test',['test.cpp'])A more complicated example is presented in the following code fragment. It demonstrates how tochange the default compiler toolchain to llvm (clang / clang++),define a native static and dynamic library from a set of files selected by wildcards,generate several executables accessing to the shared library andgenerate a wrapper library to access the C++ library from python (if swig is available).use_toolchain('llvm',version>=3.7,cpp_std='c++11',cpp_opts='-Wall')lib_files=match('*.cpp -test* -mylib.* -py_foo.cpp')static_library('libFoo',lib_files,compiler_opts='-O3')lib_reference=shared_library('libFoo',lib_files)python=find_external('python',version>2)swig=find_external('swig',version>=2)ifswigandpython:swig.wrapper('python','mylib','foo.i',libs=[lib_reference])forfninmatch('test*.cpp'):executable(fn.replace('.cpp','.bin'),[fn,lib_reference,find_external('pthread')])Many more examples with an increasing level of complexity are available in thegithubrepository.Changelog0.2.0changesrenamed external packages:clangtoclang++,gcctog++added external packages:clang,gcc,libstdc++,libc++,gfortran,link-base,link-gcc,link-llvmrenamedcompilervariable totools, changed to lower case slot names, usingcppinstead ofC++addedtoolchainandfind_toolchainto set multiple tools at once