Package: universal-build
universal-buildUniversal build utilities for containerized build pipelines.Getting Started•Features•Documentation•Support•Contribution•Changelog•FAQUniversal-build is a set of utilities designed to build, test, package, and release software. It enables you to implement your build and release pipeline with Python scripts once and run it either on your local machine, in a containerized environment viaAct, or automated viaGithub Actions. It supports a monorepo or polyrepo setup and can be used with any programming language or technology. It also provides a full release pipeline for automated releases with changelog generation.WIP: This project is still an alpha version and not ready for general usage.Highlights🐳  Implement once and run locally, containerized, or on Github Actions.🧰  Build utilities for Python, Docker, React & MkDocs.🔗  Predefined Github Action Workflows for CI & CD.🛠  Integrated withdevcontainerfor containerized development.Getting StartedInstallationRequirements: Python 3.6+.pipinstalluniversal-buildUsageTo make use of universal build for your project, create a build script with the namebuild.pyin your project root. The example below is for a single yarn-based webapp component:fromuniversal_buildimportbuild_utilsargs=build_utils.parse_arguments()version=args.get(build_utils.FLAG_VERSION)ifargs.get(build_utils.FLAG_MAKE):build_utils.log("Build the component:")build_utils.run("yarn build")ifargs.get(build_utils.FLAG_CHECK):build_utils.log("Run linters and style checks:")build_utils.run("yarn run lint:js")build_utils.run("yarn run lint:css")ifargs.get(build_utils.FLAG_TEST):build_utils.log("Test the component:")build_utils.run("yarn test")ifargs.get(build_utils.FLAGE_RELEASE):build_utils.log("Release the component:")# TODO: release the component to npm with versionNext, copy thebuild-environmentaction from theactionsfolder into the.github/actionsfolder of your repository. In addition, you need to copy thebuild-andrelease-pipelineworkflows from theworkflowsfolder into the.github/workflowsfolder of your repository as well. Your repository should now contain atleast the following files:your-repository
  - build.py
  - .github:
    - actions:
      - build-enviornment:
        - Dockerfile
        - actions.yaml
    - workflows:
      - release-pipeline.yml
      - build-pipeline.ymlOnce you have pushed thebuild-environmentaction and thebuild-andrelease-pipelines, please look into theAutomated Build PipelineandAutomated Release Pipelinesections for information on how to run your build- and release-pipelines.You can find a more detailed example project with multiple components in theexamplesfolder.Support & FeedbackThis project is maintained byBenjamin Räthlein,Lukas Masuch, andJan Kalkan. Please understand that we won't be able to provide individual support via email. We also believe that help is much more valuable if it's shared publicly so that more people can benefit from it.TypeChannel🚨Bug Reports🎁Feature Requests👩‍💻Usage Questions🗯General Discussion❓Other RequestsDocumentationBuild Script CLI•Default Flags•API Reference•Update Universal BuildBuild Script CLIAny build script that utilizes thebuild_utils.parse_arguments()method to parse the CLI arguments can be executed with the following options:pythonbuild.py[OPTIONS]Options:These options correspond to the default flags documented in the next section.--make: Make/compile/package all artifacts.--test: Run unit and integration tests.--check: Run linting and style checks.--release: Release all artifacts (e.g. to registries like DockerHub or NPM).--run: Run the component in development mode (e.g. dev server).--version VERSION: Version of the build (MAJOR.MINOR.PATCH-TAG).--force: Ignore all enforcements and warnings.--skip-path SKIP_PATH: Skips the build phases for all (sub)paths provided here. This option can be used multiple times.--test-marker TEST_MARKER: Provide custom markers for testing. The default marker for slow tests isslow. This option can be used multiple times.-h, --help: Show the help message and exit.Default FlagsAt its core, universal-build will parse all arguments provided to the build script viabuild_utils.parse_arguments()and returns a sanitized and augmented list of arguments. Those arguments are the building blocks for your build script. You can utilize those arguments in whatever way you like. Here is an example on how to use those arguments in abuild.pyscript:fromuniversal_buildimportbuild_utilsargs=build_utils.parse_arguments()version=args.get(build_utils.FLAG_VERSION)ifargs.get(build_utils.FLAG_MAKE):# Run all relevant build commands.ifargs.get(build_utils.FLAG_TEST):# Run all relevant commands for testingtest_markers=args.get(build_utils.FLAG_TEST_MARKER)if"slow"intest_markers:# Run additional slow tests.The following list contains all of the default flags currently supported by universal-build:FlagTypeDescriptionFLAG_MAKEboolBuild/compile/package all artifacts.FLAG_CHECKboolRun linting and style checks.FLAG_TESTboolRun unit and integration tests.FLAG_RELEASEboolRelease all artifacts (e.g. to registries like DockerHub or NPM).FLAG_RUNboolRun the component in development mode (e.g. dev server).FLAG_FORCEboolIgnore all enforcements and warnings.FLAG_VERSIONstrSemantic version for the build. If not provided via CLI arguments, a valid dev version will be automatically calculated.FLAG_TEST_MARKERList[str]Custom markers for testing. Can be used to skip or execute certain tests.API ReferenceIn addition to argument parsing capabilities, universal-build also contains a variety of utility functions to make building complex projects with different technologies easy. You can find all utilities in the Python API documentationhere.Update Universal BuildTo update the universal-build version of your project, simply look up the most recent version of build-environment onDockerHuband set this version in the.github/actions/build-environment/Dockerfilefile of your repository:FROMmltooling/build-environment:<UPDATED_VERSION>In case you also run your build outside of the build-environment (locally), make sure to also upgrade universal-build on your local machine fromPyPi:pipinstall--upgradeuniversal-buildFurthermore, you can also check if thebuild-andrelease-pipelineworkflows have changed. In case of changes, update the workflows in your.github/workflowsfolder of your repository as well.FeaturesSupport for Nested Components•Automated Build Pipeline•Automated Release Pipeline•Containerized Development•Simplified Versioning•MkDocs Utilities•Python Utilities•Docker Utilities•ExtensibilityAutomated Build Pipeline (CI)Universal-build enables you to run your build pipeline on your local machine, in a containerized environment viaAct, or automated viaGithub Actions(= Continuous Integration).Local machine via build script (not recommended):Requirements:universal-buildand all the build requirements that your build script is using (e.g. yarn, pipenv, maven, ...) need to be installed on your machine.Execute the following command in the root folder of any component with a validbuild.pyscript:pythonbuild.py--make--check--testExecuting the build-pipeline directly via the build scripts is not recommended.Containerized environment via Act:Requirements:DockerandActare required to be installed on your machine.Execute this command in the root folder of your repository:act-b-sBUILD_ARGS="--check --make --test"-jbuildManually via Github Actions:In the Github UI, go toActions-> selectbuild-pipeline-> selectRun Workflowand provide the build arguments, e.g.--check --make --test.Automated via Github Actions (CI):With the default configuration, the build pipeline will run automatically via Github Actions on anypushevent to your repository. This automation can be referred to as continuous integration. You can also change the events that trigger the build-pipeline by modifying theonsection in the.github/workflows/build-pipeline.ymlfile. You can find more information about Github Actions eventshere.Automated Release Pipeline (CD)To release a new version and publish all relevant artifacts to the respective registries (e.g. Docker image to DockerHub) you can either trigger our release pipeline on your local machine, in a containerized environment viaAct, or automated viaGithub Actions(= Continuous Delivery).Local machine via build script (not recommended):Requirements:universal-buildand all the build requirements that your build script is using (e.g. yarn, pipenv, maven, ...) need to be installed on your machine.Execute the following command in the root folder of any component with a validbuild.pyscript:pythonbuild.py--make--check--test--release--version="<MAJOR.MINOR.PATCH>"Executing the release step directly via the build scripts is not recommended.Containerized environment via Act:Requirements:DockerandActare required to be installed on your machine.Execute this command in the root folder of your repository:act-b-sVERSION="<MAJOR.MINOR.PATCH>"-jreleaseIn case you also want to automatically create a valid Github release, you also need to provide a validGITHUB_TOKENas a secret (-s GITHUB_TOKEN=<token>). Please refer to the next section for information on how to finish and publish the release.On Github Actions (CD):Make sure that all required secrets for you release pipeline are configured in your Github repository. More informationhere.To trigger our release pipeline from Github UI, you can either close a milestone that has a valid version name (vMAJOR.MINOR.PATCH) or execute the release pipeline manually via theworkflow_dispatchUI in the Action Tab (Actions -> release-pipeline -> Run Workflow). The release pipeline will automatically run the build, check, test, and release steps, and create a pull request for the new version as well as a draft release on Github. This automation can be referred to as continuous delivery.After successful execution of the release pipeline, the following steps are required to finish the release:Merge the release PR intomain. Preferably via merge commit to keep the version tag in themainbranch. We suggest to use the following message for the merge commit:Finalize release for version <VERSION> (#<PR>).Adapt the changelog of the draft release on Github (in the release section). Mention all other changes that are not covered by pull requests.Publish the release.Resolve an unsuccessful release:In case the release pipeline fails at any step, we suggest to fix the problem based on the release pipeline logs and create a new release with an incrementedpatchversion. To clean up the unsuccessful release, make sure to delete the following artifacts (if they exist): the release branch, the release PR, the version tag, the draft release, and any release artifact that was already published (e.g. on DockerHub, NPM or PyPi).Support for Nested ComponentsYou can find the implementation of this multi-nested example in theexamplesfolder.Universal-build has excellent support for repositories that contain multiple nested components (aka Monorepo). The followingexamplesrepository has four components:docs,react-webapp,docker, andpython-lib:examples:
  - build.py
  - docs:
    - build.py
  - react-webapp:
    - build.py
  - docker:
    - build.py
  - python-lib:
    - build.pyEvery component needs its ownbuild.pyscript in the component root folder that implements all the logic to build, check, test, and release the given component. Thebuild.pyscript in the repo root folder contains the build logic that orchestrates all component builds. Universal-build provides thebuild_utils.build()function that allows to call the build script of a sub-component with the parsed arguments (find more info onbuildfunction in theAPI documentation).In between the build steps, you can execute any required operations, for example, duplicating build artifacts from one component to another. The following example, shows thebuild.pyscript that would support theexamplesrepository structure:fromuniversal_buildimportbuild_utilsargs=build_utils.parse_arguments()build_utils.build("react-webapp",args)build_utils.build("python-lib",args)build_utils.duplicate_folder("./python-lib/docs/","./docs/docs/api-docs/")build_utils.build("docker",args)build_utils.build("docs",args)With this setup, you can execute the build pipeline for the full project or any individual component. In case you only apply changes to a single component, you only need to execute thebuild.pyscript of the given component. This is a major advantage since it might massively speed up your development time.To run the build pipeline on you local machine only for a specific component, navigate to the component and run thebuild.pyscript in the component root folder (you can find all CLI build argumentshere):cd"./docs"&&pythonbuild.py[BUILD_ARGUMENTS]Alternatively, you can also run the component build containerized via Act:act-b-sBUILD_ARGS="[BUILD_ARGUMENTS]"-sWORKING_DIRECTORY="./docs"-jbuildOr directly from the Github UI:Actions->build-pipeline->Run workflow. The Github UI will allow you to set the build arguments and working directory.Simplified VersioningOnlysemantic versioningis supported at the moment.If you do not provide an explicit version via the build arguments (--version), universal-build will automatically detect the latest version via Git tags and pass a dev version to your build scripts. The dev version will have the following format:<MAJOR>.<MINOR>.<PATCH>-dev.<BRANCH>. This should be sufficient for the majority of development builds. However, the release step still requires to have a valid semantic version provided via the arguments.Python UtilitiesThebuild_pythonmodule of universal-build provides a collection of utilities to simplify the process of building and releasing Python packages. Refer to theAPI documentationfor full documentation on all python utilities. An example for a build script for a Python package is shown below:fromuniversal_buildimportbuild_utilsfromuniversal_build.helpersimportbuild_python# Project specific configurationMAIN_PACKAGE="template_package"args=build_python.parse_arguments()version=args.get(build_utils.FLAG_VERSION)# Update version in __version__.pybuild_python.update_version(os.path.join(HERE,f"src/{MAIN_PACKAGE}/__version__.py"),str(version))ifargs.get(build_utils.FLAG_MAKE):# Install pipenv dev requirementsbuild_python.install_build_env()# Build distribution via setuptoolsbuild_python.build_distribution()ifargs.get(build_utils.FLAG_CHECK):build_python.code_checks()ifargs.get(build_utils.FLAG_TEST):build_utils.run('pipenv run pytest -m "not slow"')if"slow"inargs.get(build_utils.FLAG_TEST_MARKER):build_python.test_with_py_version(python_version="3.6.12")ifargs.get(build_utils.FLAG_RELEASE):# Publish distribution on pypibuild_python.publish_pypi_distribution(pypi_token=args.get(build_python.FLAG_PYPI_TOKEN),pypi_repository=args.get(build_python.FLAG_PYPI_REPOSITORY))Thebuild_python.parse_arguments()argument parser has the following additional flags:FlagTypeDescriptionFLAG_PYPI_TOKENstrPersonal access token for PyPI account.FLAG_PYPI_REPOSITORYstrPyPI repository for publishing artifacts.And the following additional CLI options:--pypi-token: Personal access token for PyPI account.--pypi-repository: PyPI repository for publishing artifacts.Docker UtilitiesThebuild_dockermodule of universal-build provides a collection of utilities to simplify the process of building and releasing Docker images. Refer to theAPI documentationfor full documentation on all docker utilities. An example for a build script for a Docker image is shown below:fromuniversal_buildimportbuild_utilsfromuniversal_build.helpersimportbuild_dockerIMAGE_NAME="build-environment"DOCKER_IMAGE_PREFIX="mltooling"args=build_docker.parse_arguments()version=args.get(build_utils.FLAG_VERSION)ifargs.get(build_utils.FLAG_MAKE):build_docker.build_docker_image(COMPONENT_NAME,version)ifargs.get(build_utils.FLAG_CHECK):build_docker.lint_dockerfile()ifargs.get(build_utils.FLAG_RELEASE):build_docker.release_docker_image(IMAGE_NAME,version,DOCKER_IMAGE_PREFIX)Thebuild_docker.parse_arguments()argument parser has the following additional flags:FlagTypeDescriptionFLAG_DOCKER_IMAGE_PREFIXstrDocker image prefix. This should be used to define the container registry where the image should be pushed to.And the following additional CLI options:--docker-image-prefix: Docker image prefix. This should be used to define the container registry where the image should be pushed to.MkDocs UtilitiesThebuild_mkdocsmodule of universal-build provides a collection of utilities to simplify the process of building and releasing MkDocs documentation. Refer to theAPI documentationfor full documentation on all MkDocs utilities. An example for a build script for MkDocs documentation is shown below:fromuniversal_buildimportbuild_utilsfromuniversal_build.helpersimportbuild_mkdocsargs=build_utils.parse_arguments()ifargs.get(build_utils.FLAG_MAKE):# Install pipenv dev requirementsbuild_mkdocs.install_build_env()# Build mkdocs documentationbuild_mkdocs.build_mkdocs()ifargs.get(build_utils.FLAG_CHECK):build_mkdocs.lint_markdown()ifargs.get(build_utils.FLAG_RELEASE):# Deploy to Github pagesbuild_mkdocs.deploy_gh_pages()ExtensibilityExtend your build-environment image with additional toolsInstall the tools in the Dockerfile in your.github/actions/build-environment/Dockerfileas demonstrated in this example:FROMmltooling/build-environment:0.6.15# Install Go RuntimeRUNapt-getupdate\&&apt-getinstall-ygolang-goExtend the entrypoint of the build-environmentYou can extend or overwrite the default entrypoint with your custom entrypoint script (e.g.extended-entrypoint.sh) as shown below:FROMmltooling/build-environment:0.6.15COPYextended-entrypoint.sh/extended-entrypoint.shRUNchmod+x/extended-entrypoint.shENTRYPOINT["/tini","-g","--","/extended-entrypoint.sh"]The followingextended-entrypoint.shexample demonstrates how to extend and reuse the existing default entrypoint:# Stops script execution if a command has an errorset-eecho"Setup Phase"# TODO: Do your custom setups here# Call the default build-environment entrypoint.# Disable the immediate script execution stop so that the cleanup phase can run in any caseset+e# Thereby, you can reuse the existing implementation:/bin/bash/entrypoint.sh"$@"# Save the exit code of the previous commandexit_code=$?echo"Cleanup Phase"# TODO: Do additional cleanup# Exit the script with the exit code of the actual entrypoint executionexit$exit_codeSupport additional build argumentsThe following example demonstrates how you can support custom build arguments (CLI) in yourbuild.pyscript:importargparsefromuniversal_buildimportbuild_utilsparser=argparse.ArgumentParser()parser.add_argument("--deployment-token",help="Token to deploy component.",default="")args=build_utils.parse_arguments(argument_parser=parser)deployment_token=args.get("deployment_token")Once it is implemented in your build script, you can provide the build argument via the CLI options:python build.py --deployment-token=my-token. If your custom argument is astringand has a default string value (e.g.default=""), you can also provide the build argument via environment variables:DEPLOYMENT_TOKEN=mytoken python build.py.To use your custom build arguments inside the release pipeline, you need to add theDEPLOYMENT_TOKENas a secret to your Github repository (more infohere) and adapt the.github/workflows/release-pipeline.ymlfile by adding theDEPLOYMENT_TOKENas an environment variable (env) to the steps that need this build argument, for example:-name:release-componentsuses:./.github/actions/build-environmentenv:DEPLOYMENT_TOKEN:${{ secrets.DEPLOYMENT_TOKEN }}Use custom test markers to select tests for executionYou can provide any number of custom test markers via the--test-markerbuild argument. The following example shows how to react to custom test markers in your build script:ifargs.get(build_utils.FLAG_TEST):# Run your default testsif"integration"inargs.get(build_utils.FLAG_TEST_MARKER):# Run integration testsContainerized DevelopmentThebuild-environmentcan also be used for development inside a container. It is fully compatible with thedevcontainerstandard that is used by VS Code and Github Codespaces. The big advantage of using the build-environment for containerized development is that you only have to define your project dependencies in one location, and use this for development, local builds, and automated CI / CD pipelines.To use the build-environment for containerized development, just define a.devcontainer/devcontainer.jsonconfiguration inside your repository and link thebuild.dockerfileto the build-environment action in the.github/actions/build-environment/Dockerfilefolder. A minimaldevcontainer.jsonconfiguration could look like this:{
  "name": "build-environment",
  "build": {
    "dockerfile": "../.github/actions/build-environment/Dockerfile"
  },
  "settings": {
    // Set default container specific vs code settings
    "terminal.integrated.shell.linux": "/bin/bash"
  },
  "extensions": [
    // Add required extensions
  ]
}You can find a full examplehere.FAQ & Known IssuesAct: Error response from daemon - volume is in use(click to expand...)Sometimes the act containers are not removed properly and are blocking any subsequent act executions of your workflow. As a workaround, you can just remove all act containers:dockerrm-f$(dockerps-a--filter="name=^act-"-q)How to access the host from Docker Containers in GitHub Actions / Act or containers from the host(click to expand...)If you want to access the host (in act the pipeline container and on GitHub Actions the Linux VM) from within a container, you can set an environment variable in the workflow file with this step:-name:set-host-iprun:echo "::set-env name=_HOST_IP::$(hostname -I | cut -d ' ' -f 1)"# new syntax which is not yet supported on act:# run: echo "_HOST_IP=$(hostname -I | cut -d ' ' -f 1)" >> "$GITHUB_ENV"and then access the environment variable from within a container. This way you can, for example, access other containers with published ports or other host services.If you want to access a container directly without going through the host, you can get the IP address for example in the following way:container_id=<CONTAINER-ID-OR-NAME>container_ip=$(dockerinspect$container_id|jq-r'.[0].NetworkSettings.Networks.bridge.IPAddress')Note that the tooljqhas to be installed. If you run a python script and use the Docker client, the command looks different, of course.When you don't put starting containers into a custom network, the container is now reachable from the host (GitHub Actions & Act) as well as other containers under this$container_ipaddress. Yet, it isnotreachable from your local machine (e.g. your Mac). For that, you have to publish the port and use the$_HOST_IPaddress as explained above. The host port should be assigned randomly so that the setup is as host-independent as possible. To dynamically get the random port you can get it in the following way via bash:container_id=<CONTAINER-ID-OR-NAME>container_port=<INNER-CONTAINER-PORT>container_host_port=$(dockerinspect$container_id|jq-r'.[0].NetworkSettings.Ports["'$container_port'/tcp"][0].HostPort')In your code, you should then check whether the$_HOST_IPvariable is set and if not, uselocalhost. This way, it will work on GitHub Actions, Act, and your local machine. Here is a Python example:importdockerclient=docker.from_env()container_name="test-container"container_port=8080container=client.containers.run("some-image:1.2.3",name=container_name,ports={f"{container_port}/tcp":None},detach=True,)container.reload()ip_address=os.getenv("_HOST_IP","localhost")os.environ["CONTAINER_NAME"]=container_nameos.environ["CONTAINER_IP"]=ip_addresscontainer_host_port=container.attrs["NetworkSettings"]["Ports"][f"{container_port}/tcp"][0]["HostPort"]os.environ["CONTAINER_HOST_PORT"]=container_host_portContributionPull requests are encouraged and always welcome. Read ourcontribution guidelinesand check outhelp-wantedissues.Submit Github issues for anyfeature request and enhancement,bugs, ordocumentationproblems.By participating in this project, you agree to abide by itsCode of Conduct.Thedevelopment sectionbelow contains information on how to build and test the project after you have implemented some changes.DevelopmentRequirements:DockerandActare required to be installed on your machine to execute the build process.To simplify the process of building this project from scratch, we provide build-scripts that run all necessary steps (build, check, test, and release) within a containerized environment. To build and test your changes, execute the following command in the project root folder:act-b-jbuildRefer to ourcontribution guidesfor more detailed information on our build scripts and development process.LicensedMIT. Created and maintained with ❤️  by developers from Berlin.