Package: buildpipe
BuildpipeA Buildkite plugin to dynamically generate pipelines. Especially useful
for monorepos where you want to create dependencies between projects.Exampleinitial_pipeline.ymlsteps:-label:":buildkite:"plugins:-jwplayer/buildpipe#v0.8.0:dynamic_pipeline:dynamic_pipeline.ymldynamic_pipeline.ymlprojects:-label:project1path:project1/# changes in this dir will trigger steps for project1skip:deploy*# skip steps with label matching deploy* (e.g. deploy-prd)-label:project2skip:testpath:-project2/-project1# you can trigger a project using multiple paths-label:project3skip:# you can skip a list of projects-test-deploy-stgpath:project3/somedir/# subpaths can also be triggeredsteps:# the same schema as regular buildkite pipeline steps-label:testenv:BUILDPIPE_SCOPE:project# this variable ensures a test step is generated for each projectcommand:-cd $$BUILDPIPE_PROJECT_PATH# BUILDPIPE_PROJECT_PATH will be set by buildpipe-make test-wait-label:buildbranches:"master"env:BUILDPIPE_SCOPE:projectcommand:-cd $$BUILDPIPE_PROJECT_PATH-make build-make publish-imageagents:-queue=build-wait-label:tagbranches:"master"command:-make tag-release-wait-label:deploy-stgbranches:"master"env:BUILDPIPE_SCOPE:projectcommand:-cd $$BUILDPIPE_PROJECT_PATH-make deploy-staging-wait-block:":rocket:Release!"branches:"master"-wait-label:deploy-prdbranches:"master"env:BUILDPIPE_SCOPE:projectcommand:-cd $$BUILDPIPE_PROJECT_PATH-make deploy-prodThe above pipelines specify the following:There are three projects to track in the repository.The env variableBUILDPIPE_SCOPE: projecttells buildpipe to
generate a step for each project if that project changed.Theskipoption will skip any step label matchingdeploy*.The env variableBUILDPIPE_PROJECT_PATHis created by buildpipe as
the project's path. If multiple paths are specified for a project,
it's the first path.Full working exampleFor a full working example, check outBuildkite Monorepo Example.ConfigurationPluginOptionRequiredTypeDefaultDescriptiondefault_branchNostringmasterDefault branch of repositorydiff_prNostringOverride command for non-default branch (see below for a better explanation of the defaults)diff_defaultNostringOverride command for default branch (see below for a better explanation of the defaults)dynamic_pipelineYesstringThe name including the path to the pipeline that contains all the actual stepslog_levelNostringINFOThe Level of logging to be used by the python script underneath; pass DEBUG for verbose logging if errors occurProject schemaOptionRequiredTypeDefaultDescriptionlabelYesstringProject labelpathYesarrayThe path(s) that specify changes to a projectskipNoarrayExclude steps that have labels that match the ruleOther useful things to note:Optionskipmake use of Unix shell-style wildcards (Look at
.gitignore files for inspiration)If multiple paths are specified, the environment variableBUILDPIPE_PROJECT_PATHwill be the first path.diff_commandsDepending on yourmerge
strategy,
you might need to use different diff command.Buildpipe assumes you are using a merge strategy on the default branch, which is assumed to bemaster.The command for the non-default branch (e.g. when you have a PR up) is:gitlog--name-only--no-merges--pretty=format:origin..HEADThe command for the default branch you merge to is currently:gitlog-m-1--name-only--pretty=format:$BUILDKITE_COMMITRequirementsPython3 is currently required, but we areplanningto convert buildpipe to a binary using Go.Just make sure to install Python3 in your agent bootstrap script or Dockerfile.Cloudformation bootstrap script# Install python3yum-yinstallpython3python3-pip
pip3install-UsetuptoolswheelAgent DockerfileFROM buildkite/agent:3.0

RUN apk add --no-cache \
  # Languages
  python3 py-setuptoolsTroubleshootingBuildpipe is incorrectly showing project as changedBuildkite doesn't by default do clean checkouts. To enable clean
checkouts set theBUILDKITE_CLEAN_CHECKOUTenvironment variable. An
example is to modify the pre-checkout hook,.buildkite/hooks/pre-checkout:#!/bin/bashset-euopipefailecho'--- :house_with_garden: Setting up pre-checkout'exportBUILDKITE_CLEAN_CHECKOUT="true"TestingmaketestLicenseMITAcknowledgementsThe rewrite to a plugin was inspired bygit-diff-conditional-buildkite-plugin.