PipelineLib/Reference

= Overview =
Reference documentation for PipelineLib configuration and interfaces. For the time being, this article is manually maintained according to the canonical code-generated documentation at doc.wikimedia.org.
= Configuration =
The .pipeline/config.yaml file specifies a dict/mapping with the following structure and values.
; pipelines: {...}: Required. A dict/mapping of arbitrary pipeline names to each pipeline's distinct configuration.
; Example: pipelines:
 foo:
 # foo pipeline definition ...
 bar:
 # ...
== Pipelines ==
Each pipeline is again a dict/mapping, with the following structure:
; blubberfile: file: Optional. The name of the Blubber configuration file used to build container image variants for the pipeline. The name is relative to the project's .pipeline/ directory. If not given, the default foo/blubber.yaml is used, where foo is the name of the pipeline.
; directory: directory: Optional. The directory, relative to the root of the project, where the application source code used by this pipeline is stored.
; fetch: {}: Optional. Git clone/fetch options for the patchset.
:; shallow: true|false: Optional. Whether to perform a shallow clone/fetch.
:: Default: true
:; depth: n: Optional. History depth of the fetch operation if shallow.
:: Default: 20
:; submodules: true|false: Optional. Whether to fetch/update submodules. Note this is performed recursively using the same shallow/depth options.
:: Default: true
; stages: [...]: Required. List of stage definitions. (See Stages below for supported stage configuration.)
; execution: [...]: Optional. The pipeline's stage execution graph represented as a list of stage adjacency lists. The default execution graph is simply all the defined stages performed serially.
; Example: pipelines:
 foo:
 blubberfile: my/blubber.yaml
 directory: src/
 stages:
 - name: built
 # stage actions ...
 - name: functional
 - name: analyzed
 - name: staged
 - name: accepted
 execution:
 # by default, the stage execution graph would be:
 #
 # built → analyzed → functional → staged → accepted
 #
 # but this defines a branching (parallelized) execution of:
 #
 # analyzed
 # ⇗ ⇘
 # built staged → accepted
 # ⇘ ⇗
 # functional
 #
 - [built, analyzed, staged] # static follows built, staged follows analyzed
 - [built, functional, staged] # functional follows built, staged follows functional
 - [staged, accepted] # accepted follows staging
== Stages ==
Each stage is a dict/mapping that provides a distinct stage name and a number of actions.
; name: stageName: Required. Name of the stage. Each stage name must be unique within the pipeline.
== Actions ==
A stage may include one or more of the actions listed below. Should a stage contain more than one action, they will be executed in the same order they are listed here.
Actions may result in dynamic output (image IDs, Helm release names, etc.) bound to predefined variable names. See each action's output variables reference and examples for details.
=== Setup ===
A special built-in action that clones the project repo and binds a number of initial output variables for use in subsequent pipeline stages. This action cannot be specified or altered by user configuration.
; Output variables:
 ${setup.project}
 ZUUL_PROJECT parameter value if getting a patchset from Zuul.
 Jenkins JOB_NAME value otherwise.
 
 ${setup.projectShortName}
 The string after the last forward slash in the ZUUL_PROJECT parameter if getting a patchset from Zuul.
 The string after the last forward slash in the Jenkins JOB_NAME value otherwise.
 ${setup.timestamp}
 Timestamp at the start of pipeline execution. Used in image tags, etc.
 ${setup.imageLabels}
 Default set of image labels:
 jenkins.job,
 jenkins.build,
 ci.project,
 ci.pipeline
 
 ${setup.commit}
 Git commit SHA of the checked out patchset.
 ${setup.branch}
 Git branch of the checked out patchset.
 ${setup.remote}
 Git remote URL of the checked out patchset.
 ${setup.tag}
 Newly created git tag if one has been created/pushed.
 (Additional job parameters and checkout variables)
 In addition to the above, all job parameters and checkout variables
 are included as well.
 For example, if the job is configured with a parameter called "FOO",
 the parameter value will be available as ${setup.FOO}.
 
 
=== Build ===
build: variant|{}
Blubber variant name and build context options.
Specifying build: "foo" expands to build: { variant: "foo", context: "." }.
variant
Blubber variant to build
Default: {$.stage} (same as the stage name)
context
Build context directory or URL supported by docker build
Default: "."
excludes
Patterns of files/directories to exclude from the build context. Providing this will result in any local .dockerignore file being overwritten prior to the build and only has an effect when the context is a local directory. Patterns must be valid Docker ignore rules.
; Output variables:
 ${[stage].imageID}
 Image ID of built image.
 ${[stage].imageLocalName}
 Randomly assigned image name for use within subsequent steps or
 stages where an ID is insufficient. For example, if you want to build
 another image that references this one as a base image, you must
 reference the name not an ID.
 
=== Run ===
; run: true|{}: Optional. Run a container using a previously built image variant.
:; image: id: Optional. A valid image ID (SHA256).
:: Default: ${.imageID} ID of image built in this stage using a build action.
:; arguments: []: Optional. List of run-time arguments passed to the variant's entry point.
:; env: {}: Optional. Environment variables to inject into the container at run time.
:; credentials: []: Optional. List of Jenkins credentials to inject into the container as environment variables at run time.
::; - id: string: The id of the credentials in Jenkins.
::; name: string: The name for the environment variable.
:; tail: integer: Optional. Save this many lines of trailing output as ${[stage].output}. Default: 0
; Output variables:
 ${[stage].output}
 If tailis specified, that number of trailing lines from the output (stdout) of container process.
 
; Example: pipelines:
 test:
 stages:
 - name: functional
 build: test
 run: true
; Example: pipelines:
 test:
 stages:
 - name: built
 build: test
 - name: functional
 run:
 image: '${built.imageID}'
 arguments: [unittests]
 env:
 VAR1_NAME: ${VALUE}
 VAR2_NAME: 'some value'
 credentials:
 - id: SONAR_API_KEY
 name: SONAR_API_KEY
=== Copy ===
copy
List of files/directories to copy from previously run variants to the local build context.
Default: []
from
Stopped container from which to copy files
Default: ${.container} (the container that ran during this stage's run step)
source
Globbed file path resolving any number of files relative to the container's root (/) directory.
destination
Destination file path relative to the local context
Default: source (the source path made relative to the local context directory).
archive
 Optional: Whether to archive file from destination in Jenkins, available at the project's lastSuccessfulBuild url found at: integration.wikimedia.org/ci/job/{pipeline-name}/lastSuccessfulBuild/ 
 This is most useful to pipeline users who need to generate small artifacts during a post-merge build process and later use/download those artifacts remotely. 
 Default: false 
; Shorthand: [source, source, ...]
=== Publish ===
 publish
 
 
 image
 Publish an image to the WMF Docker registry
 
 
 id
 ID of a previously built image variant
 Default: ${.imageID} (image built in this stage)
 name
 Published name of the image. Note that this base name will be
 prefixed with the globally configured registry/repository name
 before being pushed.
 Default: ${setup.project} (project identifier;
 see setup())
 tag
 Primary tag under which the image is published
 Default: ${setup.timestamp}-${.stage}
 tags
 Additional tags under which to publish the image
 Default: [${setup.tag}] The git tag if one has
 been pushed. Otherwise, nothing.
 
 
 
 
 Exports
 
 ${[stage].imageName}
 Short name under which the image was published
 ${[stage].imageFullName}
 Fully qualified name (registry/repository/imageName) under which the
 image was published
 ${[stage].imageTag}
 Primary tag under which the image was published
 ${[stage].publishedImage}
 Full qualified name and tag (${.imageFullName}:${.imageTag})
 
 
; Example: pipelines:
 integration:
 stages:
 - name: built
 build: production
 publish:
 image: true
; Example: pipelines:
 integration:
 stages:
 - name: built
 build: production
 - name: published
 publish:
 image:
 id: '${built.imageID}'
 name: my-project-image
 tag: production
 tags: [stable]
=== Promote ===
; promote: true|[]: Optional. Promote artifacts built by previous stages or this stage's previous actions by having PipelineBot submit a Gerrit patch (example).
:; - chart: chartName: Optional. Name of the deployment-chart to promote to.
::: Default: ${setup.project} Sanitized project name.
:; environments: []: Optional. List of the environments to promote to.
::: Default: [] An empty list, indicating all environments.
:; version: imageTag: Optional. A published image tag.
::: Default: ${.imageTag} Primary tag of this stage's published image.
; Example: pipelines:
 integration:
 stages:
 - name: built
 build: production
 publish:
 image: true
 promote: true
; Example: pipelines:
 integration:
 stages:
 - name: built
 build: production
 - name: published
 publish:
 image:
 id: '${built.imageID}'
 name: my-project-image
 tag: production
 tags: [stable]
 promote:
 - chart: my-chart-name
 environments: ['staging']
 version: production
=== Deploy ===
 deploy
 
 
 image
 Reference to a previously published image
 Default: ${.publishedImage} (image published in the
 publish() of this stage)
 cluster
 Cluster to target
 Default: "ci"
 Currently only "ci" is supported and this configuration is
 effectively ignored
 chart
 
 
 name
 Chart name from the ChartMuseum.
 Default: ${setup.projectShortName}
 version
 Chart version.
 Default: ${.imageTag} (The primary tag of image
 built in this stage.)
 
 
 test
 Whether to run helm test against this deployment
 Default: true
 overrides
 Additional values provided to the Helm chart.
 Default: none
 
 
 
; Example: pipelines:
 acceptance:
 stages:
 - name: staged
 build: production
 publish:
 image: true
 deploy:
 chart:
 name: my-thing
 version: 0.1.0
; Example: pipelines:
 acceptance:
 stages:
 - name: candidate
 build: production
 publish:
 image: true
 - name: staged
 deploy:
 chart:
 name: my-thing
 version: 0.1.0
 image: '${candidate.publishedImage}'
 test: true
 - name: accepted
 build: acceptance-tester
 run:
 arguments: ['https://${staged.releaseName}:8080/']
=== Export ===
; exports: {}: Optional. Binds new output values from the current stage to variable names that can be used in subsequent stage definitions.
; Example: pipelines:
 acceptance:
 stages:
 - name: staged
 build: production
 publish:
 image: true
 deploy:
 chart:
 name: my-project
 version: 0.1.0
 exports:
 url: 'https://${.releaseName}:8080/'
 - name: accepted
 build: acceptance-tester
 run:
 arguments: ['${staged.url}']
=== Trigger ===
 trigger
 
 
 name
 Name of the downstream job
 parameters
 Optional: Object specifying parameters to pass to the job
 progagate
 Optional: Boolean. If true (the default), then the result of this
 step is that of the downstream build (e.g., success, unstable, failure,
 not built, or aborted). If false, then this step succeeds
 even if the downstream build is unstable, failed, etc.
 
 wait
 Optional: Boolean. If true, wait for completion of the downstream
 build before completing this step. Default: false