Project: Chef InSpec (Provisioning / Security & Compliance)
File: ci-cd.md

is the main coordinator of all CI-CD activity at Chef. It is configured through a . is the engine that actually executes all of the tests and builds for InSpec. While Buildkite coordinates all of the work, the actual infrastructure is managed by Chef in a variety of locations, including various clouds and datacenters. Buildkite is divided into two orgs, and (Open Source Software). Chef is private and Chef-OSS is public. You will need to have an account on BuildKite and be a member of both orgs to fully utilize the system. The defines the tests harness to be run. Most of the test scripts come down to executing "rake test" or similar. Omnibus is a system for building OS-specific packages of software, including all dependencies including Ruby runtimes. We use Omnibus to make RPMs, DEBs, MSIs, DMGs, and several other OS-specific formats that deploy inspec and its dependencies natively to the OS. The omnibus configuration for InSpec is stored at https://github.com/inspec/inspec/tree/master/omnibus and the main configuration file is . InSpec is published as a set of 4 gems - inspec, inspec-core, inspec-bin, and inspec-core-bin. When we release a new version to the public (a process we call "promotion" and typically happens on Wednesdays), we publish to rubygems.org. Rubygems are configured by declaring them in the expeditor configuration file in the `rubygems` section. Artifactory stores build artifacts (such as RPMs, MSIs, and gems) on a temporary basis. Artifactory is protected by the Progress VPN. . When a PR is opened, the subscription is activated. In addition, several defaults also apply. One default pipeline that gets activated is the Buildkite . This default is documented . The verify pipeline runs the linter, the unit tests, and the functional tests. It verifies that the code being submitted is sound. The verify pipeline is defined first in the file, which defines the separate Ruby versions, platforms, and environment variables to be passed to each one. Each runner calls a shell script, either or . These scripts are thin wrappers that install Bundler dependencies from a cache, then call into the Rakefile. The habitat artifact pipeline runs a smoke test to verify that the habitat build of inspec is valid. The habitat artifact pipeline is defined first in the file. It simply defines a linux runner and a windows runner, each with a dedicated script, or . The scripts install habitat, setup an origin key, build the package, and then run a When a PR is merged, the Expeditor actions under `merge_actions` are executed. Watch the Slack channel #inspec-notify for messages about the success or failure of various steps. Connect to the Progress VPN to fetch Expeditor logs in the event of a failure. This is controlled by the `builtin:bumpversion` and `bash:.expeditor/update_version.sh` subscriptions. If there are no GitHub labels on the PR, the patchlevel of the version will be bumped by executing the

First the VERSION file is updated, then the script runs to update the versions in the Ruby files. `builtin:bumpversion` is in charge of bumping versions in VERSION, and is controlled by GitHub labels on the PR. Most, though not all, PRs should not have any Expeditor control labels. Here are the Expeditor control labels, and the circumstances under which they should be used: Expeditor: Bump Minor Version - Use when a significant new feature is being released. Expeditor: Bump Major Version - Use when a major release is made - rarely used. Expeditor: Skip Version Bump - Use for non-code-change PRs, such as website or CI changes. This is controlled by the `trigger_pipeline:omnibus/release` subscription. The Omnibus build creates operating-system-specific packages for each platform on which we release Chef InSpec. Its drives a , which lists exactly which platforms to build. The Omnibus build is generally reliable, if somewhat slow. When the omnibus build succeeds, omnitruck delivers the packages to various package repos in `unstable` channels for public consumption. The packages are also delivered to (VPN required) The Chef Habitat build creates Habitat .hart packages for Linux and Windows. The drives a . When the hab build succeeds, the packages will be placed on the Hab builder in the `unstable` channel for public consumption. We also release a Docker image (see ), which contains a Linux system and Chef InSpec installed from a gem, with the ENTRYPOINT of the Docker image being `inspec` (see ). It's a simple way to ship the dependencies of `inspec`. When it succeeds, the Docker build is labeled as `current`. The `inspec`, `inspec-bin`, `inspec-core`, and `inspec-core-bin` gems are all built and placed on the internal Chef (VPN required) server. During promotion later, they publish to rubygems.org. The difference between the gems is as follows: `inspec` is a library gem, with full heavyweight dependencies, not encumbered by commercial licensing `inspec-bin` contains an `inspec` executable and is encumbered by commercial licensing `inspec-core` is a library gem, with lightweight dependencies and no compilation required at install time, and is not encumbered by commercial licensing `inspec-core-bin` contains an `inspec` executable and is encumbered by commercial licensing When expeditor is told to promote a release, using the slack command `/expeditor promote inspec/inspec:master 4.36.4` (for example), Expeditor automatically promotes the Omnibus packages from the unstable channel to the stable channel, publishing them to the various downloads sites. It also creates the `artifact_published:stable` event, which has numerous . Some of the more important ones: The Dockerfile is updated - mainly to update version numbers - and then the published Docker image is tagged with the labels "stable" and "latest". The gems are taken from Artifactory and published to Rubygems.org. This is done using an Expeditor built-in action. The gems must be owned by the user `chef`. The are copied to AWS S3 by a , and then reset back to an empty state. Another takes the release notes from S3 and creates a post on Discourse.