Chef Best Practices — Profile, Cookbook, and Policy
By: Larry, Peng Yang
Tags: Azure Devops, Automate, Chef, Best Practices, Cookbooks

Chef Best Practices — Profile, Cookbook, and Policy

Photo by isword on Unsplash

Highlights

Write InSpec Profiles once and use them everywhere Test-Driven Development using InSpec Profiles Test locally and build/release using CI/CD pipelines Use chef_cookbook_generator to generate the cookbooks and policies Use data bags even with kitchen test Use Azure DevOps as repositories and CI/CD pipelines Use kitchen-azurerm as the test driver Use Artifactory to store cookbooks and binaries Use Audit cookbook in Policies

Overview

Unlike other types of development, developing, testing, and releasing Chef stuff can be difficult and challenging. I am gonna walk you through how I minimize the work and maximize the efficiency of the development and deployment. Hope you will gain some inspiration from this article.

InSpec Profile

Profiles are the core of the Chef InSpec testing experience. Use Chef InSpec profiles to manage everything you need to run a security or compliance scan–attributes, metadata, and the tests themselves. The Profiles will be used as the tests of cookbooks and policies.

Chef Cookbook

A cookbook is a collection of recipes, attributes, files, templates, etc that are required to do automatic configuration management for your server (nodes).

Chef Policy

Policy maps business and operational requirements, process, and workflow to settings and objects stored on the Chef Infra Server. It is a collection of recipes from different cookbooks which will be set on Chef nodes.

Design Considerations

Naming Conventions

To manage and develop each profile, cookbook, and policy independently, we keep each of them in a separate repository, and the naming conventions for those repositories are as below.

Profiles: <organization_name>_<application_name>_ip or <organization_name>_<application_name>_<action_name>_ip , e.g. google_splunk_ip , google_splunk_upgrade_ip . Note: ip stands for InSpec Profile.

or , e.g. , . Note: ip stands for InSpec Profile. Cookbooks: <organization_name>_<application_name>_cb or e.g. google_splunk_cb . Note: cb stands for Cookbook. The reason we do not have action_name here is we generally write different recipes for different actions or tasks in a single cookbook, and use these recipes in Policies.

or e.g. . Note: cb stands for Cookbook. The reason we do not have here is we generally write different recipes for different actions or tasks in a single cookbook, and use these recipes in Policies. Policies: <organization_name>_<application_name>_pl (default action is create or install) or <organization_name>_<application_name>_<action_name>_pl, e.g. google_splunk_pl, google_splunk_upgrade_pl. Note: pl stands for Policy.

Relationship

A single cookbook can be used in multiple policies and a policy may include recipes from different cookbooks. The relationship between cookbook, profile, and policy is as below. As you can see, we generally associate a profile to a policy, but you could also associate multiple profiles to a policy if that works better.

cookbooks, policies, and profiles

Test, Build, and Release profiles, cookbooks, and policies

As shown in the diagram below, we test our profiles, cookbooks as well as policies from our local PC and Azure DevOps pipeline. And there are build and release pipelines for each of the repositories.

The build pipelines of them do the following:

Lint Profiles with cookstyle and check if the profile is valid or not with inspec check Lint Cookbooks with cookstyle and run kitchen test to test Lint Policies with cookstyle and run kitchen test to test

The release pipelines of them do the following:

Upload Profiles to Chef Automate Server with inspec compliance upload Upload Cookbooks to Artifactory with knife artifactory share Upload Policies to Chef Infra Server with chef push POLICY_GROUP POLICY_FILE

Architecture

Now, we will show how we develop, test, and release profiles, cookbooks and profiles in detail below.

1. Writing Profile

1.1 Generate profile

$ inspec init profile org_sandbox_ip

$ ls -lh

total 2.0K

drwxr-xr-x+ 1 *** *** 0 Dec 18 22:37 controls

-rwxr-xr-x+ 1 *** *** 268 Dec 18 22:42 inspec.yml

-rwxr-xr-x+ 1 *** *** 89 Dec 18 22:37 README.md

1. 2 Add .rubocop.yml

This file may not be required if you are not using Windows as your development environment. Please also check rubocop.

EndOfLine:

Enabled: false

LineLength:

Enabled: false

1.3 Add build-pipeline.yml

This file defines the build pipeline used in Azure DevOps. It mainly includes the following tasks:

Check version — to ensure the current version is greater than the version in master branch Install and configure Chef Workstation Validate InSpec Profile using the command below

inspec vendor ./ --overwrite

inspec check ./

4. Publish InSpec Profile to Release pipeline (only on master branch, which triggers the release pipeline to run)

1.4 Modify controls/example.rb

Rename example.rb to default_test.rb and write some tests in the file.

default_test.rb

1.5 Update inspec.yml

inspec.yml

1.6 Run local test

Just simply do the lint check and validate the profiles.

$ cookstyle .

C:/Users/***/org_sandbox_ip/.rubocop.yml: Warning: no department given for EndOfLine.

C:/Users/***/org_sandbox_ip/.rubocop.yml: Warning: no department given for LineLength.

Inspecting 1 file

. 1 file inspected, no offenses detected $ inspec check .

Location : .

Profile : org_sandbox_ip

Controls : 1

Timestamp : 2020-12-18T14:11:49+00:00

Valid : true No errors or warnings

1.7 Commit and push code to remote repository

$ git add .

$ git commit -m "initial commit"

$ git push -u origin --all

1.8 Configure Build pipeline and Release pipeline

Once the code is available on Azure DevOps, we can now use build-pipeline.yml to create the build pipeline and create a release pipeline that contains the following tasks.

Install Chef Workstation Upload Profile to Chef Automate Cleanup

The release pipeline will be triggered automatically when the corresponding artifact is published by the build pipeline.

2. Writing Cookbook

2.1 Generate cookbook

As we use chef_cookbook_generator to generate our cookbook, we can customize some of the files that are common in all cookbooks, these files include:

Policy.rb build-pipeline.rb — used as the template of the build pipeline in Azure DevOps kitchen.yml

$ chef generate cookbook org_sandbox_cb

2.2 Write recipe

default.rb

2.3 Update test

Make the folder and file in the test directory as below, this is where we use the Profile we wrote previously.

directory and file in test

Update inspec.yml as below

inspec.yml

There are several ways of specifying the source of profiles. Use path when you are still changing the profiles, use git when the profile is uploaded in the remote repo, you will need to configure the ACCESS_TOKEN (see how to generate it here) environment variable. Use compliance when the profile is released in Chef Automate, you will need to log into Chef Automation once using inspec compliance login so that it can pull profiles when you run your test.

(fetch profile from git url, you can configure this line in your ~/.bash_profile)

$ export ACCESS_TOKEN="xxx" (fetch profile from Chef Automate)

$ inspec comliance login <Automate URL> --insecure --use "<USER>" --ent=default --token="<TOKEN>"

Update default_test.rb as below which includes the controls we wrote in the depends section in inspec.yml.

default_test.rb

2.4 Write Policyfile.rb

We generally do not need to edit this file as it is generated from the template when chef generate cookbook is issued, but here is how it looks like. As you can see, we use the Artifactory as the default source for cookbooks which means it will install cookbooks from that URL to solve dependencies.

Policyfile.rb

2.5 Write kitchen.yml

We generally do not need to edit this file as it is generated from the template when chef generate cookbook is issued. And the following code demonstrates how the data bags and tests are configured. We will talk about data bags a little bit more later.

2.6 Update metadata.rb

metadata.rb

2.7 Write build-pipeline.yml

We generally do not need to edit this file as it is generated from the template. It basically has the following tasks:

Check version — to ensure the current version is greater than the version in master branch Install and configure Chef Workstation Run Kitchen Test Publish Cookbook to Release pipeline (only on master branch, which triggers the release pipeline to run)

2.8 Run Kitchen Test locally

We define multiple platforms to be tested on in our kitchen.yml, we generally test against all of them before we submit our code to remote repositories. You will notice that the profile will be loaded from git which is what I have configured in inspec.yml in test/integration/default folder.

$ cookstyle .

$ kitchen create

$ kitchen converge

$ kitchen verify

$ kitchen destroy

kitchen test

2.9 Commit and push code to remote repository

After we have tested our cookbooks, it is time to submit them to the remote repository.

$ git add .

$ git commit -m "initial commit"

$ git push -u origin --all

2.10 Configure Build pipeline and Release pipeline

Once the code is available on Azure DevOps, we can now use build-pipeline.yml to create the build pipeline and create a release pipeline that contains the following tasks.

Install and configure Chef Workstation Publish Cookbook on Artifactory Cleanup

The release pipeline will be triggered automatically when the corresponding artifact is published by the build pipeline.

3. Writing Policy

3.1 Generate policy

A policy has the same structure as a cookbook, and we also use chef_cookbook_generator to generate it, we can customize some of the files that are common in all cookbooks, these files include:

<org>_<app>_pl.rb, e.g org_sandbox_pl.rb, this is the policy file for the policy build-pipeline.rb — used as the template of the build pipeline in Azure DevOps kitchen.yml

$ chef generate cookbook org_sandbox_pl

3.2 Write attributes

As we will include the recipe of the Audit cookbook in our run list, we first need to specify the profiles that are associated with the Policy in the attributes. Although we can specify the source of the profile using other formats. e.g: git, as the policy will only be used by the nodes that are bootstrapped which allows the profiles to be fetched from Chef Automate via Chef Infra Server, it is better to use compliance here.

attributes/default.rb

3.3 Update test

Delete the file test/integration/default/default_test.rb as we use the tests (profiles) configured in kitchen.yml instead.

3.4 Write recipe

Basically, we include the recipes of the cookbook we have just developed and the recipes of the wrapper cookbook of Audit cookbook so that each time when the chef-client with this policy runs, the scan result will be generated as well — The result is available from the GUI of Chef Automate.

3.5 Write Policyfile — org_sandbox_pl.rb

This is the policy file for the Policy. We generally do not need to edit this file as it is generated from the template when chef generate cookbook is issued, but here is how it looks like.

org_sandbox_pl.rb

3.6 Update metadata.rb

As we include the recipes of org_sandbox_cb and org_audit_cb in the recipe of the policy, we need to write the dependencies in metadata.rb.

metadata.rb

3.7 Update kitchen.yml

We generally do not need to edit this file as it is generated from the template when chef generate cookbook is issued. Again, we can specify the profiles with different sources, here, let’s use the git URL just like what we did in the cookbook.

kitchen.yml

3.8 Write build-pipeline.yml

We generally do not need to edit this file as it is generated from a template, it basically has the following tasks:

Check version — to ensure the current version is greater than the version in master branch Install and configure Chef Workstation Run Kitchen Test Publish Policy to Release pipeline (only on the master branch, which triggers the release pipeline to run)

3.9 Run Kitchen Test locally

Similar to how we test cookbooks, we first test the policy locally.

$ cookstyle .

$ kitchen create

$ kitchen converge

$ kitchen verify

$ kitchen destroy

kitchen test

3.9 Commit and push code to remote repository

Now, it is time to push the code to the remote repository.

$ git add .

$ git commit -m "initial commit"

$ git push -u origin --all

3.10 Configure Build pipeline and Release pipeline

Once the code is available on Azure DevOps, we can now use build-pipeline.yml to create the build pipeline and create a release pipeline that contains the following tasks.

Install and configure Chef Workstation Publish Policy to Chef Infra Server Cleanup

The release pipeline will be triggered automatically when the corresponding artifact is published with the build pipeline.

About data bags

Data bags can be used to store environment variables, especially credentials. Although we did not use data bags in the example above, we use them often when we need to download packages or files from Artifactory and the API key of the Artifactory is stored in data bags. The code below shows how to use data bags in recipe.

use data bag items

It is trivial to use data bag items in recipes if the data bag items are stored in Chef Infra Server and the node is bootstrapped. However, it is tricky if we want to use the same code when we run kitchen test when the test driver (here, Azure VM) being tested knows nothing about Chef Infra Server, see here as well. As we can see from the explanation previously, we will run kitchen test at 2 different places, and we have to make data bags available at those places.

On our local PC In Azure DevOps build pipeline

Prepare data bags on local PC

To make the data bags available on our local PC across all the cookbooks and policies, we can create a data_bags folder in our working directory and write data bag items in the JSON file in that folder like below.

├── data_bags

├── vault

├── artifactory.json

├── org_sandbox_cb

├── org_sandbox_ip

├── org_sandbox_pl

Prepare data bags in Azure DevOps pipeline

We can first upload the JSON file (say, artifactory.json) as a secure file on Azure DevOps library and download it with the task in our pipeline. Then, we can create the same file structure (data_bags/vault/artifactory.json) using the JSON file like how we do on the local PC.

Conclusion

We have learned how to write and organize Profiles, Cookbooks, and Policies, as well as how to test them locally and release them using CI/CD pipelines. We have also understood how we can write the Profiles once and use them in Cookbooks and Policies by specifying the source of it (path, git, or compliance). Now, we have the policies released on Chef Infra Server, we can set the policies to nodes using the command below and then go to check your Chef Automate dashboard with a cup of coffee. :)

# set a policy to a node

$ knife node policy set <policy_group> <policy_name> # set policy to multiple nodes

$ knife exec -E 'nodes.find("name:svr*") {|n| n.default["org_sandbox_pl"]["server_name"]="prod"; n.save}'

References