Validating Cloud Custodian on AWS with Moto
By: Zak Scholl
Tags: AWS, Cloud Custodian, Testing, Mocking, Zendesk

At Zendesk, we adopted Cloud Custodian as a framework to implement compliance across our AWS accounts. Cloud Custodian has a Domain Specific Language that allows anyone with a penchant for reading documentation to create policies that can report or act on resources in AWS, GCP, or Azure.

Deleting or re-configuring resources with automation is scary. Especially in production. So we set out to rigorously test Cloud Custodian policies to provide us some assurance they would not cause an outage when deployed.

Traditionally, users test these policies by running Cloud Custodian in --dry-run mode. The dry-run reports all resources the tested policy identifies. Users can then review those resources for accuracy.

But this method raises some concerns:

What if a policy has the potential to identify hundreds of resources? Manual review can be tedious and error prone at such a scale.

How can we be assured that a particular action defined in Cloud Custodian’s DSL actually does what it says it does?

How do we know that upgrading the version of Cloud Custodian does not introduce a bug in a deployed policy’s execution logic?

We had these questions too. Our answer was Moto.

Moto is a mocking library for AWS. It allows a person to mock a specific service with a simple decorator on a Python unit test.

With this in mind, we implemented unit tests for our Cloud Custodian policies, starting with policies that had the greatest potential for negative impact.

Programmatically running Cloud Custodian

In order to test policies with a Python unit test, we needed a way to invoke Cloud Custodian in code. We were able to setup Cloud Custodian’s config and call their entry function from within our testing code. We wrapped this into a class and abstracted a run_policy method.

The c7n imports within the run_policy function are necessary to ensure that any mocks established by Moto are in place prior to a Boto3 import inside of Cloud Custodian.

The other important piece to this class is the get_all_resources method.

This method uses the captured result of patching Cloud Custodian’s method of dumping resources, c7n.utils.dumps , to produce a list of all resources that were identified by Cloud Custodian when executing the policy.

Returning this from the run_policy function allows us to assert that certain resources were or were not identified by Cloud Custodian.

Testing a policy

The Cloud Custodian policies we wanted to test examine all of the IAM Users in our accounts and check:

If access keys exist Have they gone unused for more than 90 days Are they at least 90 days old

Cloud Custodian should deactivate the keys when all of these conditions are met.

We can implement this logic with Cloud Custodian by building two policies. One that checks for keys that are at least 90 days old and never used. And another that looks for keys that are at least 90 days old and were last used more than 90 days ago.

Those policies look like this:

You can find out more about Cloud Custodian’s policy syntax in their docs.

The policy configuration above will parse AWS’ credential report and check the values for each IAM user’s Access Keys to see if:

the created date ( last_rotated ) is more than 90 days ago

) is more than 90 days ago it is active

has either never been used or was used more than 90 days ago

If these conditions are true, Cloud Custodian should disable the keys that match.

To test these policies, we needed to use Moto to create IAM users with and without access keys. We also needed to manipulate the create date and last used date of those keys.

But how can we fake access keys that are 90 days old for every test?

Enter Freezegun, a project by the same author as Moto!

Setting up the test

We subclassed our CustodianPolicyTest with an IamPolicyTest class and wrote our test methods there.

There is a lot to take in here, but the mechanics are straightforward. First, we patched two functions in Cloud Custodian that allowed us to capture and prevent Cloud Custodian writing its results to disk.

@patch("c7n.policy.Policy._write_file") @patch("c7n.utils.dumps")

@mock_iam

def test_disables_only_expired_keys(self, dumps, wf):

We also used Moto’s mock_iam decorator to mock all AWS IAM calls.

Next, we grabbed a reference to Moto’s backend for IAM

moto_iam_internal = get_backend("iam")["global"]

This snippet is important because Moto is not a perfect replica of AWS and does not have a complete implementation of all AWS API calls for each service. Moto’s backend provides direct access to the Python objects that store all the associated information for a service. In our case we used the IAM backend to directly modify the last_used and create_date of IAM access keys.

Following this, we used freeze_time to make our boto calls “happen” 91 days in the past. Then we created our users and keys, then set the last_used date explicitly using Moto’s IAM backend.

Then we did the same procedure 89 days in the past, also kicking in a canary user with no keys that should not have been touched.

This set up a few test cases:

IAM user #1 has a 91 day old key that is not used, which should be disabled IAM user #2 has a 91 day old key that was used 91 days ago, which should also be disabled IAM user #1 also has an 89 day old key that is not used, which should be left alone IAM user #2 also has an 89 day old key that was used 89 days ago, which should be untouched An IAM user has no keys and should not be touched

After setting up the test cases, we were able to run our Cloud Custodian policy, check the resources that the policy identified, and check the state of the IAM users after Cloud Custodian completed its actions.

Conclusions

While Cloud Custodian rules look straightforward to test, it took a lot of effort to get this working well, so we recommend this approach cautiously.

One particular difficulty we encountered was determining when a test failed if it was Cloud Custodian’s fault or a bug in Moto’s implementation of a particular AWS service. Ultimately, we discovered that the credential report endpoints for Moto were not implemented in exactly the same fashion as AWS behaves, so we had to make a few changes.

Another challenge was Moto’s incompleteness. They have partial implementations for many core AWS APIs, but in all likelihood you will run into an API that is not implemented when using Moto to test Cloud Custodian policies.

A caveat of this testing is that tests are only as good as you make them. If you miss an edge case that you did not consider, you could potentially cause an outage by destroying infrastructure in production. Overconfidence in your existing tests could cause blindness to a test case left unconsidered.

That being said, if there is good coverage for an API you are writing Cloud Custodian policies against, and you need a high level of assurance your policies will work as expected, use Moto! It is a highly active project and we hope it continues to be so we can use it for all of our policies in the future.

At the end of our testing process, we deployed these policies to several highly active AWS accounts with only one policy issue that was the result of a Cloud Custodian bug.