Protect your Infrastructure with Real-time Notifications of AWS Console User Changes
By: Matthew Harper
Tags: Web Development, AWS, Programming, Infrastructure As Code, Cloudtrail

You’ve embraced infrastructure as code (IoC) and painstakingly created Terraform or CloudFormation to model your EC2 Instances, AutoScaling Groups, ECS clusters, and everything else. The devops and development teams have committed to only modify and create infrastructure using your chosen IoC tools — but, surprise, some people just won’t comply. Maybe someone gets lazy during feature development or takes a shortcut during incident response. Maybe you’ve outsourced your level one support and you don’t entirely trust that group yet. Whatever the reason, you would love to know when a user logs in to the AWS Console web site and manually changes things.

There are credible reasons for concern around this topic. Environmental drift is the term often used to describe the difference between the desired state (represented in your infrastructure as code) and the actual, live state of your infrastructure. This drift can eventually lead to failed deployments, scaling issues that impact customers, security gaps, or worst of all, a data breach.

In this article, we’re going to build a system to monitor your AWS infrastructure and send notifications whenever a user makes a change directly from the AWS Console. We’ll leverage CloudTrail to record the events and write them to S3. An S3 trigger will invoke a Lamba function. That function will filter the events and post the relevant ones to an SNS topic that your team can subscribe to. If that sounds intimidating, don’t worry; AWS does the heavy lifting and we just need to connect the pieces. Each step is detailed below — let’s get started.

Enable CloudTrail

CloudTrail is the key AWS service in this project. You might be able to build swap out the other components with functional equivalents, but CloudTrail monitors your entire AWS account for changes and publishes that data to log files in S3.

CloudTrail is often used for security investigations, compliance, and auditing purposes. and you can use its web interface to search the events it logs. Need to know who launched that expensive instance that is sitting idle? Check the CloudTrail logs. Track down who modified a launch configuration to find out why? CloudTrail. Basic search functionality for CloudTrail is available right on its AWS web interface, and you can also use AWS Athena to execute SQL style queries against the event logs.

By default, CloudTrail keeps an event log of the past 90 days of activity in your account. But those logs are not accessible in S3; for that, we need to create a new trail. AWS allows one trail of management events to be created for free.

Navigate to the CloudTrail console and choose ‘Create Trail.’ Provide a name, and select ‘Write-only’ for the event filter, because that’s all we’re interested in.

Scroll down to the Storage Location section, and select or create an S3 bucket to store your logs.

Now, to test this, go ahead and make some change in your AWS environment — launch a t.2 micro instance, create an empty Lambda function, anything will work. Then take a look at the S3 bucket that should contain our CloudTrail logs:

If you download a log file from this bucket, you’ll see an array (“Records”) of events. Each event has a ton of metadata attached to it — here’s a sample event (shortened) that is generated when you launch a new EC2 instance:

Create a Lambda to Process CloudTrail Logs

Now that each modification to our AWS infrastructure logged to S3, we need to write a function to parse the data. We’ll do this in Lambda because it’s serverless and that fits our need to invoke the function only when new events are recorded.

Navigate to the Lambda console and create a new function. Choose empty function and Python 3.8 as the runtime environment.

The function needs permission to read from S3 and to publish notifications to SNS. Select “Create a new role from AWS policy templates,” and choose both “S3 object read-only” and “SNS publish” from the Policy template drop-down. Provide a name for your role and then select Create Function.

After your function is created, scroll down to find the inline code editor and a very basic Python function.

Go ahead and copy all of the code below and paste it in the editor, replacing all the existing code:

Let’s review the code. Scroll down to find lambda_handler — that’s the entry point to our application. Because the Lambda is invoked by an S3 trigger, the parameter ‘event’ contains a list of S3 Events. Each event contains the key for an object that was created in the S3 bucket that our CloudTrail logs are written to. We loop through the list, and for each event, we download the file specified from S3 and unzip it.

Within each file, we have a list of CloudTrail events in JSON format. We don’t want to be notified of every CloudTrail event; the point of this exercise is to catch only events triggered by users via the AWS Console. Amazon does not provide an easy way to differentiate manual user changes from those invoked by the SDK or command line. Arkadiy Tetelma wrote a fantastic blog here in which he works to solve the same issue. We’re going to filter the CloudTrail events in that manner that Arkadiy describes inside our Lamda.

To execute that filter, we use a Python list comprehension to run filter_user_events on each item in the log and create output_dict containing only the matches.

The filter_user_events function checks relevant fields in the event against the filters we specify. First, it looks at the userAgent and tries to match a series of regular expressions that mimic the AWS Console URLs. Then it screens read-only events, like those that begin with ‘Get’ or ‘Describe’. The third check is against a set of specific event names that we ignore (like AWS Console logins.) The final check is whether the event was invoked by ‘AWS Internal.’

After we have the subset of events we’re interested in, we want to send the actual notification. We’ll use Amazon’s Simple Notification Service (SNS), which is a very straightforward service that allows you to create a topic and then publish/subscribe to it. The code to publish to the SNS topic is in our sns_publish function, and there are two wrapper methods (post_to_sns publishes the userName and eventName only, while post_to_sns_details publishes the entire JSON event.)

Note the sns_arn variable referenced here and defined near the top of the file; you’ll need to replace that with the ARN of the SNS topic that you want to publish to. Click save near the upper right of the screen to update your Lambda function with the code you pasted in; this change will take effect immediately. Then navigate to the SNS section of the AWS console, select Topics, and Create Topic.

Name the topic appropriately, and use the default setting for all of the other sections. Optionally, you can edit the Access Policy settings; the default policy is to allow only the topic owner to publish and subscribe to this topic. That’s fine for this demo, but if you want to use this in a larger organization, you’ll want to tailor that policy to your liking. Select ‘Create Topic’ to complete the setup process.

From this screen, you can copy the ARN of your topic and head back over to Lambda to update the code (replacing the value of the sns_arn variable.) You’ll also note that the topic has zero subscriptions; select ‘Create Subscription’, choose Email as the protocol, enter your email address, and select ‘Create Subscription’ again.

Make sure you check your email and confirm the subscription, otherwise you will not receive any of the messages published to the topic.

Trigger Lambda on S3 Events

So — a quick recap. CloudTrail logging is enabled, an SNS topic exists and we subscribed to it, our Lambda function is live (and the sns_arn variable is updated.) All that’s left to do is to create the trigger to invoke our Lambda function whenever a new CloudTrail log is created in the S3 bucket.

Navigate back to the Lambda console, click through to your function details, and select the ‘Add Trigger’ option near the top of the designer window.

Select the S3 bucket where your CloudTrail logs are written to. For Event Type, choose ‘all object create events’ to invoke our function whenever new objects are created within that bucket. Make sure ‘Enable Trigger’ is selected to activate this trigger immediately, and then click ‘Add.’

You should see the trigger on the designer view after it is created.

Test It Out

Now to test it out! A simple (and free) way to cause an event that will trigger our alert is to create a new Lambda function. If everything worked, shortly after the function is created, you’ll receive an email like this:

Since the code is set up to post both the short and detailed versions of the message to SNS, you should receive another email with the entire JSON message. If everything is working, you’re all set! Feel free to change to the code or the settings to meet your needs. If you make any improvements to the script, please send a PR to the GitHub repo. If something’s not quite right, check out the troubleshooting tips below.

Troubleshooting

Check the CloudTrail S3 bucket; are logs being created?

Check the Cloudwatch logs for the Lambda function. Logs should exist each time you expect the Lambda to be triggered. Look for write messages in the logs.

Check the S3 trigger for the Lambda function; is it monitoring the correct bucket?

Check the IAM role assigned to the Lambda; does it have the proper permissions to read from S3 and publish to SNS?

Does the sns_arn variable in the Lambda Python code match your SNS topic?

Did you subscribe to the SNS topic with your email address, and confirm the subscription?

If all else fails, you can uncomment and run the unit test in the Python code on your local machine. You can also add more extensive print() logging to the code and review the Cloudwatch logs to get a better idea of what’s happening while your function executes.

Resources

Detecting Manual AWS Console Actions — Arkadiy Tetelman

PyCloudTrailProcessor GitHub Repo