Setup AWS “Instance Profile” for on-prem servers
By: Jackie Chen
Tags: AWS, Aws System Manager, AWS Lambda, Aws Ssm

I assume you already knew what the EC2 instance profile is. Basically, the instance profile defines the permissions that a EC instance has. As it is associated with an IAM role which has a bunch of IAM policies attached. And the AWS credential in the EC2 metadata is automatically rotated by the instance profile.

In a hybrid environment (e.g on-prem + AWS cloud), it is not uncommon to run some AWS related tasks from on-prem servers (e.g deploy Cloudformation stack from on-prem Bamboo server). How do you normally manage the AWS credentials on those on-prem servers? The common practice that I have seen is to use either IAM user’s credential or service account to assume IAM role (federated access). But it looks to me, neither is ideal. As there is still an overhead to manage the IAM user or the service account (e.g rotate the password). I was wondering if it is possible to setup the instance profile for the on-prem servers?

I worked out a solution when I was working on the Confluence and Jira AWS migration project. There are about 2T data needs to be migrated to AWS from on-prem, and I chose to use S3 sync ( why not using AWS DataSync?). To run AWS S3 CLI on the on-prem servers, it requires AWS credentials. For security and ease of management overhead, I designed a solution that is able to automatically rotate the AWS credential on the on-prem server, and also does not impact the running job — Renew AWS credential for a long run AWS CLI process.

The idea is simple:

Setup a CodeBuild project in AWS which does two things: assume a least privilege role then pass the credential to the on-prem server via Ansible per schedule. e.g updating the content of the file ~/.aws/credential.

{

"Version": 1,

"AccessKeyId": "an AWS access key",

"SecretAccessKey": "your AWS secret access key",

"SessionToken": "the AWS session token for temporary credentials",

"Expiration": "ISO8601 timestamp when the credentials expire"

}

Setup a dynamic AWS profile in the on-prem server. e.g ~/.aws/credential

[default]

credential_process = cat ~/.aws/temp.json

And the solution worked great! We migrated the 2T data without any issues. The use case of the pattern can be much wider. e.g Rotate the credential for on-prem Bamboo agents.

The above “Instance Profile” for on-prem servers solution is still not easy enough, e.g the port 22 has to be open between On-prem and the VPC to allow Ansible to ssh into the on-prem box.

Let’s name above as version 1. I recently worked out the version 2, which is easier and more powerful by using Lambda, SSM managed instance and SSM Run Command.

The idea is still the same, but the implementation is better — Use Lambda to assume the role, then pass it to the SSM managed instance (mi-* Setting up AWS Systems Manager for hybrid environments) via SSM run command. Obviously, the benefits are less management overhead, and it can apply to multiple instances based on tagging.

Here is the sample Lambda function to rotate the credentials for on-prem bamboo agents (tag on-prem:bamboo-agent).

Originally published at http://jackiechen.blog on December 10, 2020.