Integrate Azure Key Vault With Azure Kubernetes Service
By: Noah Ispas
Tags: Secrets, Kubernetes, Azure Key Vault, Azure

Now we are good to go and create a simply configured Kubernetes cluster.

The listed commands will create a resource group named aks2akvrg in the eastus region and a Kubernetes cluster inside. The Kubernetes cluster is named myk8s and contains 1 worker node. The last command will prepare your kubectl to be connected to the freshly created myk8s Kubernetes cluster.

Create an Azure Key Vault

Next, we will create an Azure Key Vault and already add a secret. Therefore we need the following commands.

After running those commands, you should have an Azure Key Vault named myk8skv in our aks2akvrg resource group as well as your first secret inside the Key Vault named ExamplePassword. This was the simple part.

Allow AKS to access Azure Key Vault

The more complicated part, as we have our Kubernetes Cluster and the Key Vault in place now, is to connect both. This basically means we have to allow Kubernetes to access the Key Vault.

CSI Secret Store Provider for Azure

The first step to connect the Kubernetes Cluster with the Key Vault is to add the CSI Secret Store Provider. The provider consists of two components. One is responsible to get the secrets out of the Key Vault while the other one is responsible for mounting the secrets into Kubernetes pods. Mounting secrets into Kubernetes pods refers to the build-in way of secret handling.

With these commands we have prepared the connection between Kubernetes and Key Vault, now we need to care about authentication. Just check if the CSI provider was added properly by invoking the following command:

kubectl get pods

This should result in an output like this

Authenticate Kubernetes against Key Vault

Access to Azure Key Vault is managed by the Azure Active Directory. In order to allow Kubernetes to get secrets out of the Key Vault, it has to authorize against the Key Vault through the Active Directory.

You can use either managed identities or a service principal to achieve that. We will focus on the method that uses a managed identity. If you are interested in the method that uses a service principal, have a look at this article. Both methods are described there.

The picture shows the flow of a Kubernetes pod obtaining a secret from the Key Vault. The following steps will walk you through the preparation of everything that is needed for the flow.

Identity Permissions for Kubernetes

First, we need to grant Kubernetes and it’s nodes access to managed identities of Azure. Therefore we first need to get the clientId and the resource-group of the Kubernetes nodes. After that, we can create the respective role assignments. Therefore run the following commands.

AAD Pod Identity

Now, we can install the aad pod identity, which will allow a Kubernetes pod to make use of a Managed Identity in order to authenticate against the Key Vault.

Again, you should check if the respective pods are being deployed by running kubectl get pods which should result in a similar output:

Finally, create the Azure Managed Identity

az identity create -g aks2akvrg -n aks2kvIdentity

Read Permissions for Key Vault

The new Managed Identity will need to have read access on the Key Vault as well as the permission to grab secrets. Therefore run the following commands:

Create Azure Identity Binding

We have the AAD pod identity helm chart installed and the Managed Identity created with the access rules it needs. Now we need to connect them by creating an Azure Identity Binding. Create a YAML file with the following content and deploy it with kubectl apply -f filename.yaml . You can get the subscriptionId with echo $subId and the clientId with echo $clientId if you followed the last commands showed in the gists.

You can see that within the Azure Identity Binding manifest we reference our Managed Identity and we specify a selector. The selector will be used for pods that should be able to actually use the Managed Identity in order to authenticate against the Key Vault through the AAD.

Inject Secrets as Environment Variables

If you followed all the steps and everything went fine, congratulations on completing the most tricky part. The next steps will be much easier and more straight forward.

Create Custom Secret Provider

We created all the required components for the integration of Key Vault with Kubernetes. Now it’s time to actually grab secrets from the Key Vault. We have to be explicit here and create a Secret Provider Class, where we define all the secrets we need to import from the Key Vault.

Please create a YAML file with the following content. Note, that the <tenant id> should reference the tenantId of the Key Vault. You can retrieve it using this command: az keyvault show -n myk8skv | grep tenantId

There are two things of interest here.

In the spec:parameter:objects section, we define which secrets we want to actually grab from the key vault. In the spec:secrets section we can specify which of the grabbed secrets should be created as Kubernetes Secrets.

Don’t forget to deploy the Secret Provider Class with kubectl apply -f filename.yaml .

Mount Secrets into Application

By deploying the Secret Provider Class, the secrets will not be created in Kubernetes yet. This will only happen with the first pod, which mounts a volume utilizing CSI and referencing our Secret Provider Class. Also, the pod must make use of the selector that we specified in the Azure Identity Binding.

Have a look at the following pod definition

Create a YAML file containing the content and deploy it using kubectl apply -f filename.yaml

A successful deployment should result in Kubernetes secrets being created. Also if you connect to the created pod, the SECRET environment variable should be filled with hVFkk965BuUv .

Troubleshooting

If your secret is not mounted to the pod the first thing I would do is to check if the Kubernetes secret is created by running kubectl get secrets . After that, you might want to check the different parts that play a role in the whole process:

Describe the pod that mounts secrets volume using kubectl describe pod/inject-secrets-from-akv and you will see some errors in case something went wrong. The next place you could watch out for errors is the aad-pod-identity-nmi-* pod, where * is a random string. You will find the exact name of the pod by using kubectl get pods | grep aad-pod-identity-nmi . Look at the logs with kubectl logs pod/aad-pod-identity-nmi-* . If you could not find any hint, also have a look at the logs of the aad-pod-identity-mic-* pods. Again * is a random string that you can find out using kubectl get pods | grep aad-pod-identity-mic . You will get 2 results out of that command. Look at the logs of both with the following command: kubectl logs pod/aad-pod-identity-mic-* .

Considerations

If you followed all steps of the article with success, you should be able to inject secrets of Key Vault into your applications running inside of AKS. Now, let’s talk about some considerations for your future setup.

Volume Mounting

The secrets inside the Key Vault are only created as Kubernetes secrets with the first pod that mounts the volume pointing to the Secret Provider Class. What you can do is to add that volume to every pod that makes use of secrets. I don’t know how you think about that, but somehow I don’t like the idea that I need to mount a volume containing all the secrets of the Key Vault (at least all of them that are also defined in your Secret Provider Class) only to use a subset of them through environment variables.

The setup I used in my latest deployment scenario is the following. I have only one deployment containing a pod with the volume mounted that points to the Secret Provider Class. I named it create-secrets-from-akv.yml and it looks quite similar to the example deployment YAML I showed (named inject-secrets-from-akv ). Maybe you get it straight away. I am assuming that this deployment is up and running before any other deployment is made which needs secrets. This way you don’t need for every other deployment to explicitly add and mount the volume pointing to the Secret Provider Class and still, Kubernetes secrets are created.

The downside is obvious. If that deployment is not deployed upfront, secrets will not be created inside of Kubernetes.

Add or update Secret in Key Vault

An important question that will arise is what happens if you update a secret inside the Key Vault. Another might be, how to add new secrets.

For new secrets, it’s quite obvious that they will not be created inside Kubernetes, as every secret has to be defined explicitly in the Secret Store Provider Class. But at least for already existing secrets, I assumed that they will be updated in Kubernetes as well. Unfortunately, they will not be updated out of the box.