How to Manage Kubernetes Secrets Securely in Git
By: Coder Society
Tags: Kubernetes, Git, Clustering, Coding, Programming

Introduction

Kubernetes has a declarative approach to managing resources. While it’s a common practice to keep the Kubernetes manifest files in a Git repository, storing and managing Kubernetes secrets has always been a challenge. Sealed Secrets was developed to address this problem.

Sealed Secrets is a Kubernetes operator that allows you to store secrets in Git. It uses asymmetric key encryption to encrypt the secrets so that only the operator in the Kubernetes cluster can decrypt them. Therefore, the encrypted SealedSecrets are safe to store in a Git repository.

In this article, we will install the Sealed Secrets operator and demonstrate how to use it.

Sealed Secrets is composed of two parts:

The Sealed Secrets operator which runs in the Kubernetes cluster

The kubeseal client-side command-line tool, which is used to encrypt the secrets and generates the SealedSecrets Kubernetes resources.

Prerequisites

kubectl connected to a Kubernetes cluster

Install kubeseal

kubeseal is a command-line tool to encrypt secrets and generate SealedSecrets. On macOS, you can install kubeseal with Homebrew.

brew install kubeseal

You can find installation instructions for other platforms here: https://github.com/bitnami-labs/sealed-secrets#installation

Create a Kubernetes namespace for Sealed Secrets

Let’s start by creating a namespace for the Sealed Secrets operator.

kubectl create namespace sealed-secrets

Deploy the Sealed Secret operator

Download the Kubernetes manifests for the Sealed Secret operator.

curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/controller.yaml > sealed-secrets.yaml

As we do not want the operator to run in the kube-system namespace, we use sed to change it to sealed-secrets.

sed -i -e 's/kube-system/sealed-secrets/g' sealed-secrets.yaml

Now we use kubectl to deploy the Sealed Secret operator:

kubectl apply -f sealed-secrets.yaml

service/sealed-secrets-controller created

rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created

rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created

clusterrole.rbac.authorization.k8s.io/secrets-unsealer created

serviceaccount/sealed-secrets-controller created

customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created

role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created

role.rbac.authorization.k8s.io/sealed-secrets-key-admin created

clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created

deployment.apps/sealed-secrets-controller created

When the Sealed Secret operator starts, it will generate a public/private RSA key pair if it does not exist yet. Afterwards, it will print the public key which can be used for encryption / SealedSecret creation.

Verify deployment

Verify the deployment by checking the status of Kubernetes pods in the sealed-secrets namespace.

kubectl get pods --namespace sealed-secrets



NAME READY STATUS RESTARTS AGE

sealed-secrets-controller-6f5785b7f-m9hmq 1/1 Running 0 20m

Create the demo Kubernetes namespace

kubectl create namespace demo



namespace/demo created

Create a Kubernetes secret

Use kubectl to create a local secret manifest file.

We need to pass the following options:

namespace=demo – Specify the namespace where the secret should live

– Specify the namespace where the secret should live from-literal – Specify the secret key=value pairs

– Specify the secret key=value pairs output=yaml – Specify the output format of kubectl

– Specify the output format of kubectl dry-run=client – Only print the secret without actually creating it / sending it to the Kubernetes API server.

– Only print the secret without actually creating it / sending it to the Kubernetes API server. > db-secret.yaml – This tells kubectl to store its output in db-secret.yaml

kubectl create secret generic db-secret

--namespace=demo

--from-literal=username=admin

--from-literal=password=Y4nys7f11

--dry-run=client

--output=yaml > db-secret.yaml



cat db-secret.yaml



apiVersion: v1

data:

password: WTRueXM3ZjEx

username: YWRtaW4=

kind: Secret

metadata:

creationTimestamp: null

name: db-secret

namespace: demo

As you can see above, the normal Kubernetes resource contains the the kev=value pair, where the value is just base64 encoded and not encrypted.

Encrypt Kubernetes secret

Use kubeseal to encrypt the db-secret . The output is a SealedSecret resource that contains the encrypted secret.

We need to pass the following options:

controller-namespace=sealed-secret – Specify the namespace where the Sealed Secret operator runs (default is kube-system)

– Specify the namespace where the Sealed Secret operator runs (default is kube-system) format=yaml – Specify the output format of kubeseal (default is JSON)

– Specify the output format of kubeseal (default is JSON) < db-secret.yaml – This tells kubeseal to take the db-secret.yaml as input

– This tells kubeseal to take the db-secret.yaml as input > sealed-db-secret.yaml – This tells kubeseal to store its output in sealed-db-secret.yaml

kubeseal

--controller-namespace=sealed-secrets

--format=yaml < db-secret.yaml > sealed-db-secret.yaml



cat sealed-db-secret.yaml



apiVersion: bitnami.com/v1alpha1

kind: SealedSecret

metadata:

creationTimestamp: null

name: db-secret

namespace: demo

spec:

encryptedData:

password: AgDj/I4L89v9km3cuYyckAe1/sNcjH67fjhPAjZep37bQ+CCxPvQzEXJrQvI0fzzjF0cHU003JYCFUok4DFMi8nMw9YnXM//JiyBGvlEhmmQ/VI+HgruWcOEIuWw+NfR6GWVn0dLidnYOwnvJG4sAx3iPXcR2oyivj2v5yICx4CgAOS88tZF1AaSxYqhbAYfnD8IiNdfH6+gwYXwLp2pS1f/HLawtmnvAZLVb9JqP1Nidvj34a79xVbGZ7wz04W55uU/gNNZkP5QIvJ49DZkx4JqmKLISrXziPkgt/ZqB+346oXEM+iKoPOgn0sX8M6NcY8hd7SiT9RJ3fQ4TDGOfufn8fTXVtleSN3tOicMpbMhZ7XP/0x1bvN72yYbMPAbZxf+G6Qs6CwlG0ftU/FrD3W5e38ypg5E9Uk7GW0o2A74IOrpmUswDDeR1j+0TGIlOYcwAuQREy1aDcgvm+9Cn1KFp0Fd+q0DxnGf2V31a22DoE1Bxj+R4Jq5bLzI0ZBt/VYO65b1MnXetz3cy3T8gKTFzeflorPU5GBIwKs7AeDUgvXpefp/XfW70afBKhk1la6LstBUjVP6TPI7yKpVWq0FgR0MjeyZFp4eEezDoQJOX/siy5/weWv7sOfifpDp2lxSqPcTd+c1wa512tb6KSapu0vvb7JL9VTyNWdw6Th2ymD2RWtOV8dRXTLXkzd9Q34/8Y0OR4n/cSY=

username: AgB+7E5iWzUELYgjxwwwnAk1SlR+ZwQV7OafBsZIPA47KgSlp8Xys7slJ2STIQ4nfmg/UTJimTN8mNvF3QK6zEfAeglN75r0HxMe8aWPnJ4OrKwuy8LUpEIKtdlPHLnyHNF2v5Nos/Gej1cshuA27EliqCwRLE7Z4FCxwiUIX0zBraWOJcXXzVylRK3WA2uGrBsH9P9k7+V1h/B66K4djCL47uOLaBQTXRlW45av1cSsDWLpq1em/6Og+GA5txxsNaCDADR2ybqpPg6RRoC1ayf7z+cYrFV84Wc9KNb3+cxHjuEEW/c3PT0xqyqZq7Iqr6254SVTv1bB0joD7wK7yEPUFJSc8M3GnXsG3Eq81QgpjeKHY1uZnT4B5/ztxSeczQoKksjvICUeysb09tZMRgF80A74uAv4e0gyRXBEgk7uPQg/JTqKFo77qntK5lzAGSVJQlumAq8FBeAKwwzOJrltbC8yMprQzapXBj8Vo43gEY1Avkc2Pov/bceppuah2Q9JFPCetpEtI+lruRGlZUR9Iw3T4QcunLXfgC73xWiPY6B/I+dFh2oDoiabo4gonGDLEhtAmtQU6MARWVBbs6AFGHQYRdvU23Xc5ISocG69mwFwevtwkOHQtx+VOcMN1Tmnj5htCb1CaK+rD2xfHjaWZ3VjorWuUCG5kXz5pjeRw95s+h8ru2emRqokj1jUf6xqc03QIg==

template:

metadata:

creationTimestamp: null

name: db-secret

namespace: demo

You can now add this SealedSecret resource in Git. As you see it contains the secret, but the values are encrypted. Only the Sealed Secret operator in the cluster has the private key to decrypt the information.

Apply the SealedSecret resource

Use kubectl to apply the SealedSecret resource.

kubectl apply -f sealed-db-secret.yaml



sealedsecret.bitnami.com/db-secret created

Once you apply the SealedSecret, the Sealed Secret operator gets triggered, which is using its private key to decrypt the information and is generating the corresponding Kubernetes secret resource.

Verify deployment

Verify the deployment by checking the SealedSecret and Kubernetes secrets in the demo namespace.

kubectl get sealedsecrets,secrets -n demo



NAME AGE

sealedsecret.bitnami.com/db-secret 55s kubectl get secrets db-secret -n demo -o yaml



NAME TYPE DATA AGE

secret/db-secret Opaque 2 55s



apiVersion: v1

data:

password: WTRueXM3ZjEx

username: cHJvZHVzZXI=

kind: Secret

metadata:

creationTimestamp: "2020-07-02T09:39:15Z"

name: db-secret

namespace: demo

ownerReferences:

- apiVersion: bitnami.com/v1alpha1

controller: true

kind: SealedSecret

name: db-secret

uid: 77d1231f-921c-4335-823f-643ec82fdb78

resourceVersion: "55133137"

selfLink: /api/v1/namespaces/demo/secrets/db-secret

uid: 2df6c273-ca03-4523-b45d-f9849b39b92e

type: Opaque

Summary

Sealed Secret was designed to easily fit into automated workflows such as GitOps. Once a Kubernetes secret is converted into a SealedSecret, only the operator in the cluster can decrypt the original secret. If you are interested, you can also see how we use Terraform to create and manage a Kubernetes cluster by clicking the link. With SealedSecret, the creation and update of Kubernetes secrets does not require high privileges, which makes it easy and secure. Thanks for reading!