Securing Kubernetes secrets : How to efficiently secure access to etcd and protect your secrets
By: Anton Mishel
Tags: Etcd, Security, Kubernetes, DevOps

Etcd is a distributed, consistent and highly-available key value store used as the Kubernetes backing store for all cluster data, making it a core component of every K8s deployment.

Due to its central role etcd may contain sensitive information related to access of the deployed services and their associated components, such as database credentials, CA keys, LDAP logins credentials it is a premium target for malicious attacks.

Historically, in traditional, non-containerised environments, this data was NOT stored in such a centralised manner as credentials were usually under an ownership of a specific team that was responsible for maintaining a certain component of the stack: the DB access credentials, for example, were known only to the DBA team, CA keys have been in the hands of few selected System Administrators etc.

With K8s, the required approach is notably different as credentials are now kept within a single central place (etcd), which, if not properly hardened, can lead to serious security breaches as the attacker may now create fake certificates, access databases and applications.

Managing and hardening your secrets becomes even more critical with tools such as Helm and Tiller; these tools allow you to install (or redeploy) an entire K8s based datacenter within minutes and they constantly interact with etcd.

The Center for Internet Security (CIS) came up with this publicly available document providing guidance on how to properly harden and secure your Kubernetes cluster.

The only single recommendation CIS provides regarding hardening etcd is using TLS:

ETCD is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. Its access should be restricted to specifically designated clients and peers only. Authentication to ETCD is based on whether the certificate presented was issued by a trusted certificate authority. There is no checking of certificate attributes such as common name or subject alternative name. As such, if any attackers were able to gain access to any certificate issued by the trusted certificate authority, they would be able to gain full access to the ETCD database. Use a different certificate authority for ETCD from the one used for Kubernetes.

However, using TLS on its own is not sufficient as a solution. Every certificate created and signed with the same CA has the potential to access every service inside the cluster. The problem is further exacerbated if a single CA is used for all k8s clusters. Even when each kubernetes cluster has a dedicated CA, new client keys can be easily created but as easily revoked. Once again, any new keys created automatically have access to every service in the targeted k8s cluster.

Because of the severity of the security risks associated with etcd, we will look into 2 additional methods that can be implemented to further secure your etcd data:

Encrypting secrets (and/or other resources) in etcd

Using certificates to stop clients from accessing the etcd server

To follow the steps illustrated in the following sections, it is necessary to start up a Kubernetes cluster. This can be done using any of the methods immediately below.

Vanilla K8s:

install script for latest kubernetes 1.10. This is the first version that installs etcd with tls

install script for older kubernetes versions, when etcd was not installed with tls by default

This also works on openshift platform. You can see the install script here. All relative commands for openshift are in this script.

The install scripts have been tested on AWS Centos ami. It should work for you too if you use the same image.

Starting with K8s 1.7 (and etcd v3) you can encrypt resources inside etcd using several different algorithms. At the very least, you should encrypt all your secrets. It is especially true if you are using Helm as a lot of Helm charts require LDAP or DB credentials to be directly made available in the ConfigMaps.

The encryption follows a very simple rule:

encrypt using the first provider defined

decrypt after locating a functional provider at checking each provider in the order the providers are defined

To implement the full workflow, it is necessary to add the experimental-encryption-provider-config flag to the apiserver

Define the EncryptionConfig config file (place the content in /etc/kubernetes/pki/encryption-config.yaml)

kind: EncryptionConfig

apiVersion: v1

resources:

- resources:

- secrets

providers:

- identity: {}

Within the file, the resources.resources field is an array of Kubernetes resource names that should be encrypted. The providers array is an ordered list of the possible encryption providers.

Enable experimental-encryption-provider-config in the kube-apiserver. Edit /etc/kubernetes/manifests/kube-apiserver.yaml and add:

spec:

containers:

- command:

- kube-apiserver

- --experimental-encryption-provider-config=/etc/kubernetes/pki/encryption-config.yaml

Restart the apiserver. Because the API server is being run as a static pod, kubelet will restart it when the configuration change is detected. Otherwise, you will need to restart the service yourself.

We also install the etcd package in order to print the data from inside the etcd server:

# yum install etcd -y

Resolving Dependencies

--> Running transaction check

---> Package etcd.x86_64 0:3.2.18-1.el7 will be installed

--> Finished Dependency Resolution Dependencies Resolved ==================================================================================================================================================================================================================================

Package Arch Version Repository Size

==================================================================================================================================================================================================================================

Installing:

etcd x86_64 3.2.18-1.el7 optymyze_external_rpms 9.3 M Transaction Summary

==================================================================================================================================================================================================================================

Install 1 Package Total download size: 9.3 M

Installed size: 42 M

Downloading packages:

etcd-3.2.18-1.el7.x86_64.rpm | 9.3 MB 00:00:00

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : etcd-3.2.18-1.el7.x86_64 1/1

Uploading Package Profile

Verifying : etcd-3.2.18-1.el7.x86_64 1/1 Installed:

etcd.x86_64 0:3.2.18-1.el7 Complete!

Uploading Enabled Repositories Report

Loaded plugins: fastestmirror, priorities, product-id

Also, to make the commands shorter, set an alias for etcdctl command with TLS parameters. Here we will use the certificates paths created by kubeadm-1.10. You should update them for your specific cluster if needed (check by running grep -- '--etcd' /etc/kubernetes/manifests/kube-apiserver.yaml ).

Meaning of variables:

DIR — path where the k8s certificates are created

SSL_OPS — etcdctl parameters to enable TLS connectivity

SECRETS_PATH — path in etcd where kubernetes keeps secrets

DIR=/etc/kubernetes/pki/ SSL_OPTS="--cacert=${DIR}/etcd/ca.crt --cert=${DIR}/apiserver-etcd-client.crt --key=${DIR}/apiserver-etcd-client.key --endpoints=localhost:2379"

SECRETS_PATH=/registry/secrets

Test that we can list stuff in etcd:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only=true --prefix $SECRETS_PATH

/registry/secrets/default/default-token-rhwwn /registry/secrets/kube-public/default-token-9qfc8 /registry/secrets/kube-system/attachdetach-controller-token-clvsn .............

No Encryption

To demonstrate the difference of our solution, we begin with no encryption. This provider doesn’t do any encryption. It can be used in case you want to decrypt everything or just to test.

Let’s create a secret and read it directly from etcd. You should be able to clearly see the key name and key value:

# kubectl create secret generic secret1 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ

secret "secret1" created

# kubectl get secret secret1 -o yaml

apiVersion: v1

data:

XX_mykey_XX: WlpfbXlkYXRhX1pa

kind: Secret

metadata:

creationTimestamp: 2018-06-18T13:11:54Z

name: secret1

namespace: default

resourceVersion: "20410585"

selfLink: /api/v1/namespaces/default/secrets/secret1

uid: 2bb3b7df-72f9-11e8-ad5f-005056b1028d

type: Opaque

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value

"Value" : "k8s\x00

\f

\x02v1\x12\x06Secret\x12s

L

\asecret1\x12\x00\x1a\adefault\"\x00*$2bb3b7df-72f9-11e8-ad5f-005056b1028d2\x008\x00B\b\b\x9aߞ\xd9\x05\x10\x00z\x00\x12\x1b

\vXX_mykey_XX\x12\fZZ_mydata_ZZ\x1a\x06Opaque\x1a\x00\"\x00"

Apply an Encryption Algorithm

Let’s add an encryption algorithm to see what happens. We choose aescbcbecause this is the recommended choice for encryption at rest.

Update encryption-config.yaml:

kind: EncryptionConfig

apiVersion: v1

resources:

- resources:

- secrets

providers:

- aescbc:

keys:

- name: key1

secret: c2VjcmV0IGlzIHNlY3VyZQ==

- name: key2

secret: dGhpcyBpcyBwYXNzd29yZA==

- identity: {}

Since kubelet only monitors pods defined in /etc/kubernetes/manifests, this change will not be caught, so we need to restart the apiserver manually:

docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')

Test:

# kubectl create secret generic secret2 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ

secret "secret2" created

# kubectl get secret secret2 -o yaml

apiVersion: v1

data:

XX_mykey_XX: WlpfbXlkYXRhX1pa

kind: Secret

metadata:

creationTimestamp: 2018-06-18T14:23:06Z

name: secret2

namespace: default

resourceVersion: "20418382"

selfLink: /api/v1/namespaces/default/secrets/secret2

uid: 1e4f5d2f-7303-11e8-8c2c-005056b1028d

type: Opaque

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value

"Value" : "k8s:enc:aescbc:v1:key1:7^İ\xe9\xc8\x1e\xa7̔=D+\x9e%\x1a\xf4\x10o@\xec\xc14&<Z\xd1\xde\xfa\xca-'#\xa2K\x1c\xff\x101a\x86\xb0\xd7.\xa9\x19\x04\x93m\xa1\xee\xacDe\x95/\xd8\xe7\xaehp~\xc9\x0e\xe9\x8f}\x9a\x8a\xb0f\xf9\xeb\xb7\u007f@\x87\xa0\xa6\x98\xe78\xd0+\xd45\"S\x17\x8c\x84\xa6ㅽb\xda\xe6\xfc\xa1\xd9[[~\x82\xfbKS\x82\xf0>o\xc1 \x8b&{\xa1\r\x14Un\x03\xf7\x1f=\xe5\x1b \xa7t\xed[\x8a\xec\xb8\xf1\xe4\xe2\xc1\x81\xb00=cbl·ɬ\x12`\xf2|\x1b\t\xe4#\xcd"

The new secret was encrypted now with “k8s:enc:aescbc:v1:key1”.

Let’s encrypt all the other secrets as well:

# kubectl get secrets --all-namespaces -o json | kubectl replace -f -

secret "default-token-rhwwn" replaced

secret "secret1" replaced

secret "secret2" replaced

secret "default-token-9qfc8" replaced

secret "attachdetach-controller-token-clvsn" replaced

secret "bootstrap-signer-token-xgnfg" replaced

....

Check that the old secret is now encrypted:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value

"Value" : "k8s:enc:aescbc:v1:key1:\xda

W0~\x83\xe4\x80Ճ$J\x1e\xa2\x02z\xc9\v\xd1\xd0$)\xb2K\x9f\xc2\xff\xcdJ5\xfa\"\x13\xc4\f\x86\xc0{P\xceW\x9e\xd1z;b$\x97\xe8\xb4l\xd0\xfa\xd8 \xe2Vc\x8c\xa2\xcd\xe5\xb0\x04(l\x18\x13\xbf\xe2\xb7|\xf1m\xef)\xfd\x97\xcbk-\"\xba\x819\xcf,_\xf6\fxP\xf2\x13\x94\x9b\xca\xf4\xde{d\xcb\xceq\x84q\xae\xaa\x06\x14\xb7q\x1d|L\x8eS\x8c\xc9$\x8e\x80D\xf0\xda\xe2si\xb6,@\xa2\xf9\xae\xf2~\xe3w\x8e4fr{e\x0f'\xcc\xf6\xe7\xadd\x83^\xdb\x03\xf1jT\x13>"

Each resource is encrypted with a specific key. If you change the value of that key, kubernetes will not be able to decode it anymore. You can test this by swapping the name of the keys and try to retrieve it:

# cat /etc/kubernetes/pki/encryption-config.yaml

kind: EncryptionConfig

apiVersion: v1

resources:

- resources:

- secrets

providers:

- aescbc:

keys:

- name: key2

secret: c2VjcmV0IGlzIHNlY3VyZQ==

- name: key1

secret: dGhpcyBpcyBwYXNzd29yZA==

- identity: {}

# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')

000e03b50c0f

# kubectl get secret secret2 -o yaml

Error from server (InternalError): Internal error occurred: invalid PKCS7 data (empty or not padded)

Using multiple algorithms

In this example we encrypt a secret with a new algorithm and check that different secrets are encrypted with different providers. After that encrypt everything with the new provider. This will change the encryption algorithm to all previous keys to the new one.

Change encryption-config.yaml, so that the first provider to be the secretbox provider:

kind: EncryptionConfig

apiVersion: v1

resources:

- resources:

- secrets

providers:

- secretbox:

keys:

- name: key1

secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=

- aescbc:

keys:

- name: key1

secret: c2VjcmV0IGlzIHNlY3VyZQ==

- name: key2

secret: dGhpcyBpcyBwYXNzd29yZA==

- identity: {}

Restart the apiserver:

# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')

Verify that secrets are encrypted correctly: old secret is using aescbc, new one will use secretbox.

# kubectl create secret generic secret3 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ

secret "secret3" created

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value

"Value" : "k8s:enc:aescbc:v1:key1:\xda

W0~\x83\xe4\x80Ճ$J\x1e\xa2\x02z\xc9\v\xd1\xd0$)\xb2K\x9f\xc2\xff\xcdJ5\xfa\"\x13\xc4\f\x86\xc0{P\xceW\x9e\xd1z;b$\x97\xe8\xb4l\xd0\xfa\xd8 \xe2Vc\x8c\xa2\xcd\xe5\xb0\x04(l\x18\x13\xbf\xe2\xb7|\xf1m\xef)\xfd\x97\xcbk-\"\xba\x819\xcf,_\xf6\fxP\xf2\x13\x94\x9b\xca\xf4\xde{d\xcb\xceq\x84q\xae\xaa\x06\x14\xb7q\x1d|L\x8eS\x8c\xc9$\x8e\x80D\xf0\xda\xe2si\xb6,@\xa2\xf9\xae\xf2~\xe3w\x8e4fr{e\x0f'\xcc\xf6\xe7\xadd\x83^\xdb\x03\xf1jT\x13>"

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value

"Value" : "k8s:enc:secretbox:v1:key1:\xba\xf8,Q@\xb9\xb6q3$k\x04\xeeV\x99|Z'\xdeE<\xa5\xa9n\x91u\xb9]RY\xccc\xe3\x13\x8b\u07b4Q\x91\x9cR2\xcc\xc5\xd9\x0e\x19?\xca\x1ch\xde\x1d%\xa3N\x85H\xb0\xf6֢\xe6\xab\x06\xf6\x960{\xdb\xd8^eQ\xb3\x05\x03\x06)\x05JH\x16\x18\fp\x9eu<t\xea\x06\x12\xf1۹y\u007f\x15\xe5\x1d\xef\x8a2G\x85'\x94

\x1d\x99\x85ku3\xa2~\x12\x04\xe5\x84~\xaaG\xd3n\x98\x95\xa0\xc8_1B\xcb\x0f\xb7;\x80\xe1xR\x86ĳ\f\xef\xd7SA\x950MQfz~)\x13\xc5\xf1\xf8\x91\x14\x9d_\xba\x82[=M\x81O\x1dFNj\xc1\x98\xe4"

Migrate all secrets to the new provider:

# kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Key rotation

Here we use the same provider for encryption, but we add a new key. Everything from this moment will be encrypted with the new key. Old values are encrypted with the previous key. At the end we migrate everything to be encrypted with the new key.

Add a new key to be the first for the secretbox provider (which is still the first provider)

kind: EncryptionConfig

apiVersion: v1

resources:

- resources:

- secrets

providers:

- secretbox:

keys:

- name: key2

secret: sAkccgM28JdPNCX9FfTcloYet1zp4OEAtHyViT038zM=

- name: key1

secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=

- aescbc:

keys:

- name: key1

secret: c2VjcmV0IGlzIHNlY3VyZQ==

- name: key2

secret: dGhpcyBpcyBwYXNzd29yZA==

- identity: {}

Restart and verify that the new values are encrypted with the new key:

# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')

4bdac1937570

# kubectl create secret generic secret4 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ

secret "secret4" created

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value

"Value" : "k8s:enc:secretbox:v1:key2:\x92\xeeyj\x96\xfc쵪-8\x0e\xa7\x9a\xb0\x16\xe2\xb8J\f_\x81\xec\xf65\xa9\x1a\xe5\\xۛ%Ҝ\xbb\ax\xbf\x00Kz\xabaD\x1c\x94\x87\xaervsP\xf3q\xf3\xaeH\xb8\x95-\xef\r*[yl\xf3/\xc4\x0f\x00\a\x132\f\xe1\x17\xbf\xff\xb4;<\xec\xc2\x01\xa8\xc8f\xff\xcd\xf3ʦ\x83P\x01\xcdu\x16\x16\xfa\xba\x8f\xe6\xe5\x05\x96\xf7k,\xaa\xea\x0f\x99\x8f\xb3\xc7\xe6\xa4=\x93\x8a\xf3S\x17\xc6S\r\xee\xea㟷\x00\x945o\xe8\x8e:W\xacot\xeaj,P\x14\xbe\xd0\x13\xf91Y\xf0\xf0\x93fW\xcczD3\xb9\xa0\xb4\x9e\xef\x1aE\x16\xc8j_TX\xae"

ETCD authorization

Etcd can use 2 methods to authorize users:

With username and password

With certificates if started with ” --client-cert-auth=true ”. It will use the CN from the certificate as the username.

Unfortunately there are a couple of issues with this:

https://github.com/coreos/etcd/issues/9816: auth doesn’t work at all with the default etcd version. You need to update your etcd to version 3.2.18

https://github.com/coreos/etcd/issues/9691: this affects you if you try to create the openshift user inside etcd. But for this one there is also an workaround

We use cfssl to create the certificates we need to connect to etcd. Lets download the binaries and put them in our path:

# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O && /bin/mv ./cfssl_linux-amd64 /bin/cfssl && chmod +x /bin/cfssl

# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O && /bin/mv ./cfssljson_linux-amd64 /bin/cfssljson && chmod +x /bin/cfssljson

In order to enable etcd ACL, first create the root user:

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add root:secretpass

User root created

Create some roles to test our certificates.

Users with this role should have read access in the entire cluster:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_all

Role readonly_all created

# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_all --prefix=true read /

Role readonly_all updated

Allows users to read and write everywhere:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_all

Role readwrite_all created

# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_all --prefix=true readwrite /

Role readwrite_all updated

User with this role should only be allowed to access part of etcd tree:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_secrets

Role readonly_secrets created

# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_secrets --prefix=true read $SECRETS_PATH

Role readonly_secrets updated

Role that allows a user to read/write a specific key only:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_secret4

Role readwrite_secret4 created

# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_secret4 readwrite $SECRETS_PATH/default/secret4

Role readwrite_secret4 updated

Create users and add assign specific roles to them. Generate random passwords because we don’t expect to use them:

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add reader:$(head -c 32 /dev/urandom | base64)

User reader created

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add viewsecrets:$(head -c 32 /dev/urandom | base64)

User viewsecrets created

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add admin:$(head -c 32 /dev/urandom | base64)

User admin created

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add usersecret4:$(head -c 32 /dev/urandom | base64)

User usersecret4 created # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role reader readonly_all

Role readonly_all is granted to user reader

# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role viewsecrets readonly_secrets

Role readonly_secrets is granted to user viewsecrets

# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role admin readwrite_all

Role readwrite_all is granted to user admin

# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role usersecret4 readwrite_secret4

Role readwrite_secret4 is granted to user usersecret4

Since we are enabling user authorization, we need to have special permissions for the user used by the apiserver to connect to the etcd cluster: we will give it the root role. Kubernetes installation has 2 connections to the etcd server: apiserver and the livenessProbe.

openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver-etcd-client.crt | grep "Subject:" openssl x509 -noout -text -in /etc/kubernetes/pki/etcd/healthcheck-client.crt | grep "Subject:"

Create an user with the name from the CN field of the certificate:

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-apiserver-etcd-client:$(head -c 32 /dev/urandom | base64)

User kube-apiserver-etcd-client created

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-etcd-healthcheck-client:$(head -c 32 /dev/urandom | base64)

User kube-etcd-healthcheck-client created # ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-apiserver-etcd-client root

Role root is granted to user kube-apiserver-etcd-client

# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-etcd-healthcheck-client root

Role root is granted to user kube-etcd-healthcheck-client

Enable authentication:

# ETCDCTL_API=3 etcdctl $SSL_OPTS auth enable

Authentication Enabled

From this moment, nothing can connect to the etcd cluster without proper certificates. Let’s create a certificate with a user that it’s not define in etcd and check that it doesn’t have access at all:

# function create_certificates {

NAME=$1



cat <<EOF | cfssl gencert -config=ca-config.json -profile=client -ca $CA_PATH/ca.crt -ca-key $CA_PATH/ca.key - | cfssljson -bare $NAME

{"CN": "$NAME","key": {"algo": "rsa","size": 2048}}

EOF SSL_OPTS="--cacert=$CA_PATH/ca.crt --cert=$PWD/$NAME.pem --key=$PWD/$NAME-key.pem --endpoints=$HOSTNAME:2379"

} # CA_PATH=/etc/kubernetes/pki/etcd

# cfssl print-defaults config > ca-config.json

# create_certificates tester

2018/06/18 18:52:54 [INFO] generate received request

2018/06/18 18:52:54 [INFO] received CSR

2018/06/18 18:52:54 [INFO] generating key: rsa-2048

2018/06/18 18:52:54 [INFO] encoded CSR

2018/06/18 18:52:54 [INFO] signed certificate with serial number 722235405009026418318053946143102861163105227800

2018/06/18 18:52:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (

specifically, section 10.2.3 ("Information Requirements"). 2018/06/18 18:52:54 [INFO] generate received request2018/06/18 18:52:54 [INFO] received CSR2018/06/18 18:52:54 [INFO] generating key: rsa-20482018/06/18 18:52:54 [INFO] encoded CSR2018/06/18 18:52:54 [INFO] signed certificate with serial number 7222354050090264183180539461431028611631052278002018/06/18 18:52:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable forwebsites. For more information see the Baseline Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum ( https://cabforum.org );specifically, section 10.2.3 ("Information Requirements"). # ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true

Error: etcdserver: permission denied

Allow an admin user to access the cluster for 2 hours only:

# cfssl print-defaults config | sed s/8760/2/ > ca-config.json

# create_certificates admin

Admin user can do list:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true

/registry/apiregistration.k8s.io/apiservices/v1. /registry/apiregistration.k8s.io/apiservices/v1.apps /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io

Get a key:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value

"Value" : "k8s:enc:aescbc:v1:key1:\xda

W0~\x83\xe4\x80Ճ$J\x1e\xa2\x02z\xc9\v\xd1\xd0$)\xb2K\x9f\xc2\xff\xcdJ5\xfa\"\x13\xc4\f\x86\xc0{P\xceW\x9e\xd1z;b$\x97\xe8\xb4l\xd0\xfa\xd8 \xe2Vc\x8c\xa2\xcd\xe5\xb0\x04(l\x18\x13\xbf\xe2\xb7|\xf1m\xef)\xfd\x97\xcbk-\"\xba\x819\xcf,_\xf6\fxP\xf2\x13\x94\x9b\xca\xf4\xde{d\xcb\xceq\x84q\xae\xaa\x06\x14\xb7q\x1d|L\x8eS\x8c\xc9$\x8e\x80D\xf0\xda\xe2si\xb6,@\xa2\xf9\xae\xf2~\xe3w\x8e4fr{e\x0f'\xcc\xf6\xe7\xadd\x83^\xdb\x03\xf1jT\x13>"

Delete the key:

# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret1

1

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/secret1 -w fields | grep Value

Increase the date and try to list:

# date $(date +%m%d%H%M%Y.%S -d '+1 hour')

# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /

/registry/apiregistration.k8s.io/apiservices/v1. /registry/apiregistration.k8s.io/apiservices/v1.apps /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io /registry/apiregistration.k8s.io/apiservices/v1.autoscaling /registry/apiregistration.k8s.io/apiservices/v1.batch # date $(date +%m%d%H%M%Y.%S -d '+1 hour')

# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /

Error: context deadline exceeded

# date $(date +%m%d%H%M%Y.%S -d '-2 hour')

Check that user ‘reader’ has access everywhere and can’t delete anything:

# cfssl print-defaults config > ca-config.json

# create_certificates reader # ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value

"Value" : "k8s:enc:aescbc:v1:key1:7^İ\xe9\xc8\x1e\xa7̔=D+\x9e%\x1a\xf4\x10o@\xec\xc14&<Z\xd1\xde\xfa\xca-'#\xa2K\x1c\xff\x101a\x86\xb0\xd7.\xa9\x19\x04\x93m\xa1\xee\xacDe\x95/\xd8\xe7\xaehp~\xc9\x0e\xe9\x8f}\x9a\x8a\xb0f\xf9\xeb\xb7\u007f@\x87\xa0\xa6\x98\xe78\xd0+\xd45\"S\x17\x8c\x84\xa6ㅽb\xda\xe6\xfc\xa1\xd9[[~\x82\xfbKS\x82\xf0>o\xc1 \x8b&{\xa1\r\x14Un\x03\xf7\x1f=\xe5\x1b \xa7t\xed[\x8a\xec\xb8\xf1\xe4\xe2\xc1\x81\xb00=cbl·ɬ\x12`\xf2|\x1b\t\xe4#\xcd" # ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret2

Error: etcdserver: permission denied

Check that user ‘viewsecrets’ has access only to read secrets:

# create_certificates viewsecrets

# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /

Error: etcdserver: permission denied

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value

"Value" : "k8s:enc:secretbox:v1:key1:\xba\xf8,Q@\xb9\xb6q3$k\x04\xeeV\x99|Z'\xdeE<\xa5\xa9n\x91u\xb9]RY\xccc\xe3\x13\x8b\u07b4Q\x91\x9cR2\xcc\xc5\xd9\x0e\x19?\xca\x1ch\xde\x1d%\xa3N\x85H\xb0\xf6֢\xe6\xab\x06\xf6\x960{\xdb\xd8^eQ\xb3\x05\x03\x06)\x05JH\x16\x18\fp\x9eu<t\xea\x06\x12\xf1۹y\u007f\x15\xe5\x1d\xef\x8a2G\x85'\x94

\x1d\x99\x85ku3\xa2~\x12\x04\xe5\x84~\xaaG\xd3n\x98\x95\xa0\xc8_1B\xcb\x0f\xb7;\x80\xe1xR\x86ĳ\f\xef\xd7SA\x950MQfz~)\x13\xc5\xf1\xf8\x91\x14\x9d_\xba\x82[=M\x81O\x1dFNj\xc1\x98\xe4"

# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret3

Error: etcdserver: permission denied

Check that ‘usersecret4’ can access only a specific key:

# create_certificates usersecret4

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value

Error: etcdserver: permission denied

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value

"Value" : "k8s:enc:secretbox:v1:key2:\x92\xeeyj\x96\xfc쵪-8\x0e\xa7\x9a\xb0\x16\xe2\xb8J\f_\x81\xec\xf65\xa9\x1a\xe5\\xۛ%Ҝ\xbb\ax\xbf\x00Kz\xabaD\x1c\x94\x87\xaervsP\xf3q\xf3\xaeH\xb8\x95-\xef\r*[yl\xf3/\xc4\x0f\x00\a\x132\f\xe1\x17\xbf\xff\xb4;<\xec\xc2\x01\xa8\xc8f\xff\xcd\xf3ʦ\x83P\x01\xcdu\x16\x16\xfa\xba\x8f\xe6\xe5\x05\x96\xf7k,\xaa\xea\x0f\x99\x8f\xb3\xc7\xe6\xa4=\x93\x8a\xf3S\x17\xc6S\r\xee\xea㟷\x00\x945o\xe8\x8e:W\xacot\xeaj,P\x14\xbe\xd0\x13\xf91Y\xf0\xf0\x93fW\xcczD3\xb9\xa0\xb4\x9e\xef\x1aE\x16\xc8j_TX\xae"

$ ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret4

1

Conclusion

In a cluster with multiple masters, where etcd servers listen on all interfaces and not on localhost, limiting the access to etcd is vital.

Kubernetes manages this with RBAC, but by default etcd is only protected by requiring the client to have a valid certificate.

While employing EncryptionConfig can take care of most of the issues, it is still possible to have data in etcd that is not fully encrypted. Since confidential data can now be required in configmaps, even in statefulsets and deployments as environment variables, using simply EncryptionConfig is not sufficient.

If you cut access entirely to etcd by using authentication and only allow the apiserver to connect, you protect yourself from leaking sensitive data to others. You can create short lived certificates for any other uses and no longer need to worry about the longevity of the certificates.

The solution is also applicable to the users added previously to etcd: if you don’t feel confident about what happened to certificates created for them before, you can revoke the users access, or delete them.