Kubernetes: ServiceAccounts, JWT-tokens, authentication, and RBAC authorization
By: Arseny Zinchenko, Setevoy
Tags: Authentication, Kubernetes, Security

For the authentification and authorization, Kubernetes has such notions as User Accounts and Service Accounts.

User Accounts — common user profiles used to access a cluster from the outside, while Service Accounts are used to grant access from inside of the cluster.

ServiceAccounts are intended to provide an identity for a Kubernetes Pod to be used by its container to authenticate and authorize them when performing API-requests to the Kubernetes API-server.

Content

Default ServiceAccount

Every Kubernetes Namespace has its own default ServiceAccount (SA) which is created when creating a namespace.

Let’s check the default namespace:

$ kubectl --namespace default get serviceaccount

NAME SECRETS AGE

default 1 176d

For each ServiceAccount a token is generated and stored as a Kubernetes Secret.

Check the default SA:

$ kubectl --namespace default get serviceaccount default -o yaml

apiVersion: v1

kind: ServiceAccount

metadata:

creationTimestamp: “2020–05–25T12:04:49Z”

name: default

namespace: default

resourceVersion: “296”

selfLink: /api/v1/namespaces/default/serviceaccounts/default

uid: 19cc2b5f-fbc3–403e-a7c7-d62361a4038a

secrets:

- name: default-token-292g9

Here is the token for this SA — the default-token-292g9 Secret:

…

secrets:

- name: default-token-292g9

default token

Now, check the Secret’s content:

$ kubectl get secret default-token-292g9 -o yaml

apiVersion: v1

data:

ca.crt: LS0…sdA==

token: ZXl…TWc=

kind: Secret

metadata:

annotations:

kubernetes.io/service-account.name: default

kubernetes.io/service-account.uid: 19cc2b5f-fbc3–403e-a7c7-d62361a4038a

creationTimestamp: “2020–05–25T12:04:49Z”

name: default-token-292g9

namespace: default

resourceVersion: “294”

selfLink: /api/v1/namespaces/default/secrets/default-token-292g9

uid: 07a46645–0083–45a0-a640–6e6a78ebd9b1

type: kubernetes.io/service-account-token

At first, its type is the kubernetes.io/service-account-token .

Another interesting part here is the data which keeps two records - ca.cert и token .

If a token is not from the default namespace — there will be a third field specifying a namespace to which this token belongs.

The ca.cert is signed by the cluster's master key so the cluster is playing the Certificate Authority role, and allows a pod or an application to verify the API-server.

And now, let’s go to investigate the token part.

JWT token

To make it easier to work from the terminal — save the data.token value to a variable:

$ token=”ZXl…TWc=”

Use the base64 get its content:

$ echo $token | base64 -d

eyJ[…]iJ9.eyJ[…]ifQ.g5I[…]3Mg

Here I’ve removed some data with the […], but we can see that the value is divided into three parts with dots:

the header — describes how the token was signed

the payload — actual data of the token, such as expiration date, who issued it, etc see the RFC-7519

the signature — is used to verify that the token wasn’t modified and can be used to validate the sender

See the documentation>>>.

To check the token’s content we can use the jwt utility or on the jwt.io website.

In our case, the payload section has the following lines:

{

"iss": "kubernetes/serviceaccount",

"kubernetes.io/serviceaccount/namespace": "default",

"kubernetes.io/serviceaccount/secret.name": "default-token-s8m4t",

"kubernetes.io/serviceaccount/service-account.name": "default",

"kubernetes.io/serviceaccount/service-account.uid": "b4514006-4c9a-4c30-92c8-1cc1c058b31c",

"sub": "system:serviceaccount:default:default"

}

here in the sub filed we can see the ServiceAccount name, i.e. - who is presenting this token to the Kubernetes API-server so the server will know from who this token came.

Okay, but what about a password? In the sub there is a "login" - but where is his "password"?

And here is the third part is playing — the signature.

JWT token and authentification

I wasn’t able to see these details in any from the googled materials, see the Useful links section of this post, although as for me — this is the most interesting part of the scheme.

Let’s go back to the first section of the token — the header, which in our case has the RS256 algorithm type defined i.e. RSA (Rivest-Shamir-Adleman) — the asymmetric algorithm with private and public keys and uses SHA-256 algorithm for the signature.

Let’s check our token on the jwt.io:

Invalid Signature — as we not provided the private and public keys to verify the token.

Because the masters’ private key on AWS Elastic Kubernetes Service is stored on the ConrolPlane nodes and we can’t access them — let’s use minikube for the testing.

Run a local cluster:

$ minikube start

In its default namespace we can see already existing token:

$ kubectl get secrets

NAME TYPE DATA AGE

default-token-s8m4t kubernetes.io/service-account-token 3 2m44s

Grab the token field and decode it with base64 :

$ kubectl get secrets -o jsonpath=’{.items[0].data.token}’ | base64 -d

eyJhbGciO[…]61O_LxbM_-tiLjyjeCZw

Go back to the jwt.io, paste the string received above:

Still Invalid Signature — but go to your minikube and take its public certificate - the ~/.minikube/ca.crt file:

$ cat ~/.minikube/ca.crt

— — -BEGIN CERTIFICATE — — -

MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p

…

0g+FhVM92T+yV38vYLO/HaKeiOzIcgHHkAoLJZd/K/Mu7crwIuGlcCVhrjcHoa3p

Md34ZTeqxA4J3w==

— — -END CERTIFICATE — — -

Paster it to the Public Key or Certificate field.

Find the private key of the minikube cluster - actually, it is also used to sing the ca.crt and tokens, the ~/.minikube/ca.key file:

$ cat ~/.minikube/ca.key

— — -BEGIN RSA PRIVATE KEY — — -

MIIEowIBAAKCAQEAtDRDag2D7UBaBmWQwTKVLjuKTuat4eD/oThRgfi5bcCnwooG

…

xnL96EHthflb3NaS4GKuJYzNAPhfOdMw96Ce8KtNYpMYjRhNF9TN

— — -END RSA PRIVATE KEY — — -

Paste it to the Private Key field:

Signature Verified — yup, it works! The authenticity of the bearer of the token is verified.

So, going back to the ServiceAccounts:

for a ServiceAccount a token is created which keep the SA name

the token is signed by the master key of the Kubernetes cluster

a pod make a request to the API server using this token to authenticate him

the API server validates the token by using its public key and verify that the token wasn’t modified and is relly issued by this Kubernetes clutserм

Now, let’s go to see in practice how this is working and how Kubernetes RBAC is used here.

ServiceAccounts, and RBAC

For each Pod that has no ServiceAccount specified the default ServiceAccount is attached and its default token is mounted.

Go back to our EKS cluster and run a Pod:

$ kubectl run -i --tty --rm ca-test-pod --image=radial/busyboxplus:curl

kubectl run — generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run — generator=run-pod/v1 or kubectl create instead.

If you don’t see a command prompt, try pressing enter.

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$

Check it volumeMounts , serviceAccount , and volumes :

$ kubectl get pod ca-test-pod-5c96c78d7f-wqlsq -o yaml

apiVersion: v1

kind: Pod

…

volumeMounts:

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

name: default-token-292g9

readOnly: true

…

serviceAccount: default

serviceAccountName: default

…

volumes:

- name: default-token-292g9

secret:

defaultMode: 420

secretName: default-token-292g

Inside of the pod check the /var/run/secrets/kubernetes.io/serviceaccount directory content:

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ ls -1 /var/run/secrets/kubernetes.io/serviceaccount

ca.crt

namespace

token

And recall the content of the data section of the default-token-292g9 Secret:

$ kubectl get secret default-token-292g9 -o yaml

apiVersion: v1

data:

ca.crt: LS0t[…]

namespace: ZGVmYXVsdA==

token: ZXlKaGJ

…

Try to perform a request to the API-server without authentification — use the special Service kubernetes, add the -k or --insecure to the curl to skip server's certificate validation



{

“kind”: “Status”,

“apiVersion”: “v1”,

“metadata”: {

},

“status”: “Failure”,

“message”: “forbidden: User \”system:anonymous\” cannot get path \”/\””,

“reason”: “Forbidden”,

“details”: {

},

“code”: 403

} [ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ curl -k https://kubernetes “kind”: “Status”,“apiVersion”: “v1”,“metadata”: {},“status”: “Failure”,“message”: “forbidden: User \”system:anonymous\” cannot get path \”/\””,“reason”: “Forbidden”,“details”: {},“code”: 403

Cool — we got the 403, Forbidden.

Now, add two variables — one with the ca.crt and with the token:

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ CERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

And run curl again - let's try to get a list of the pods in our namespace, this time without --insecure and with authorization by using the Authorization header:

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ curl — cacert $CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes/api/v1/namespaces/default/pods/"

{

“kind”: “Status”,

“apiVersion”: “v1”,

“metadata”: {

},

“status”: “Failure”,

“message”: “pods is forbidden: User \”system:serviceaccount:default:default\” cannot list resource \”pods\” in API group \”\” in the namespace \”default\””,

“reason”: “Forbidden”,

“details”: {

“kind”: “pods”

},

“code”: 403

}

At this time, we are able to see our user — the User "system:serviceaccount:default:default" , but it has no permissions to perform requests as by default all users and ServiceAccounts have no privileges (the principle of the least privileges, POLP).

RoleBindig for ServiceAccount

To give our SericeAccount permissions we need to create a RoleBinding or ClusterRoleBinding as for normal users.

Create a RoleBinding mapping to the default ClusterRole view, see User-facing roles:

$ kubectl create rolebinding ca-test-view --clusterrole=view --serviceaccount=default:default

rolebinding.rbac.authorization.k8s.io/ca-test-view created

And run curl again:

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ curl — cacert $CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes/api/v1/namespaces/default/pods/"

{

“kind”: “PodList”,

“apiVersion”: “v1”,

“metadata”: {

“selfLink”: “/api/v1/namespaces/default/pods/”,

“resourceVersion”: “66892356”

},

“items”: [

{

“metadata”: {

“name”: “ca-test-pod-5c96c78d7f-wqlsq”,

“generateName”: “ca-test-pod-5c96c78d7f-”,

“namespace”: “default”,

“selfLink”: “/api/v1/namespaces/default/pods/ca-test-pod-5c96c78d7f-wqlsq”,

“uid”: “f0d77cfe-38ab-48e9-aaf3-f344f1d343f3”,

“resourceVersion”: “66888089”,

“creationTimestamp”: “2020–11–17T16:08:09Z”,

“labels”: {

“pod-template-hash”: “5c96c78d7f”,

“run”: “ca-test-pod”

},

…

“qosClass”: “BestEffort”

}

}

]

ServiceAccounts and security

Remember, that having access to Secrets and ServiceAccounts any pod can have any token attached and thus can be able to perform actions allowed by such a token.

For example, by using the ServiceAccount of the ExternalDNS — such a pod can make a mess in our AWS Route53.

That’s why it is important to divide access to resources by using RBAC rules and roles for users, for example by allowing access to resources from only one namespace.

Useful links