How to hack you’re way back in Kubernetes
Tags: Help, Failure, Recovery, Security, Kubernetes

How to hack you’re way back in Kubernetes NEwa Follow Mar 11 · 7 min read

Have you try turning it off and on again ?

Kubernetes is great, Kubernetes is powerful, Kubernetes is the best way to get your containerized workload scheduled and frankly I really love it, but imagine to get this magnificent tool and not being able to do anything with it because you lost the key that get you in like the newly bought house of your dream, painful isn’t it ? Enter my scary world and learn how I manage to get in by the basement.

Right, here’s the context, I’m an infra guy with nearly 10 years of experience in network and system, this past 3 years I’ve made some move to cloud native technology and had the opportunity to get hands on some project with Kubernetes, recently I needed to make RBAC rules to limit velero’s service account to backup system namespaces.

In order to get you all at the same level, a service account is an account that will allow an application deployed inside your Kubernetes cluster to communicate with the Kubernetes api.

In my case Velero’s service account is using cluster-admin role which if your not so familiar with Kubernetes is a role with privileges to do anything on your cluster.

You remember this song from the Beastie Boys, yes this one with the tum tum tum at the beginning (https://www.youtube.com/watch?v=z5rRZdiu1UE), well after installing and removing the cluster role binding from the service account, I could hear the last sentence of the song infinitely. I did it Like every one last of us at least once in their IT life, I delete the cluster-admin role.

At first I didn’t realized it, when I did try to create a pod and get an error message I thought I was using a viewer user, but when I did changed back to an admin user and get the same error message when trying to list the pod running, let’s say I cried a little.

I have come here to chew bubblegum and kick some Kubernetes clusters

I have two options now, recreate the cluster from scratch and maybe lose some work that I know was not backed up and be ashamed to tell my colleague it’s my fault, or try to get my way back in it.

Honestly I’m a lazy guy, the kind that will eat a 4 days old bread not to go to the bakery, I may then consider a lot the option to simply recreate the cluster. But I did not know what was running on this cluster, and moreover our trainee did work on it and maybe it was an evaluation test for him, I don’t want to be responsible for his failure here.

OK then, Google is my friend, most of the time, so let’s try to get some Stack Overflow good fella who did get this kind of use case. I can’t directly ask the question on the net and wait since my colleagues could need to get access to this cluster, let’s hope someone has already did what did it on purpose or by accident like me.

Let’s say that trying to get some information about cluster role from Kubernetes is pretty easy, but for an obscure reason people who delete their cluster-admin role are not so numerous, maybe I am only human after all and failure is always an option, maybe I should reconsider all my choice in life.

Hell no, I worked with Kubernetes for at least 3 years now and know how gears interacts under the hood,so why not recover it by myself.

First thing first, Kubernetes’s resources are stored in ETCD, so lets check if we can get anything from it.

In my misery I have some luck, this cluster is deployed through PKS on an on premise VMware platform, accessing ETCD from a service provider managed Kubernetes cluster would certainly be impossible.

Anyway, I connect myself on the master node, and check if there is an etcdctl binary on it, and there is one, alleluia ! Now I need to find the ETCD certificates, a simple ps command will get me the info:

Now that I now where to find the ETCD certificate I could try to get the Kubernetes resources list, what I want to know is, am I able to recreate the role from the ETCD directly. From my previous experience with ETCD I did use the version 2 of the API, but the version 3 as way more options and since I need to redo the CKA this year let’s try it and learn to use it, one stone two birds if you know what I mean.

From what I read from the help and some example on the net, to get the list of resources from ETCD is pretty simple:

Whoohoo we got something ! Now let’s be more precise and select a role, at this point all I want is to know if I can get the full content of a role as an example to recreate the one I deleted:

And once again I got what I wanted.

We have the capability to make the worlds first bionic role

Now the fun part, try to recreate the cluster-admin role via etcdtcl.

I have two options before me, create the cluster-admin role from what I got from the admin role or connect myself on another cluster and get the ETCD key/value from it.

You know what I did, yes you know, I told you that I’m lazy, no need to scratch my head over that.

Looking at the JSON structure from the role it appears that there is an information about an UID , this could be a problem, but it’s been nearly two hours since I break the access to the cluster, times ticking.

Ok I’m a little bit stressed out now, but reading Kubernetes documentation about UID, they are system generated and unique to track object through cluster lifetime.

Since my old cluster-admin role is not there anymore, it could be harmless to push a new one without the same UID.

It’s time now to test if the creation of the resource directly in the ETCD is working by simply put the key and value with this command:

Command return said OK, logs seems fine, every block of the Kubernetes cluster runs OK or at least it looks like it, with my change I reboot the node, you know the drill : when in doubt, reboot.

The node is rebooted so lets try to connect back to my cluster and check if everything is OK.

And … it failed.

Houston, we have a problem

But the worst part is that now I can’t even communicate with the API server from the Kubernetes with kubectl.

I was so sure of myself I didn’t create a backup of the ETCD before doing that, I am feeling the heat going on, flames surrounding me, I drink my coffee straight and get back to work.

Back to the master node to check the log from the API server, and the problem was quick to find in the log.

The value for the cluster-admin resource is not right and I know why, i did forget that my double quotes from my echo command won’t be kept when send to the etcdctl, getting back the put command and putting simple quote at start and end of the echo content would do the trick:

Now the API server is running fine, no more error logs.

I just need to check if everything is OK with a simple kubectl command:

Eye of the tiger baby

Hell yeah ! I’ve did it, it was not without some pain but the result is here, the cluster is safe and sound, and I know now that I could recover this kind of disaster without too much problem.

Well OK nothing to be pretty proud about, and if backup where done consistently, I could have sold the problem in five minutes, but you know small victories are victories after all.

As I said it before, I could do this trick because I was able to access the master nodes, against a managed Kubernetes it would be more catastrophic.

I hope this trick could help you when needed, share this blog if you found it useful or just if you liked the side story.