Capturing container traffic on Kubernetes
By: Peter Gillich
Tags: Kubernetes, Tcpdump, Network, Linux, Troubleshooting

It’s easy to capture network traffic with a capture tool (for example: tcpdump) if we have access to the network interface. But it’s tricky in Kubernetes. There are several options, for example: sidecar container, capture plugin, docker container, direct access in same network namespace.

Animated visualization showing pod to service connectivity. source: https://techcommunity.microsoft.com/t5/networking-blog/troubleshooting-kubernetes-networking-on-windows-part-1/ba-p/508648

Managed Kubernetes providers hardened security configurations, so depending on our access possibilities and rights to the cluster, the below examples may work (if it’s possible to access the node OS, if container is enabled to run the container as root, etc.). Below examples are tested on a VM-based K8s cluster created for my earlier article Setup On-premise Kubernetes with Kubeadm, MetalLB, Traefik, and Vagrant.

Tools

In most cases, the container, which the network traffic would be captured, does not contain any capture tool, of course. A part of the alternatives uses a new container, which contains network capture tools. My favorite Docker image is https://github.com/nicolaka/netshoot, which contains several networking tools, for example: tcpdump, tshark, termshark, iptraf-ng. The parameters will be executed as a command, for example:

sudo docker run -it --rm --net container:k8s_nginx_my-nginx-b7d7bc74d-zxx28_default_ae4ee834-fb5d-4ec4-86b1-7834e538c666_0 nicolaka/netshoot tcpdump -i eth0 -s 0 -Xvv tcp port 80

Tcpdump can redirect captured packets to stdout, which can be saved or used on the host, for example:

tcpdump -i eth0 -s 0 -w - >/tmp/container.cap tcpdump -i eth0 -s 0 -w - | wireshark -k -i -

The -v /tmp:/tmp/tmp can be useful, if captured packets are written into a file, inside of the container.

There are a few capture tools, which needs more rights (tshark, termshark), for example in docker run CLI : --cap-add=NET_ADMIN --cap-add=CAP_NET_RAW . Finally, a whole example with having a shell to run this kind of tools:

sudo docker run -it --rm -v /tmp:/tmp/tmp --cap-add=NET_ADMIN --cap-add=CAP_NET_RAW --net container:k8s_nginx_my-nginx-b7d7bc74d-zxx28_default_ae4ee834-fb5d-4ec4-86b1-7834e538c666_0 nicolaka/netshoot termshark

nicolaka/netshoot and termshark

Direct access to the K8s node

It’s the most harmless situation because we don’t have to touch the K8s cluster.

First, we have to know the Docker container name and where it is running on, for example:

kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

my-nginx-b7d7bc74d-zxx28 1/1 Running 0 10m 10.244.2.2 node2 <none> <none> vagrant ssh node2 sudo docker ps | grep my-nginx-b7d7bc74d-zxx28 f22ab3b7130b nginx "/docker-entrypoint.…" 39 minutes ago Up 39 minutes k8s_nginx_my-nginx-b7d7bc74d-zxx28_default_ae4ee834-fb5d-4ec4-86b1-7834e538c666_0

62c7cacd66c1 k8s.gcr.io/pause:3.2 "/pause" 39 minutes ago Up 39 minutes k8s_POD_my-nginx-b7d7bc74d-zxx28_default_ae4ee834-fb5d-4ec4-86b1-7834e538c666_0

Our container is not the pause.

The container name can be cross-checked on the Traefik dashboard:

Traefik dashboard shows the backend container name

Capturing by nicolaka/netshoot container

The nicolaka/netshoot container should be run in the same network namespace (set by --net container: ):

sudo docker run -it --rm --net container:k8s_nginx_my-nginx-b7d7bc74d-zxx28_default_ae4ee835d-4ec4-86b1-7834e538c666_0 nicolaka/netshoot tcpdump -i eth0 -s 0 -Xvv tcp port 80

Capturing in the container network namespace

Docker uses Linux kernel network namespaces to isolate containers. With root access, it can be used for capturing the traffic:

sudo docker inspect --format '{{ .State.Pid }}' k8s_nginx_my-nginx-b7d7bc74d-zxx28_default_ae4ee834-fb5d-4ec4-86b1-7834e538c666_0 sudo nsenter -t 17900 -n /bin/bash -xec 'ip a; tcpdump -i eth0 -s 0 -Xvv tcp port 80'

Capturing in same network namespace

It’s possible to pipe tcpdump captured packets to Wireshark, see an example script for live remote capturing at https://github.com/pgillich/kubeadm-vagrant/blob/master/Ubuntu/capture_pod.sh. Example usage (container name is optional):

./capture_pod.sh default my-nginx-596d59c679-9t4vp nginx

Live remote capturing with tcpdump and Wireshark

Capturing in the container

If you have a statically-compiled tcpdump (see an example here: https://www.androidtcpdump.com/android-tcpdump/compile ), it’s possible to copy the singe binary to the container. See an example for live remote capturing with tcpdump and Wireshark:

kubectl cp tcpdump my-nginx-596d59c679-9t4vp:/tmp;

kubectl exec -t my-nginx-596d59c679-9t4vp -- /tmp/tcpdump -i eth0 -s 0 -w - | wireshark -k -i -

The above steps, including the static compiling, are bounded into one script. See example usage (container name is optional):

./capture_in_pod.sh default my-nginx-596d59c679-9t4vp nginx

This alternative expects the least rights in the Kubernetes cluster: only kubectl cp and exec are required to the Pod (container), without Pod restart. Side container, node access and alpha features are not needed.

Sidecar container in same Pod

This alternative has a drawback: the Pod will be restarted, because a sidecar container is added.

Adding the sidecar container:

kubectl edit deployment my-nginx spec:

(...)

template:

(...)

spec:

containers:

- command:

- /bin/sleep

- infinity

image: nicolaka/netshoot

name: netshoot

- image: nginx:latest

imagePullPolicy: Always

name: nginx

resources: {}

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

Above edit operation can be done by kubectl patch , see more details: https://medium.com/@xxradar/how-to-tcpdump-effectively-in-kubernetes-part-1-a1546b683d2f

Starting netshoot shell:

kubectl exec my-nginx-7bc9465bb9-lqnwd -ti -c netshoot -- /bin/bash

Of course, tcpdump ... command can be written instead of /bin/bash to start a simple capture immediately.

Additional capabilities can be added with securityContext.capabilities.add (for tshark and similar tools), for example:

kubectl patch deployment my-nginx --patch '

spec:

template:

spec:

containers:

- name: netshoot

image: nicolaka/netshoot

command:

- /bin/sleep

- infinity

securityContext:

capabilities:

add:

- NET_ADMIN

- NET_RAW

'

Termshark uses /root/.cache , so it’s recommended to mount an emptyDir volume to there.

It’s possible to capture the traffic from the start. In this case, emptyDir volume should be created and -U option is suggested to tcpdump. See more details:

Ephemeral Debug Containers

Kubernetes 1.16 has a new Ephemeral Containers feature that is ideal for our use case, if the container runtime supports it, it’s enabled (see: feature-gates: EphemeralContainers=true) properly. It’s in alpa yet, see the feature description: https://github.com/kubernetes/enhancements/issues/277

Example for starting a debug container:

kubectl debug -it my-nginx-b7d7bc74d-7t7nr -c netshoot --target=nginx --image=nicolaka/netshoot --image-pull-policy=IfNotPresent -- /bin/bash

See more examples here: https://downey.io/blog/kubernetes-ephemeral-debug-container-tcpdump/

It’s possible to use the raw API, too. Example to get above configuration:

kubectl get --raw /api/v1/namespaces/default/pods/my-nginx-b7d7bc74d-7t7nr/ephemeralcontainers | jq {

"kind": "EphemeralContainers",

"apiVersion": "v1",

"metadata": {

"name": "my-nginx-b7d7bc74d-7t7nr",

"namespace": "default",

"uid": "5363a259-411a-4658-9384-d136ce9b6519",

"resourceVersion": "2893",

"creationTimestamp": "2020-12-27T18:56:18Z"

},

"ephemeralContainers": [

{

"name": "netshoot",

"image": "nicolaka/netshoot",

"command": [

"/bin/bash"

],

"resources": {},

"terminationMessagePath": "/dev/termination-log",

"terminationMessagePolicy": "File",

"imagePullPolicy": "IfNotPresent",

"stdin": true,

"tty": true,

"targetContainerName": "nginx"

}

]

}

See more details here: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

References

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬