Project: Tetragon (Provisioning / Security & Compliance)
File: monitor-changes-at-kernel.md

title: "Monitor Process Credentials changes at the Kernel layer" weight: 2 description: "Monitor Process Credentials changes at the kernel layer" Monitoring Process Credentials changes at the kernel layer is also possible. This allows to capture the new that should be applied. This tracing policy can be used to answer the following questions: Which process or container is trying to change its own UIDs/GIDs in the cluster? Which process or container is trying to change its own capabilities in the cluster? In which the credentials are being changed? How to monitor changes? The main advantages of monitoring at the kernel layer compared to the : Not vulnerable to user space arguments tampering. Ability to display the full new credentials to be applied. It is more reliable since it has full context on where and how the new credentials should be applied including the user namespace. A catch all layer for all system calls, and every normal kernel path that manipulate credentials. One potential disadvantage is that this approach may generate a lot of events, so appropriate filtering must be applied to reduce the noise. First, verify that your k8s environment is all setup and that all pods are up and running, and deploy the Demo Application: ```shell kubectl create -f {{< demo-app-url >}} ``` It might take several seconds until all pods are Running: ```shell kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE default deathstar-54bb8475cc-6c6lc 1/1 Running 0 2m54s default deathstar-54bb8475cc-zmfkr 1/1 Running 0 2m54s default tiefighter 1/1 Running 0 2m54s default xwing 1/1 Running 0 2m54s kube-system tetragon-sdwv6 2/2 Running 0 27m ``` We use the Tracing Policy that hooks the kernel layer when credentials are being installed. So let's apply the Tracing Policy. ```shell kubectl apply -f

``` Then we start monitoring for events with `tetra` cli: ```shell kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents ``` In another terminal, inside a pod and as a non root user we will execute a setuid binary (suid): ```shell /tmp/su - ``` The `tetra` cli will generate the following events: ```json { "process_kprobe": { "process": { "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjE0MDczMDQyODk3MTc6MjIzODY=", "pid": 22386, "uid": 11, "cwd": "/", "binary": "/tmp/su", "arguments": "-", "flags": "execve rootcwd clone", "start_time": "2023-07-25T12:04:59.359333454Z", "auid": 4294967295, "pod": { "namespace": "default", "name": "xwing", "container": { "id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64", "name": "spaceship", "image": { "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6", "name": "docker.io/tgraf/netperf:latest" }, "start_time": "2023-07-25T11:44:48Z", "pid": 43 }, "pod_labels": { "app.kubernetes.io/name": "xwing", "class": "xwing", "org": "alliance" } }, "docker": "2e58c8357465961fd96f758e87d0269", "parentexecid": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=", "refcnt": 1, "tid": 22386 }, "parent": { "exec_id": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDY5OTA1ODc6MjIzNTI=", "pid": 22352, "uid": 11, "cwd": "/", "binary": "/bin/sh", "flags": "execve rootcwd", "start_time": "2023-07-25T12:04:47.462035587Z", "auid": 4294967295, "pod": { "namespace": "default", "name": "xwing", "container": { "id": "containerd://2e58c8357465961fd96f758e87d0269dfb5f97c536847485de9d7ec62be34a64", "name": "spaceship", "image": { "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6", "name": "docker.io/tgraf/netperf:latest" }, "start_time": "2023-07-25T11:44:48Z", "pid": 41 }, "pod_labels": { "app.kubernetes.io/name": "xwing", "class": "xwing", "org": "alliance" } }, "docker": "2e58c8357465961fd96f758e87d0269", "parentexecid": "a2luZC1jb250cm9sLXBsYW5lOjEzOTU0MDQ3NzY5NzI6MjIzNTI=", "refcnt": 2, "tid": 22352 }, "functionname": "commitcreds", "args": [ { "processcredentialsarg": { "uid": 0, "gid": 0, "euid": 0, "egid": 0, "suid": 0, "sgid": 0, "fsuid": 0, "fsgid": 0, "caps": { "permitted": [ "CAP_CHOWN", "DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_KILL", "CAP_SETGID", "CAP_SETUID", "CAP_SETPCAP", "CAPNETBIND_SERVICE", "CAPNETRAW", "CAPSYSCHROOT", "CAP_MKNOD", "CAPAUDITWRITE", "CAP_SETFCAP" ], "effective": [ "CAP_CHOWN", "DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_KILL", "CAP_SETGID", "CAP_SETUID", "CAP_SETPCAP", "CAPNETBIND_SERVICE", "CAPNETRAW", "CAPSYSCHROOT", "CAP_MKNOD", "CAPAUDITWRITE", "CAP_SETFCAP" ] }, "user_ns": { "level": 0, "uid": 0, "gid": 0, "ns": { "inum": 4026531837, "is_host": true } } } } ], "action": "KPROBEACTIONPOST" }, "node_name": "kind-control-plane", "time": "2023-07-25T12:05:01.410834172Z" } ``` In addition to the Kubernetes Identity and process metadata from exec events, events contain the arguments of the observed system call. In the above case they are: `functionname`: the kernel `commitcreds()` function to install new credentials. `processcredentialsarg`: the new to be installed on the current process. It includes the UIDs/GIDs, the capabilities and the target user namespace. Here we can clearly see that the suid binary is being executed by a user ID `11` in order to elevate its privileges to user ID `0` including capabilities. To disable the run: ```shell kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/process-credentials/process-creds-installed.yaml ```