Project: Tetragon (Provisioning / Security & Compliance)
File: events.md

title: "Events" weight: 1 description: "Documentation for Tetragon Events" Tetragon's events are exposed to the system through either the gRPC endpoint or JSON logs. Commands in this section assume the Getting Started guide was used, but are general other than the namespaces chosen and should work in most environments. The first way is to observe the raw json output from the stdout container log: ```shell kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f ``` The raw JSON events provide Kubernetes API, identity metadata, and OS level process visibility about the executed binary, its parent and the execution time. A base Tetragon installation will produce `processexec` and `processexit` events encoded in JSON as shown here, <details><summary>Process execution event</summary> <p> ```json { "process_exec": { "process": { "exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4Njc0MzIxMzczOjUyNjk5", "pid": 52699, "uid": 0, "cwd": "/", "binary": "/usr/bin/curl", "arguments": "https://ebpf.io/applications/#tetragon", "flags": "execve rootcwd", "start_time": "2023-10-06T22:03:57.700327580Z", "auid": 4294967295, "pod": { "namespace": "default", "name": "xwing", "container": { "id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3", "name": "spaceship", "image": { "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6", "name": "docker.io/tgraf/netperf:latest" }, "start_time": "2023-10-06T21:52:41Z", "pid": 49 }, "pod_labels": { "app.kubernetes.io/name": "xwing", "class": "xwing", "org": "alliance" }, "workload": "xwing" }, "docker": "551e161c47d8ff0eb665438a7bcd5b4", "parentexecid": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5", "tid": 52699 }, "parent": { "exec_id": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjcwODgzMjk5OjUyNjk5", "pid": 52699, "uid": 0, "cwd": "/", "binary": "/bin/bash", "arguments": "-c \"curl https://ebpf.io/applications/#tetragon\"", "flags": "execve rootcwd clone", "start_time": "2023-10-06T22:03:57.696889812Z", "auid": 4294967295, "pod": { "namespace": "default", "name": "xwing", "container": { "id": "containerd://551e161c47d8ff0eb665438a7bcd5b4e3ef5a297282b40a92b7c77d6bd168eb3", "name": "spaceship", "image": { "id": "docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6", "name": "docker.io/tgraf/netperf:latest" }, "start_time": "2023-10-06T21:52:41Z", "pid": 49 }, "pod_labels": { "app.kubernetes.io/name": "xwing", "class": "xwing", "org": "alliance" }, "workload": "xwing" }, "docker": "551e161c47d8ff0eb665438a7bcd5b4", "parentexecid": "Z2tlLWpvaG4tNjMyLWRlZmF1bHQtcG9vbC03MDQxY2FjMC05czk1OjEzNTQ4NjQ1MjQ1ODM5OjUyNjg5", "tid": 52699 } }, "node_name": "gke-john-632-default-pool-7041cac0-9s95", "time": "2023-10-06T22:03:57.700326678Z" } ``` </p> </details> Will only highlight a few important fields here. For a full specification of events see the Reference section. All events in Tetragon contain a `process_exec` block to identify the process generating the event. For execution events this is the primary block. For events the hook that generated the event will attach further data to this. The `process_exec` event provides a cluster wide unique id the `processexec.execid` for this process along with the metadata expected in a Kubernetes cluster `process_exec.process.pod`. The binary and args being executed are part of the event here `processexec.process.binary` and `processexec.process.args`. Finally, a `node_name` and `time` provide the location and time for the event and will be present in all event types. A default deployment writes the JSON log to `/var/run/cilium/tetragon/tetragon.log` where it can be exported through normal log collection tooling, e.g. 'fluentd', logstash, etc.. The file will be rotated and compressed by default. See [Helm Options] for details on how to customize this location. Export filters restrict the JSON event output to a subset of desirable events. These export filters are configured as a line-separated list of JSON objects, where each object can contain one or more filter expressions. Filters are combined by taking the logical OR of each line-separated filter object and the logical AND of sibling expressions within a filter

As a concrete example, suppose we had the following filter configuration: ```json {"eventset": ["PROCESSEXEC", "PROCESS_EXIT"], "namespace": "foo"} {"eventset": ["PROCESSKPROBE"]} ``` The above filter configuration would result in a match if: The event type is `PROCESSEXEC` or `PROCESSEXIT` AND the pod namespace is "foo"; OR The event type is `PROCESS_KPROBE` Tetragon supports two groups of export filters: an allowlist and a denylist. If neither is configured, all events are exported. If only an allowlist is configured, event exports are considered default-deny, meaning only the events in the allowlist are exported. The denylist takes precedence over the allowlist in cases where two filter configurations match on the same event. You can configure export filters using the provided helm options, command line flags, or environment variables. | Filter | Description | | | -- | | `eventset` | Filter process events by event types. Supported types include: `PROCESSEXEC`, `PROCESSEXIT`, `PROCESSKPROBE`, `PROCESSUPROBE`, `PROCESSTRACAEPOINT`, `PROCESS_LOADER` | | `binary_regex` | Filter process events by a list of regular expressions of process binary names (e.g. `"^/home/kubernetes/bin/kubelet$"`). You can find the full syntax . | | `health_check` | Filter process events if their binary names match Kubernetes liveness / readiness probe commands of their corresponding pods. | | `namespace` | Filter by Kubernetes pod namespaces. An empty string (`""`) filters processes that do not belong to any pod namespace. | | `pid` | Filter by process PID. | | `pid_set` | Like `pid` but also includes processes that are descendants of the listed PIDs. | | `pod_regex` | Filter by pod name using a list of regular expressions. You can find the full syntax . | | `arguments_regex` | Filter by pod name using a list of regular expressions. You can find the full syntax . | | `labels` | Filter events by pod labels using Note that this filter never matches events without the pod field (i.e. host process events). | | `policy_names` | Filter events by tracing policy names. | | `capabilities` | Filter events by Linux process capability. | In some cases, it is not desirable to include all of the fields exported in Tetragon events by default. In these cases, you can use field filters to restrict the set of exported fields for a given event type. Field filters are configured similarly to export filters, as line-separated lists of JSON objects. Field filters select fields using the under the `"fields"` key. You can define a path of fields using field names separated by period (`.`) characters. To define multiple paths in a single field filter, separate them with comma (`,`) characters. For example, `"fields":"process.binary,parent.binary,pod.name"` would select only the `process.binary`, `parent.binary`, and `pod.name` fields. By default, a field filter applies to all process events, although you can control this behaviour with the `"event_set"` key. For example, you can apply a field filter to `PROCESSCONNECT` and `PROCESSCLOSE` events by specifying `"eventset":["PROCESSCONNECT","PROCESS_CLOSE"]` in the filter definition. Each field filter has an `"action"` that determines what the filter should do with the selected field. The supported action types are `"INCLUDE"` and `"EXCLUDE"`. A value of `"INCLUDE"` will cause the field to appear in an event, while a value of `"EXCLUDE"` will hide the field. In the absence of any field filter for a given event type, the export will include all fields by default. Defining one or more `"INCLUDE"` filters for a given event type changes that behaviour to exclude all other event types by

As a simple example of the above, consider the case where we want to include only `execid` and `parentexec_id` in all event types except for `PROCESS_EXEC`: ```json {"fields":"process.execid,process.parentexecid", "eventset": ["PROCESSEXEC"], "invertevent_set": true, "action": "INCLUDE"} ``` Since Tetragon traces the entire system, event exports might sometimes contain sensitive information (for example, a secret passed via a command line argument to a process). To prevent this information from being exfiltrated via Tetragon JSON export, Tetragon provides a mechanism called Redaction Filters which can be used to string patterns to redact from exported process arguments. These filters are written in JSON and passed to the Tetragon agent via the `--redaction-filters` command line flag or the `redactionFilters` Helm value. To perform redactions, redaction filters define RE2 regular expressions in the `redact` field. Any capture groups in these RE2 regular expressions are redacted and replaced with `"*"`. {{< note >}} This feature uses RE2 as its regular expression library. Make sure that you follow RE2 regular expression guidelines as you may observe unexpected results otherwise. More information on RE2 syntax can be found . {{< /note >}} {{< warning >}} When writing regular expressions in JSON, it is important to escape backslash characters. For instance `\Wpasswd\W?` would be written as `{"redact": "\\Wpasswd\\W?"}`. {{< /warning >}} For more control, you can select which binary or binaries should have their arguments redacted with the `binary_regex` field. As a concrete example, the following will redact all passwords passed to processes with the `"--password"` argument: ```json {"redact": ["--password(?:\\s+|=)(\\S*)"]} ``` Now, an event that contains the string `"--password=foo"` would have that string replaced with `"--password=*"`. Suppose we also see some passwords passed via the -p shorthand for a specific binary, foo. We can also redact these as follows: ```json {"binary_regex": ["(?:^|/)foo$"], "redact": ["-p(?:\\s+|=)(\\S*)"]} ``` With both of the above redaction filters in place, we are now redacting all password arguments. A second way is to use the CLI. This has the advantage that it can also be used to filter and pretty print the output. The tool allows filtering by process, pod, and other fields. To install tetra see the To start printing events run: ```shell kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact ``` The `tetra` CLI is also available inside `tetragon` container. ```shell kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact ``` This was used in the quick start and generates a pretty printing of the events, To further filter by a specific binary and/or pod do the following, ```shell kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --processes curl --pod xwing ``` Will filter and report just the relevant events. ``` process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 60 ``` In addition Tetragon can expose a gRPC endpoint listeners may attach to. The gRPC is exposed by default helm install on `localhost:54321`, but the address can be configured with the `--server-address` option. This can be set from helm with the `tetragon.grpc.address` flag or disabled completely if needed with `tetragon.grpc.enabled`. ```shell helm install tetragon cilium/tetragon -n kube-system --set tetragon.grpc.enabled=true --set tetragon.grpc.address=localhost:54321 ``` An example gRPC endpoint is the Tetra CLI when its not piped JSON output directly, ```shell kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact ```