Project: gVisor (Runtime / Container Runtime)
File: observability.md

[TOC] This guide describes how to obtain Prometheus monitoring data from gVisor sandboxes running with `runsc`. NOTE: These metrics are mostly information about gVisor internals, and do not provide introspection capabilities into the workload being sandboxed. If you would like to monitor the sandboxed workload (e.g. for threat detection), refer to . `runsc` implements a HTTP metric server using the `runsc metric-server` subcommand. This server is meant to run unsandboxed as a sidecar process of your container runtime (e.g. Docker). You can export metric information from running sandboxes using the `runsc export-metrics` subcommand. This does not require special configuration or setting up a Prometheus server. ``` $ docker run -d --runtime=runsc --name=foobar debian sleep 1h c7ce77796e0ece4c0881fb26261608552ea4a67b2fe5934658b8b4433e5190ed $ sudo /path/to/runsc --root=/var/run/docker/runtime-runc/moby export-metrics c7ce77796e0ece4c0881fb26261608552ea4a67b2fe5934658b8b4433e5190ed Command-line export for sandbox c7ce77796e0ece4c0881fb26261608552ea4a67b2fe5934658b8b4433e5190ed Writing data from snapshot containing 175 data points taken at 2023-01-25 15:46:50.469403696 -0800 PST. HELP runscfsopens Number of file opens. TYPE runscfsopens counter runscfsopens{sandbox="c7ce77796e0ece4c0881fb26261608552ea4a67b2fe5934658b8b4433e5190ed"} 62 1674690410469 HELP runscfsread_wait Time waiting on file reads, in nanoseconds. TYPE runscfsread_wait counter runscfsread_wait{sandbox="c7ce77796e0ece4c0881fb26261608552ea4a67b2fe5934658b8b4433e5190ed"} 0 1674690410469 HELP runscfsreads Number of file reads. TYPE runscfsreads counter runscfsreads{sandbox="c7ce77796e0ece4c0881fb26261608552ea4a67b2fe5934658b8b4433e5190ed"} 54 1674690410469 [...] ``` Use the `runsc metric-server` subcommand: ```shell $ sudo runsc \ --root=/var/run/docker/runtime-runc/moby \ --metric-server=localhost:1337 \ metric-server ``` `--root` needs to be set to the OCI runtime root directory that your runtime implementation uses. For Docker, this is typically `/var/run/docker/runtime-runc/moby`; otherwise, if you already have gVisor set up, you can use `ps aux | grep runsc` on the host to find the `--root` that a running sandbox is using. This directory is typically only accessible by the user Docker runs as (usually `root`), hence `sudo`. The metric server uses the `--root` directory to scan for sandboxes running on the system. The `--metric-server` flag is the network address or UDS path to bind to. In this example, this will create a server bound on all interfaces on TCP port `1337`. To listen on `lo` only, you could alternatively use `--metric-server=localhost:1337`. If something goes wrong, you may also want to add `--debug --debug-log=/dev/stderr` to understand the metric server's behavior. You can query the metric server with `curl`: ``` $ curl http://localhost:1337/metrics Data for runsc metric server exporting data for sandboxes in root directory /var/run/docker/runtime-runc/moby [...] HELP processstarttime_seconds Unix timestamp at which the process started. Used by Prometheus for counter resets. TYPE processstarttime_seconds gauge processstarttime_seconds 1674598082.698509 1674598109532 End of metric data. ``` Sandbox metrics are disabled by default. To enable, add the flag `--metric-server={ADDRESS}:{PORT}` to the runtime configuration. With Docker, this can be set in `/etc/docker/daemon.json` like so: ```json { "runtimes": { "runsc": { "path": "/path/to/runsc", "runtimeArgs": [ "--metric-server=localhost:1337" ] } } } ``` NOTE: The `--metric-server` flag value must be an exact string match between the runtime configuration and the `runsc metric-server` command. Once you've done this, you can start a container and see that it shows up in the list of Prometheus metrics. ``` $ docker run -d --runtime=runsc --name=foobar debian sleep 1h 32beefcafe $ curl http://localhost:1337/metrics Data for runsc metric server exporting data for sandboxes in root directory /var/run/docker/runtime-runc/moby Writing data from 3 snapshots: [...] HELP processstarttime_seconds Unix timestamp at which the process

Used by Prometheus for counter resets. TYPE processstarttime_seconds gauge processstarttime_seconds 1674599158.286067 1674599159819 HELP runscfsopens Number of file opens. TYPE runscfsopens counter runscfsopens{iteration="42asdf",sandbox="32beefcafe"} 12 1674599159819 HELP runscfsread_wait Time waiting on file reads, in nanoseconds. TYPE runscfsread_wait counter runscfsread_wait{iteration="42asdf",sandbox="32beefcafe"} 0 1674599159819 [...] End of metric data. ``` Each per-container metric is labeled with at least: `sandbox`: The container ID, in this case `32beefcafe` `iteration`: A randomly-generated string (in this case `42asdf`) that stays constant for the lifetime of the sandbox. This helps distinguish between successive instances of the same sandbox with the same ID. If you'd like to run some containers with metrics turned off and some on within the same system, use two runtime entries in `/etc/docker/daemon.json` with only one of them having the `--metric-server` flag set. The metric server exposes a on the address given by the `--metric-server` flag passed to `runsc metric-server`. Simply point Prometheus at this address. If desired, you can change the (prefix applied to all metric names) using the `--exporter-prefix` flag. It defaults to `runsc_`. The sandbox metrics exported may be filtered by using the optional `GET` parameter `runsc-sandbox-metrics-filter`, e.g. `/metrics?runsc-sandbox-metrics-filter=fs_.*`. Metric names must fully match this regular expression. Note that this filtering is performed before prepending `--exporter-prefix` to metric names. The metric server also supports listening on a . This can be convenient to avoid reserving port numbers on the machine's network interface, or for tighter control over who can read the data. Clients should talk HTTP over this UDS. While Prometheus doesn't natively support reading metrics from a UDS, this feature can be used in conjunction with a tool such as within another context (e.g. a tightly-managed network namespace that Prometheus runs in). ``` $ sudo runsc --root=/var/run/docker/runtime-runc/moby --metric-server=/run/docker/runsc-metrics.sock metric-server & $ sudo curl --unix-socket /run/docker/runsc-metrics.sock http://runsc-metrics/metrics Data for runsc metric server exporting data for sandboxes in root directory /var/run/docker/runtime-runc/moby [...] End of metric data. Set up socat to forward requests from *:1337 to /run/docker/runsc-metrics.sock in its own network namespace: $ sudo unshare --net socat TCP-LISTEN:1337,reuseaddr,fork UNIX-CONNECT:/run/docker/runsc-metrics.sock & Set up basic networking for socat's network namespace: $ sudo nsenter --net="/proc/$(pidof socat)/ns/net" sh -c 'ip link set lo up && ip route add default dev lo' Grab metric data from this namespace: $ sudo nsenter --net="/proc/$(pidof socat)/ns/net" curl http://localhost:1337/metrics Data for runsc metric server exporting data for sandboxes in root directory /var/run/docker/runtime-runc/moby [...] End of metric data. ``` If you would like to run the metric server in a gVisor sandbox, you may do so, provided that you give it access to the OCI runtime root directory, forward the network port it binds to for external access, and enable host UDS support. WARNING: Doing this does not provide you the full security of gVisor, as it still grants the metric server full control over all running gVisor sandboxes on the system. This step is only a defense-in-depth measure. To do this, add a runtime with the `--host-uds=all` flag to `/etc/docker/daemon.json`. The metric server needs the ability to open existing UDSs (in order to communicate with running sandboxes), and to create new UDSs (in order to create and listen on

```json { "runtimes": { "runsc": { "path": "/path/to/runsc", "runtimeArgs": [ "--metric-server=/run/docker/runsc-metrics.sock" ] }, "runsc-metric-server": { "path": "/path/to/runsc", "runtimeArgs": [ "--metric-server=/run/docker/runsc-metrics.sock", "--host-uds=all" ] } } } ``` Then start the metric server with this runtime, passing through the directories containing the control files `runsc` uses to detect and communicate with running sandboxes: ```shell $ docker run -d --runtime=runsc-metric-server --name=runsc-metric-server \ --volume="$(which runsc):/runsc:ro" \ --volume=/var/run/docker/runtime-runc/moby:/var/run/docker/runtime-runc/moby \ --volume=/run/docker:/run/docker \ --volume=/var/run:/var/run \ alpine \ /runsc \ --root=/var/run/docker/runtime-runc/moby \ --metric-server=/run/docker/runsc-metrics.sock \ --debug --debug-log=/dev/stderr \ metric-server ``` Yes, this means the metric server will report data about its own sandbox: ``` $ metricserverid="$(docker inspect --format='{{.ID}}' runsc-metric-server)" $ sudo curl --unix-socket /run/docker/runsc-metrics.sock http://runsc-metrics/metrics | grep "$metricserverid" Snapshot with 175 data points taken at 2023-01-25 15:45:33.70256855 -0800 -0800: map[iteration:2407456650315156914 sandbox:737ce142058561d764ad870d028130a29944821dd918c7979351b249d5d30481] runscfsopens{iteration="2407456650315156914",sandbox="737ce142058561d764ad870d028130a29944821dd918c7979351b249d5d30481"} 54 1674690333702 runscfsread_wait{iteration="2407456650315156914",sandbox="737ce142058561d764ad870d028130a29944821dd918c7979351b249d5d30481"} 0 1674690333702 runscfsreads{iteration="2407456650315156914",sandbox="737ce142058561d764ad870d028130a29944821dd918c7979351b249d5d30481"} 52 1674690333702 [...] ``` When using Kubernetes, users typically deal with pod names and container names. On Kubelet machines, the underlying container names passed to the runtime are non-human-friendly hexadecimal strings. In order to provide more user-friendly labels, the metric server will pick up the `io.kubernetes.cri.sandbox-name` and `io.kubernetes.cri.sandbox-namespace` annotations provided by `containerd`, and automatically add these as labels (`podname` and `namespacename` respectively) for each per-sandbox metric. The metric server exports a lot of gVisor-internal metrics, and generates its own metrics as well. All metrics have documentation and type annotations in the `/metrics` output, and this section aims to document some useful ones. `processstarttime_seconds`: Unix timestamp representing the time at which the metric server started. This specific metric name is used by Prometheus, and as such its name is not affected by the `--exporter-prefix` flag. This metric is process-wide and has no labels. `numsandboxestotal`: A process-wide metric representing the total number of sandboxes that the metric server knows about. `numsandboxesrunning`: A process-wide metric representing the number of running sandboxes that the metric server knows about. `numsandboxesbroken_metrics`: A process-wide metric representing the number of sandboxes from which the metric server could not get metric data. `sandbox_presence`: A per-sandbox metric that is set to `1` for each sandbox that the metric server knows about. This can be used to join with other per-sandbox or per-pod metrics for which metric existence is not guaranteed. `sandbox_running`: A per-sandbox metric that is set to `1` for each sandbox that the metric server knows about and that is actively running. This can be used in conjunction with `sandbox_presence` to determine the set of sandboxes that aren't running; useful if you want to alert about sandboxes that are down. `sandbox_metadata`: A per-sandbox metric that carries a superset of the typical per-sandbox labels found on other per-sandbox metrics. These extra labels contain useful metadata about the sandbox, such as the version number, , and being used. `sandbox_capabilities`: A per-sandbox, per-capability metric that carries the union of all capabilities present on at least one container of the sandbox. Can optionally be filtered to only a subset of capabilities using the `runsc-capability-filter` GET parameter on `/metrics` requests (regular expression). Useful for auditing and aggregating the capabilities you rely on across multiple sandboxes. `sandboxcreationtime_seconds`: A per-sandbox Unix timestamp representing the time at which this sandbox was created.