Project: Kata Containers (Runtime / Container Runtime)
File: kata-2-0-metrics.md

Kata implements CRI's API and supports and interfaces to expose containers metrics. User can use these interfaces to get basic metrics about containers. Unlike `runc`, Kata is a VM-based runtime and has a different architecture. Kata 1.x has a number of limitations related to observability that may be obstacles to running Kata Containers at scale. In Kata 2.0, the following components will be able to provide more details about the system: containerd shim v2 (effectively `kata-runtime`) Hypervisor statistics Agent process Guest OS statistics Note: In Kata 1.x, the main user-facing component was the runtime (`kata-runtime`). From 1.5, Kata introduced the Kata containerd shim v2 (`containerd-shim-kata-v2`) which is essentially a modified runtime that is loaded by containerd to simplify and improve the way VM-based containers are created and managed. For Kata 2.0, the main component is the Kata containerd shim v2, although the deprecated `kata-runtime` binary will be maintained for a period of time. Any mention of the "Kata runtime" in this document should be taken to refer to the Kata containerd shim v2 unless explicitly noted otherwise (for example by referring to it explicitly as the `kata-runtime` binary). Kata 2.0 metrics strongly depend on , a graduated project from CNCF. Kata Containers 2.0 introduces a new Kata component called `kata-monitor` which is used to monitor the Kata components on the host. It's shipped with the Kata runtime to provide an interface to: Get metrics Get events At present, `kata-monitor` supports retrieval of metrics only: this is what will be covered in this document. This is the architecture overview of metrics in Kata Containers 2.0: And the sequence diagram is shown below: For a quick evaluation, you can check out . The `kata-monitor` management agent should be started on each node where the Kata containers runtime is installed. `kata-monitor` will: Note: a node running Kata containers will be either a single host system or a worker node belonging to a K8s cluster capable of running Kata pods. Aggregate sandbox metrics running on the node, adding the `sandbox_id` label to them. Attach the additional `criuid`, `criname` and `cri_namespace` labels to the sandbox metrics, tracking the `uid`, `name` and `namespace` Kubernetes pod metadata. Expose a new Prometheus target, allowing all node metrics coming from the Kata shim to be collected by Prometheus indirectly. This simplifies the targets count in Prometheus and avoids exposing shim's metrics by `ip:port`. Only one `kata-monitor` process runs in each node. `kata-monitor` uses a different communication channel than the one used by the container engine (`containerd`/`CRI-O`) to communicate with the Kata shim. The Kata shim exposes a dedicated socket address reserved to `kata-monitor`. The shim's metrics socket file is created under the virtcontainers sandboxes directory, i.e. `vc/sbs/${PODID}/shim-monitor.sock`. Note: If there is no Prometheus server configured, i.e., there are no scrape operations, `kata-monitor` will not collect any metrics. Kata runtime is responsible for: Gather metrics about shim process Gather metrics about hypervisor process Gather metrics about running sandbox Get metrics from Kata agent (through `ttrpc`) Kata agent is responsible for: Gather agent process metrics Gather guest OS metrics In Kata 2.0, the agent adds a new interface: ```protobuf rpc GetMetrics(GetMetricsRequest) returns (Metrics); message GetMetricsRequest {} message Metrics { string metrics = 1; } ``` The `metrics` field is Prometheus encoded content. This can avoid defining a fixed structure in protocol buffers. Metrics should not become a bottleneck for the system or downgrade the performance: they should run with minimal

Requirements: Metrics MUST* be quick to collect Metrics MUST* be small Metrics MUST* be generated only if there are subscribers to the Kata metrics service Metrics MUST* be stateless In Kata 2.0, metrics are collected only when needed (pull mode), mainly from the `/proc` filesystem, and consumed by Prometheus. This means that if the Prometheus collector is not running (so no one cares about the metrics) the overhead will be zero. The metrics service also doesn't hold any metrics in memory. |\*|No Sandbox | 1 Sandbox | 2 Sandboxes | ||||| |Metrics count| 39 | 106 | 173 | |Metrics size (bytes)| 9K | 144K | 283K | |Metrics size (`gzipped`, bytes)| 2K | 10K | 17K | Metrics size: response size of one Prometheus scrape request. It's easy to estimate the size of one metrics fetch request issued by Prometheus. The formula to calculate the expected size when no gzip compression is in place is: 9 + (144 - 9) * `number of kata sandboxes` Prometheus supports `gzip compression`. When enabled, the response size of each request will be smaller: 2 + (10 - 2) * `number of kata sandboxes` Example We have 10 sandboxes running on a node. The expected size of one metrics fetch request issued by Prometheus against the kata-monitor agent running on that node will be: 9 + (144 - 9) 10 = 1.35M* If `gzip compression` is enabled: 2 + (10 - 2) 10 = 82K* And here is some test data: End-to-end (from Prometheus server to `kata-monitor` and `kata-monitor` write response back): 20ms(avg) Agent (RPC all from shim to agent): 3ms(avg) Test infrastructure: OS: Ubuntu 20.04 Hardware: Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz, 6 Cores, and 16GB memory. Scrape interval Prometheus default `scrapeinterval` is 1 minute, but it is usually set to 15 seconds. A smaller `scrapeinterval` causes more overhead, so users should set it depending on their monitoring needs. Here are listed all the metrics supported by Kata 2.0. Some metrics are dependent on the VM guest kernel, so the available ones may differ based on the environment. Metrics are categorized by the component from/for which the metrics are collected. Note: Labels here do not include the `instance` and `job` labels added by Prometheus. Notes about metrics unit `Kibibytes`, abbreviated `KiB`. 1 `KiB` equals 1024 B. For some metrics (like network devices statistics from file `/proc/net/dev`), unit depends on label( for example `recvbytes` and `recvpackets` have different units). Most of these metrics are collected from the `/proc` filesystem, so the unit of each metric matches the unit of the relevant `/proc` entry. See the `proc(5)` manual page for further details. Prometheus offers four core metric types. Counter: A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase. Gauge: A gauge metric represents a single numerical value that can go up and down, typically used for measured values like current memory usage. Histogram: A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. Summary: A summary samples observations like histogram, it can calculate configurable quantiles over a sliding time window. See for detailed explanations about these metric types. Agent's metrics contains metrics about agent process. | Metric name | Type | Units | Labels | Introduced in Kata version | |||||| | `kataagentiostat`: <br> Agent process IO stat. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/io`)<ul><li>`cancelledwritebyte`</li><li>`rchar`</li><li>`readbytes`</li><li>`syscr`</li><li>`syscw`</li><li>`wchar`</li><li>`writebytes`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagentprocstat`: <br> Agent process stat. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/stat`)<ul><li>`cstime`</li><li>`cutime`</li><li>`stime`</li><li>`utime`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagentprocstatus`: <br> Agent process status. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/status`)<ul><li>`hugetlbpages`</li><li>`nonvoluntaryctxtswitches`</li><li>`rssanon`</li><li>`rssfile`</li><li>`rssshmem`</li><li>`vmdata`</li><li>`vmexe`</li><li>`vmhwm`</li><li>`vmlck`</li><li>`vmlib`</li><li>`vmpeak`</li><li>`vmpin`</li><li>`vmpte`</li><li>`vmrss`</li><li>`vmsize`</li><li>`vmstk`</li><li>`vmswap`</li><li>`voluntaryctxtswitches`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagentprocesscpusecondstotal`: <br> Total user and system CPU time spent in

| `COUNTER` | `seconds` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagentprocessmaxfds`: <br> Maximum number of open file descriptors. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `kataagentprocessopenfds`: <br> Number of open file descriptors. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `kataagentprocessresidentmemorybytes`: <br> Resident memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagentprocessstarttimeseconds`: <br> Start time of the process since `unix` epoch in seconds. | `GAUGE` | `seconds` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagentprocessvirtualmemorybytes`: <br> Virtual memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagentscrapecount`: <br> Metrics scrape count | `COUNTER` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagenttotalrss`: <br> Agent process total `rss` size | `GAUGE` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagenttotaltime`: <br> Agent process total time | `GAUGE` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `kataagenttotalvm`: <br> Agent process total `vm` size | `GAUGE` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | Metrics for Firecracker vmm. | Metric name | Type | Units | Labels | Introduced in Kata version | |||||| | `katafirecrackerapiserver`: <br> Metrics related to the internal API server. | `GAUGE` | | <ul><li>`item`<ul><li>`processstartuptimecpuus`</li><li>`processstartuptimeus`</li><li>`syncresponsefails`</li><li>`syncvmmsendtimeoutcount`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackerblock`: <br> Block Device associated metrics. | `GAUGE` | | <ul><li>`item`<ul><li>`activatefails`</li><li>`cfgfails`</li><li>`eventfails`</li><li>`executefails`</li><li>`flushcount`</li><li>`invalidreqscount`</li><li>`noavailbuffer`</li><li>`queueeventcount`</li><li>`ratelimitereventcount`</li><li>`ratelimiterthrottledevents`</li><li>`readbytes`</li><li>`readcount`</li><li>`updatecount`</li><li>`updatefails`</li><li>`writebytes`</li><li>`writecount`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katafirecrackergetapirequests`: <br> Metrics specific to GET API Requests for counting user triggered actions and/or failures. | `GAUGE` | | <ul><li>`item`<ul><li>`instanceinfocount`</li><li>`instanceinfofails`</li><li>`machinecfgcount`</li><li>`machinecfgfails`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackeri8042`: <br> Metrics specific to the i8042 device. | `GAUGE` | | <ul><li>`item`<ul><li>`errorcount`</li><li>`missedreadcount`</li><li>`missedwritecount`</li><li>`readcount`</li><li>`resetcount`</li><li>`writecount`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackerlatenciesus`: <br> Performance metrics related for the moment only to snapshots. | `GAUGE` | | <ul><li>`item`<ul><li>`diffcreatesnapshot`</li><li>`fullcreatesnapshot`</li><li>`loadsnapshot`</li><li>`pausevm`</li><li>`resumevm`</li><li>`vmmdiffcreatesnapshot`</li><li>`vmmfullcreatesnapshot`</li><li>`vmmloadsnapshot`</li><li>`vmmpausevm`</li><li>`vmmresumevm`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackerlogger`: <br> Metrics for the logging subsystem. | `GAUGE` | | <ul><li>`item`<ul><li>`logfails`</li><li>`metricsfails`</li><li>`missedlogcount`</li><li>`missedmetricscount`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackermmds`: <br> Metrics for the MMDS functionality. | `GAUGE` | | <ul><li>`item`<ul><li>`connectionscreated`</li><li>`connectionsdestroyed`</li><li>`rxaccepted`</li><li>`rxacceptederr`</li><li>`rxacceptedunusual`</li><li>`rxbadeth`</li><li>`rxcount`</li><li>`txbytes`</li><li>`txcount`</li><li>`txerrors`</li><li>`txframes`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackernet`: <br> Network-related metrics. | `GAUGE` | | <ul><li>`item`<ul><li>`activatefails`</li><li>`cfgfails`</li><li>`eventfails`</li><li>`macaddressupdates`</li><li>`norxavailbuffer`</li><li>`notxavailbuffer`</li><li>`rxbytescount`</li><li>`rxcount`</li><li>`rxeventratelimitercount`</li><li>`rxfails`</li><li>`rxpacketscount`</li><li>`rxpartialwrites`</li><li>`rxqueueeventcount`</li><li>`rxratelimiterthrottled`</li><li>`rxtapeventcount`</li><li>`tapreadfails`</li><li>`tapwritefails`</li><li>`txbytescount`</li><li>`txcount`</li><li>`txfails`</li><li>`txmalformedframes`</li><li>`txpacketscount`</li><li>`txpartialreads`</li><li>`txqueueeventcount`</li><li>`txratelimitereventcount`</li><li>`txratelimiterthrottled`</li><li>`txspoofedmaccount`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katafirecrackerpatchapirequests`: <br> Metrics specific to PATCH API Requests for counting user triggered actions and/or failures. | `GAUGE` | | <ul><li>`item`<ul><li>`drivecount`</li><li>`drivefails`</li><li>`machinecfgcount`</li><li>`machinecfgfails`</li><li>`networkcount`</li><li>`networkfails`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackerputapirequests`: <br> Metrics specific to PUT API Requests for counting user triggered actions and/or failures. | `GAUGE` | | <ul><li>`item`<ul><li>`actionscount`</li><li>`actionsfails`</li><li>`bootsourcecount`</li><li>`bootsourcefails`</li><li>`drivecount`</li><li>`drivefails`</li><li>`loggercount`</li><li>`loggerfails`</li><li>`machinecfgcount`</li><li>`machinecfgfails`</li><li>`metricscount`</li><li>`metricsfails`</li><li>`networkcount`</li><li>`networkfails`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackerrtc`: <br> Metrics specific to the RTC device. | `GAUGE` | | <ul><li>`item`<ul><li>`errorcount`</li><li>`missedreadcount`</li><li>`missedwritecount`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katafirecrackerseccomp`: <br> Metrics for the seccomp filtering. | `GAUGE` | | <ul><li>`item`<ul><li>`numfaults`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katafirecrackersignals`: <br> Metrics related to signals. | `GAUGE` | | <ul><li>`item`<ul><li>`sigbus`</li><li>`sigsegv`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackeruart`: <br> Metrics specific to the UART device. | `GAUGE` | | <ul><li>`item`<ul><li>`errorcount`</li><li>`flushcount`</li><li>`missedreadcount`</li><li>`missedwritecount`</li><li>`readcount`</li><li>`writecount`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackervcpu`: <br> Metrics specific to VCPUs' mode of functioning. | `GAUGE` | | <ul><li>`item`<ul><li>`exitioin`</li><li>`exitioout`</li><li>`exitmmioread`</li><li>`exitmmiowrite`</li><li>`failures`</li><li>`filtercpuid`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katafirecrackervmm`: <br> Metrics specific to the machine manager as a whole. | `GAUGE` | | <ul><li>`item`<ul><li>`deviceevents`</li><li>`paniccount`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katafirecrackervsock`: <br> VSOCK-related metrics. | `GAUGE` | | <ul><li>`item`<ul><li>`activatefails`</li><li>`cfgfails`</li><li>`conneventfails`</li><li>`connsadded`</li><li>`connskilled`</li><li>`connsremoved`</li><li>`evqueueeventfails`</li><li>`killqresync`</li><li>`muxereventfails`</li><li>`rxbytescount`</li><li>`rxpacketscount`</li><li>`rxqueueeventcount`</li><li>`rxqueueeventfails`</li><li>`rxreadfails`</li><li>`txbytescount`</li><li>`txflushfails`</li><li>`txpacketscount`</li><li>`txqueueeventcount`</li><li>`txqueueeventfails`</li><li>`txwritefails`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | Guest OS's metrics in hypervisor. | Metric name | Type | Units | Labels | Introduced in Kata version | |||||| | `kataguestcputime`: <br> Guest CPU stat. | `GAUGE` | | <ul><li>`cpu` (CPU no. and total for all CPUs)<ul><li>`0` (CPU 0)</li><li>`1` (CPU 1)</li><li>`total` (for all CPUs)</li></ul></li><li>`item` (Kernel/system statistics, from `/proc/stat`)<ul><li>`guest`</li><li>`guestnice`</li><li>`idle`</li><li>`iowait`</li><li>`irq`</li><li>`nice`</li><li>`softirq`</li><li>`steal`</li><li>`system`</li><li>`user`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `kataguestdiskstat`: <br> Disks stat in system. | `GAUGE` | | <ul><li>`disk` (disk name)</li><li>`item` (see `/proc/diskstats`)<ul><li>`discards`</li><li>`discardsmerged`</li><li>`flushes`</li><li>`inprogress`</li><li>`merged`</li><li>`reads`</li><li>`sectorsdiscarded`</li><li>`sectorsread`</li><li>`sectorswritten`</li><li>`timediscarding`</li><li>`timeflushing`</li><li>`timeinprogress`</li><li>`timereading`</li><li>`timewriting`</li><li>`weightedtimeinprogress`</li><li>`writes`</li><li>`writesmerged`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `kataguestload`: <br> Guest system load. | `GAUGE` | | <ul><li>`item`<ul><li>`load1`</li><li>`load15`</li><li>`load5`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `kataguestmeminfo`: <br> Statistics about memory usage on the system. | `GAUGE` | | <ul><li>`item` (see `/proc/meminfo`)<ul><li>`active`</li><li>`activeanon`</li><li>`activefile`</li><li>`anonhugepages`</li><li>`anonpages`</li><li>`bounce`</li><li>`buffers`</li><li>`cached`</li><li>`cmafree`</li><li>`cmatotal`</li><li>`commitlimit`</li><li>`committedas`</li><li>`directmap1G`</li><li>`directmap2M`</li><li>`directmap4M`</li><li>`directmap4k`</li><li>`dirty`</li><li>`hardwarecorrupted`</li><li>`highfree`</li><li>`hightotal`</li><li>`hugepagesfree`</li><li>`hugepagesrsvd`</li><li>`hugepagessurp`</li><li>`hugepagestotal`</li><li>`hugepagesize`</li><li>`hugetlb`</li><li>`inactive`</li><li>`inactiveanon`</li><li>`inactivefile`</li><li>`kreclaimable`</li><li>`kernelstack`</li><li>`lowfree`</li><li>`lowtotal`</li><li>`mapped`</li><li>`memavailable`</li><li>`memfree`</li><li>`memtotal`</li><li>`mlocked`</li><li>`mmapcopy`</li><li>`nfsunstable`</li><li>`pagetables`</li><li>`percpu`</li><li>`quicklists`</li><li>`sreclaimable`</li><li>`sunreclaim`</li><li>`shmem`</li><li>`shmemhugepages`</li><li>`shmempmdmapped`</li><li>`slab`</li><li>`swapcached`</li><li>`swapfree`</li><li>`swaptotal`</li><li>`unevictable`</li><li>`vmallocchunk`</li><li>`vmalloctotal`</li><li>`vmallocused`</li><li>`writeback`</li><li>`writebacktmp`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `kataguestnetdevstat`: <br> Guest net devices

| `GAUGE` | | <ul><li>`interface` (network device name)</li><li>`item` (see `/proc/net/dev`)<ul><li>`recvbytes`</li><li>`recvcompressed`</li><li>`recvdrop`</li><li>`recverrs`</li><li>`recvfifo`</li><li>`recvframe`</li><li>`recvmulticast`</li><li>`recvpackets`</li><li>`sentbytes`</li><li>`sentcarrier`</li><li>`sentcolls`</li><li>`sentcompressed`</li><li>`sentdrop`</li><li>`senterrs`</li><li>`sentfifo`</li><li>`sentpackets`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `kataguesttasks`: <br> Guest system load. | `GAUGE` | | <ul><li>`item`<ul><li>`cur`</li><li>`max`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `kataguestvmstat`: <br> Guest virtual memory stat. | `GAUGE` | | <ul><li>`item` (see `/proc/vmstat`)<ul><li>`allocstalldma`</li><li>`allocstalldma32`</li><li>`allocstallmovable`</li><li>`allocstallnormal`</li><li>`balloondeflate`</li><li>`ballooninflate`</li><li>`compactdaemonfreescanned`</li><li>`compactdaemonmigratescanned`</li><li>`compactdaemonwake`</li><li>`compactfail`</li><li>`compactfreescanned`</li><li>`compactisolated`</li><li>`compactmigratescanned`</li><li>`compactstall`</li><li>`compactsuccess`</li><li>`droppagecache`</li><li>`dropslab`</li><li>`htlbbuddyallocfail`</li><li>`htlbbuddyallocsuccess`</li><li>`kswapdhighwmarkhitquickly`</li><li>`kswapdinodesteal`</li><li>`kswapdlowwmarkhitquickly`</li><li>`nractiveanon`</li><li>`nractivefile`</li><li>`nranonpages`</li><li>`nranontransparenthugepages`</li><li>`nrbounce`</li><li>`nrdirtied`</li><li>`nrdirty`</li><li>`nrdirtybackgroundthreshold`</li><li>`nrdirtythreshold`</li><li>`nrfilepages`</li><li>`nrfreecma`</li><li>`nrfreepages`</li><li>`nrinactiveanon`</li><li>`nrinactivefile`</li><li>`nrisolatedanon`</li><li>`nrisolatedfile`</li><li>`nrkernelstack`</li><li>`nrmapped`</li><li>`nrmlock`</li><li>`nrpagetablepages`</li><li>`nrshmem`</li><li>`nrshmemhugepages`</li><li>`nrshmempmdmapped`</li><li>`nrslabreclaimable`</li><li>`nrslabunreclaimable`</li><li>`nrunevictable`</li><li>`nrunstable`</li><li>`nrvmscanimmediatereclaim`</li><li>`nrvmscanwrite`</li><li>`nrwriteback`</li><li>`nrwritebacktemp`</li><li>`nrwritten`</li><li>`nrzoneactiveanon`</li><li>`nrzoneactivefile`</li><li>`nrzoneinactiveanon`</li><li>`nrzoneinactivefile`</li><li>`nrzoneunevictable`</li><li>`nrzonewritepending`</li><li>`oomkill`</li><li>`pageoutrun`</li><li>`pgactivate`</li><li>`pgallocdma`</li><li>`pgallocdma32`</li><li>`pgallocmovable`</li><li>`pgallocnormal`</li><li>`pgdeactivate`</li><li>`pgfault`</li><li>`pgfree`</li><li>`pginodesteal`</li><li>`pglazyfree`</li><li>`pglazyfreed`</li><li>`pgmajfault`</li><li>`pgmigratefail`</li><li>`pgmigratesuccess`</li><li>`pgpgin`</li><li>`pgpgout`</li><li>`pgrefill`</li><li>`pgrotated`</li><li>`pgscandirect`</li><li>`pgscandirectthrottle`</li><li>`pgscankswapd`</li><li>`pgskipdma`</li><li>`pgskipdma32`</li><li>`pgskipmovable`</li><li>`pgskipnormal`</li><li>`pgstealdirect`</li><li>`pgstealkswapd`</li><li>`pswpin`</li><li>`pswpout`</li><li>`slabsscanned`</li><li>`swapra`</li><li>`swaprahit`</li><li>`unevictablepgscleared`</li><li>`unevictablepgsculled`</li><li>`unevictablepgsmlocked`</li><li>`unevictablepgsmunlocked`</li><li>`unevictablepgsrescued`</li><li>`unevictablepgsscanned`</li><li>`unevictablepgsstranded`</li><li>`workingsetactivate`</li><li>`workingsetnodereclaim`</li><li>`workingsetrefault`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | Hypervisors metrics, collected mainly from `proc` filesystem of hypervisor process. | Metric name | Type | Units | Labels | Introduced in Kata version | |||||| | `katahypervisorfds`: <br> Open FDs for hypervisor. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katahypervisoriostat`: <br> Process IO statistics. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/io`)<ul><li>`cancelledwritebytes`</li><li>`rchar`</li><li>`readbytes`</li><li>`syscr`</li><li>`syscw`</li><li>`wchar`</li><li>`writebytes`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katahypervisornetdev`: <br> Net devices statistics. | `GAUGE` | | <ul><li>`interface` (network device name)</li><li>`item` (see `/proc/net/dev`)<ul><li>`recvbytes`</li><li>`recvcompressed`</li><li>`recvdrop`</li><li>`recverrs`</li><li>`recvfifo`</li><li>`recvframe`</li><li>`recvmulticast`</li><li>`recvpackets`</li><li>`sentbytes`</li><li>`sentcarrier`</li><li>`sentcolls`</li><li>`sentcompressed`</li><li>`sentdrop`</li><li>`senterrs`</li><li>`sentfifo`</li><li>`sentpackets`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katahypervisorprocstat`: <br> Hypervisor process statistics. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/stat`)<ul><li>`cstime`</li><li>`cutime`</li><li>`stime`</li><li>`utime`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katahypervisorprocstatus`: <br> Hypervisor process status. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/status`)<ul><li>`hugetlbpages`</li><li>`nonvoluntaryctxtswitches`</li><li>`rssanon`</li><li>`rssfile`</li><li>`rssshmem`</li><li>`vmdata`</li><li>`vmexe`</li><li>`vmhwm`</li><li>`vmlck`</li><li>`vmlib`</li><li>`vmpeak`</li><li>`vmpin`</li><li>`vmpmd`</li><li>`vmpte`</li><li>`vmrss`</li><li>`vmsize`</li><li>`vmstk`</li><li>`vmswap`</li><li>`voluntaryctxtswitches`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katahypervisorthreads`: <br> Hypervisor process threads. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | Metrics about monitor itself. | Metric name | Type | Units | Labels | Introduced in Kata version | |||||| | `katamonitorgogcduration_seconds`: <br> A summary of the pause duration of garbage collection cycles. | `SUMMARY` | `seconds` | | 2.0.0 | | `katamonitorgo_goroutines`: <br> Number of goroutines that currently exist. | `GAUGE` | | | 2.0.0 | | `katamonitorgo_info`: <br> Information about the Go environment. | `GAUGE` | | <ul><li>`version` (golang version)<ul><li>`go1.13.9` (environment dependent variable)</li></ul></li></ul> | 2.0.0 | | `katamonitorgomemstatsalloc_bytes`: <br> Number of bytes allocated and still in use. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsallocbytestotal`: <br> Total number of bytes allocated, even if freed. | `COUNTER` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsbuckhashsys_bytes`: <br> Number of bytes used by the profiling bucket hash table. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsfrees_total`: <br> Total number of frees. | `COUNTER` | | | 2.0.0 | | `katamonitorgomemstatsgccpufraction`: <br> The fraction of this program's available CPU time used by the GC since the program started. | `GAUGE` | | | 2.0.0 | | `katamonitorgomemstatsgcsysbytes`: <br> Number of bytes used for garbage collection system metadata. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsheapallocbytes`: <br> Number of heap bytes allocated and still in use. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsheapidlebytes`: <br> Number of heap bytes waiting to be used. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsheapinusebytes`: <br> Number of heap bytes that are in use. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsheap_objects`: <br> Number of allocated objects. | `GAUGE` | | | 2.0.0 | | `katamonitorgomemstatsheapreleasedbytes`: <br> Number of heap bytes released to OS. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsheapsysbytes`: <br> Number of heap bytes obtained from system. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatslastgctime_seconds`: <br> Number of seconds since 1970 of last garbage collection. | `GAUGE` | `seconds` | | 2.0.0 | | `katamonitorgomemstatslookups_total`: <br> Total number of pointer lookups. | `COUNTER` | | | 2.0.0 | | `katamonitorgomemstatsmallocs_total`: <br> Total number of `mallocs`. | `COUNTER` | | | 2.0.0 | | `katamonitorgomemstatsmcacheinusebytes`: <br> Number of bytes in use by `mcache` structures. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsmcachesysbytes`: <br> Number of bytes used for `mcache` structures obtained from system. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsmspaninusebytes`: <br> Number of bytes in use by `mspan` structures. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsmspansysbytes`: <br> Number of bytes used for `mspan` structures obtained from

| `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsnextgcbytes`: <br> Number of heap bytes when next garbage collection will take place. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsothersysbytes`: <br> Number of bytes used for other system allocations. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsstackinusebytes`: <br> Number of bytes in use by the stack allocator. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatsstacksysbytes`: <br> Number of bytes obtained from system for stack allocator. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgomemstatssys_bytes`: <br> Number of bytes obtained from system. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorgo_threads`: <br> Number of OS threads created. | `GAUGE` | | | 2.0.0 | | `katamonitorprocesscpuseconds_total`: <br> Total user and system CPU time spent in seconds. | `COUNTER` | `seconds` | | 2.0.0 | | `katamonitorprocessmaxfds`: <br> Maximum number of open file descriptors. | `GAUGE` | | | 2.0.0 | | `katamonitorprocessopenfds`: <br> Number of open file descriptors. | `GAUGE` | | | 2.0.0 | | `katamonitorprocessresidentmemory_bytes`: <br> Resident memory size in bytes. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorprocessstarttime_seconds`: <br> Start time of the process since `unix` epoch in seconds. | `GAUGE` | `seconds` | | 2.0.0 | | `katamonitorprocessvirtualmemory_bytes`: <br> Virtual memory size in bytes. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorprocessvirtualmemorymaxbytes`: <br> Maximum amount of virtual memory available in bytes. | `GAUGE` | `bytes` | | 2.0.0 | | `katamonitorrunningshimcount`: <br> Running shim count(running sandboxes). | `GAUGE` | | | 2.0.0 | | `katamonitorscrape_count`: <br> Scape count. | `COUNTER` | | | 2.0.0 | | `katamonitorscrapedurationshistogram_milliseconds`: <br> Time used to scrape from shims | `HISTOGRAM` | `milliseconds` | | 2.0.0 | | `katamonitorscrapefailedcount`: <br> Failed scape count. | `COUNTER` | | | 2.0.0 | Metrics about Kata containerd shim v2 process. | Metric name | Type | Units | Labels | Introduced in Kata version | |||||| | `katashimagentrpcdurationshistogrammilliseconds`: <br> RPC latency distributions. | `HISTOGRAM` | `milliseconds` | <ul><li>`action` (RPC actions of Kata agent)<ul><li>`grpc.CheckRequest`</li><li>`grpc.CloseStdinRequest`</li><li>`grpc.CopyFileRequest`</li><li>`grpc.CreateContainerRequest`</li><li>`grpc.CreateSandboxRequest`</li><li>`grpc.DestroySandboxRequest`</li><li>`grpc.ExecProcessRequest`</li><li>`grpc.GetMetricsRequest`</li><li>`grpc.GuestDetailsRequest`</li><li>`grpc.ListInterfacesRequest`</li><li>`grpc.ListProcessesRequest`</li><li>`grpc.ListRoutesRequest`</li><li>`grpc.MemHotplugByProbeRequest`</li><li>`grpc.OnlineCPUMemRequest`</li><li>`grpc.PauseContainerRequest`</li><li>`grpc.RemoveContainerRequest`</li><li>`grpc.ReseedRandomDevRequest`</li><li>`grpc.ResumeContainerRequest`</li><li>`grpc.SetGuestDateTimeRequest`</li><li>`grpc.SignalProcessRequest`</li><li>`grpc.StartContainerRequest`</li><li>`grpc.StatsContainerRequest`</li><li>`grpc.TtyWinResizeRequest`</li><li>`grpc.UpdateContainerRequest`</li><li>`grpc.UpdateInterfaceRequest`</li><li>`grpc.UpdateRoutesRequest`</li><li>`grpc.WaitProcessRequest`</li><li>`grpc.WriteStreamRequest`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimfds`: <br> Kata containerd shim v2 open FDs. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgogcdurationseconds`: <br> A summary of the pause duration of garbage collection cycles. | `SUMMARY` | `seconds` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgogoroutines`: <br> Number of goroutines that currently exist. | `GAUGE` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgoinfo`: <br> Information about the Go environment. | `GAUGE` | | <ul><li>`sandboxid`</li><li>`version` (golang version)<ul><li>`go1.13.9` (environment dependent variable)</li></ul></li></ul> | 2.0.0 | | `katashimgomemstatsallocbytes`: <br> Number of bytes allocated and still in use. | `GAUGE` | `bytes` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgomemstatsallocbytestotal`: <br> Total number of bytes allocated, even if freed. | `COUNTER` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsbuckhashsysbytes`: <br> Number of bytes used by the profiling bucket hash table. | `GAUGE` | `bytes` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgomemstatsfreestotal`: <br> Total number of frees. | `COUNTER` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgomemstatsgccpufraction`: <br> The fraction of this program's available CPU time used by the GC since the program started. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsgcsysbytes`: <br> Number of bytes used for garbage collection system metadata. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsheapallocbytes`: <br> Number of heap bytes allocated and still in use. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsheapidlebytes`: <br> Number of heap bytes waiting to be used. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsheapinusebytes`: <br> Number of heap bytes that are in

| `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsheapobjects`: <br> Number of allocated objects. | `GAUGE` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgomemstatsheapreleasedbytes`: <br> Number of heap bytes released to OS. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsheapsysbytes`: <br> Number of heap bytes obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatslastgctimeseconds`: <br> Number of seconds since 1970 of last garbage collection. | `GAUGE` | `seconds` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgomemstatslookupstotal`: <br> Total number of pointer lookups. | `COUNTER` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgomemstatsmallocstotal`: <br> Total number of `mallocs`. | `COUNTER` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgomemstatsmcacheinusebytes`: <br> Number of bytes in use by `mcache` structures. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsmcachesysbytes`: <br> Number of bytes used for `mcache` structures obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsmspaninusebytes`: <br> Number of bytes in use by `mspan` structures. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsmspansysbytes`: <br> Number of bytes used for `mspan` structures obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsnextgcbytes`: <br> Number of heap bytes when next garbage collection will take place. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsothersysbytes`: <br> Number of bytes used for other system allocations. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsstackinusebytes`: <br> Number of bytes in use by the stack allocator. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatsstacksysbytes`: <br> Number of bytes obtained from system for stack allocator. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimgomemstatssysbytes`: <br> Number of bytes obtained from system. | `GAUGE` | `bytes` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimgothreads`: <br> Number of OS threads created. | `GAUGE` | | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimiostat`: <br> Kata containerd shim v2 process IO statistics. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/io`)<ul><li>`cancelledwritebytes`</li><li>`rchar`</li><li>`readbytes`</li><li>`syscr`</li><li>`syscw`</li><li>`wchar`</li><li>`writebytes`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimnetdev`: <br> Kata containerd shim v2 network devices statistics. | `GAUGE` | | <ul><li>`interface` (network device name)</li><li>`item` (see `/proc/net/dev`)<ul><li>`recvbytes`</li><li>`recvcompressed`</li><li>`recvdrop`</li><li>`recverrs`</li><li>`recvfifo`</li><li>`recvframe`</li><li>`recvmulticast`</li><li>`recvpackets`</li><li>`sentbytes`</li><li>`sentcarrier`</li><li>`sentcolls`</li><li>`sentcompressed`</li><li>`sentdrop`</li><li>`senterrs`</li><li>`sentfifo`</li><li>`sentpackets`</li></ul></li><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimpodoverheadcpu`: <br> Kata Pod overhead for CPU resources(percent). | `GAUGE` | percent | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimpodoverheadmemoryinbytes`: <br> Kata Pod overhead for memory resources(bytes). | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimprocstat`: <br> Kata containerd shim v2 process statistics. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/stat`)<ul><li>`cstime`</li><li>`cutime`</li><li>`stime`</li><li>`utime`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimprocstatus`: <br> Kata containerd shim v2 process status. | `GAUGE` | | <ul><li>`item` (see `/proc/<pid>/status`)<ul><li>`hugetlbpages`</li><li>`nonvoluntaryctxtswitches`</li><li>`rssanon`</li><li>`rssfile`</li><li>`rssshmem`</li><li>`vmdata`</li><li>`vmexe`</li><li>`vmhwm`</li><li>`vmlck`</li><li>`vmlib`</li><li>`vmpeak`</li><li>`vmpin`</li><li>`vmpmd`</li><li>`vmpte`</li><li>`vmrss`</li><li>`vmsize`</li><li>`vmstk`</li><li>`vmswap`</li><li>`voluntaryctxtswitches`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimprocesscpusecondstotal`: <br> Total user and system CPU time spent in seconds. | `COUNTER` | `seconds` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimprocessmaxfds`: <br> Maximum number of open file descriptors. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimprocessopenfds`: <br> Number of open file descriptors. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimprocessresidentmemorybytes`: <br> Resident memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimprocessstarttimeseconds`: <br> Start time of the process since `unix` epoch in seconds. | `GAUGE` | `seconds` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimprocessvirtualmemorybytes`: <br> Virtual memory size in bytes. | `GAUGE` | `bytes` | <ul><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimprocessvirtualmemorymaxbytes`: <br> Maximum amount of virtual memory available in bytes. | `GAUGE` | `bytes` | <ul><li>`sandbox_id`</li></ul> | 2.0.0 | | `katashimrpcdurationshistogrammilliseconds`: <br> RPC latency distributions. | `HISTOGRAM` | `milliseconds` | <ul><li>`action` (Kata shim v2 actions)<ul><li>`checkpoint`</li><li>`closeio`</li><li>`connect`</li><li>`create`</li><li>`delete`</li><li>`exec`</li><li>`kill`</li><li>`pause`</li><li>`pids`</li><li>`resizepty`</li><li>`resume`</li><li>`shutdown`</li><li>`start`</li><li>`state`</li><li>`stats`</li><li>`update`</li><li>`wait`</li></ul></li><li>`sandboxid`</li></ul> | 2.0.0 | | `katashimthreads`: <br> Kata containerd shim v2 process threads. | `GAUGE` | | <ul><li>`sandbox_id`</li></ul> | 2.0.0 |