Project: Teleport (Provisioning / Security & Compliance)
File: 0108-agent-census.md

authors: Vitor Enes (vitor@goteleport.com) state: implemented (v11.3.8, v12.1.1) Engineering: @zmb3 && @jimbishopp Product: @xin || @klizhentas Security: @wadells This RFD details how we'll track more information about agents (aka Agent Census). A brief description of this task can be found in . Track more information about each Teleport agent (such as OS, OS version, architecture, installation methods, container runtime and others) Detail how this information will be analyzed / visualized. We want to understand how agents are installed and where they are running so that we can prioritize the work around . Service: A Teleport service manages access to resources such as SSH nodes, kubernetes clusters, internal web applications, databases, and windows desktops. Agent: A `teleport` process that runs one or more Teleport services (depending on the configuration). PreHog: A microservice used to capture user events across several Teleport tools. This section is divided in the following subsections: : which data about each agent will be tracked : how such data will flow from the agents to PreHog : how to compute such data We want to start tracking the following data in PreHog: Teleport version Teleport enabled services (`node`, `kube`, `app`, `db` and `windows_desktop`) OS (`linux` or `darwin`, as these are the only two OS currently supported) OS version (e.g. Linux distribution) Host architecture (e.g. `amd64`) `glibc` version (Linux only) (Dockerfile, Helm, `install-node.sh`) Container runtime (e.g. Docker) Container orchestrator (e.g. Kubernetes) Cloud environment (e.g. AWS, GCP, Azure) Currently, when an agent first starts, the (ICS) sends an message to the auth server. This message has the following fields: ```protobuf message UpstreamInventoryHello { string Version = 1; string ServerID = 2; repeated string Services = 3 [(gogoproto.casttype) = "github.com/gravitational/teleport/api/types.SystemRole"]; string Hostname = 4; } ``` The `Version` field contains the Teleport version, while the `Services` field contains the subset of the system roles that are currently active at the agent. While initially we considered extending this message to contain all the agent metadata we want track, we decided to instead add a new message type `UpstreamInventoryAgentMetadata` (see the message definition below). Some of the agent metadata may be slow to compute (due to HTTP requests), and thus blocking the sending of the `UpstreamInventoryHello` until such metadata is computed could potentially increase the agent start-up/connection time. Instead, when the auth server handle is created at the agent (), a new goroutine will be spawned in order to fetch the agent metadata in the background and send it every time a new stream with the auth server is established. ```protobuf // UpstreamInventoryAgentMetadata is the message sent up the inventory control stream containing // metadata about the instance. message UpstreamInventoryAgentMetadata { // OS advertises the instance OS ("darwin" or "linux"). string OS = 1; // OSVersion advertises the instance OS version (e.g. "ubuntu 22.04"). string OSVersion = 2; // HostArchitecture advertises the instance host architecture (e.g. "x86_64" or "arm64"). string HostArchitecture = 3; // GlibcVersion advertises the instance glibc version of linux instances (e.g. "2.35"). string GlibcVersion = 4; // InstallMethods advertises the install methods used for the instance (e.g. "dockerfile"). repeated string InstallMethods = 5; // ContainerRuntime advertises the container runtime for the instance, if any (e.g.

string ContainerRuntime = 6; // ContainerOrchestrator advertises the container orchestrator for the instance, if any // (e.g. "kubernetes-v1.24.8-eks-ffeb93d"). string ContainerOrchestrator = 7; // CloudEnvironment advertises the cloud environment for the instance, if any (e.g. "aws"). string CloudEnvironment = 8; } ``` When the auth server receives an `UpstreamInventoryAgentMetadata` message, it will take the information in the message and send it to PreHog. For this, a new PreHog `AgentMetadataEvent` message will be added (note that only the `UpstreamInventoryHello.Hostname` won't be sent to PreHog as it can contain PII but also because it doesn't seem useful): ```protobuf message AgentMetadataEvent { string version = 1; string host_id = 2; repeated string services = 3; string os = 4; string os_version = 5; string host_architecture = 6; string glibc_version = 7; repeated string install_methods = 8; string container_runtime = 9; string container_orchestrator = 10; string cloud_environment = 11; } ``` Some of the fields above are `repeated`. In PostHog, instead of storing these field values as arrays, we will create one event property for each element in the array (which will likely help visualizing this information in PostHog). If, for example, `AgentMetadataEvent.services` contains both `node` and `kube`, in PostHog we'll have the following three properties: `tp.agent.services = [node, kube]` `tp.agent.service.node = true` `tp.agent.service.kube = true` The same applies for `AgentMetadataEvent.install_methods`. Both the Teleport version and active Teleport services are already tracked in the ICS. We detail below how the remaining data will be computed. `UpstreamInventoryAgentMetadata.OS` will be set to the value on the `GOOS` environment variable. This will give us either `darwin` or `linux` as they are the only two supported OS for now. On `darwin`, `UpstreamInventoryAgentMetadata.OSVersion` will be set to the outcome of (something equivalent to) `$(swvers -productName) $(swvers -productVersion)` (e.g. `"macOS 13.2"`). This is what `gopsutil` is doing (). On `linux`, we'll inspect `/etc/os-release` and combine the values associated with `"NAME="` and `"VERSION_ID="` (e.g. "Ubuntu 22.04"). If this file does not exist (unlikely, as it seems widely supported), we can fallback to `/etc/lsb-release` and combine the values associated with `"DISTRIBID="` and `"DISTRIBRELEASE="` (which is what `gopsutil` is doing ()). Following this approach is more reliable than using `/usr/bin/lsbrelease` directly as it is not always available (e.g. `docker run -ti ubuntu:22.04 lsbrelease` fails). `UpstreamInventoryAgentMetadata.HostArchitecture` will be set to the value on the `GOARCH` environment variable. In the future we may use `sysctl -n sysctl.proc_translated` in order to detect if a macOS agent is running under Rosetta. If on `linux`, `UpstreamInventoryAgentMetadata.GLibCVersion` will be set to the output of `gnugetlibc_version`. ```go // #include <gnu/libc-version.h> import "C" func fetchGlibcVersion() string { return C.GoString(C.gnugetlibc_version()) } ``` Different installation methods will be tracked by setting new `TELEPORTINSTALLMETHOD_$NAME` environment variables to `true` (where `$NAME` is the installation method). We have one environment variable for each installation method as some of the installation methods below may occur at the same time (e.g. `Dockerfile` and `teleport-kube-agent`, or `install-node.sh` and `APT` and `systemctl`). : `ENV TELEPORTINSTALLMETHOD_DOCKERFILE=true` will be added to the Dockerfile. Helm chart: `TELEPORTINSTALLMETHODHELMKUBE_AGENT` will be set to `true` in the . : `export TELEPORTINSTALLMETHODNODESCRIPT="true"` will be added to this script. It is the recommended way to install SSH nodes, apps and many databases. Even though `export` doesn't persist across restarts, we can have the agent persist such value (and maybe all of the values sent in `UpstreamInventoryAgentMetadata`) when it first

`systemctl`: Tracking whether the agent is running using `systemctl` does not require a new environment variable. For this, we'll simply check if `systemctl status teleport.service` succeeds and, if so, if it contains the string `"active (running)"`. The installation methods that follow won't be tracked for now. Later on, we may try to track these if, once we start tracking the above installation methods, we notice that we're not yet covering most methods. tarball: We can add `export TELEPORTINSTALLMETHOD_TARBALL="true"` to the script. (However, if the customer does not use the `install` script and instead moves the binaries manually, we won't be able to track this installation method.) `.deb`/`.rpm`/`.pkg` packages, APT or YUM repository, and : It's unclear ATM how these can be tracked. built from source: While it's technically possible for customers to build Teleport from source, we won't try to track this installation method as it seems an unlikely use-case. `homebrew`: It's also possible to install Teleport on macOS using `homebrew`. The Teleport package in `homebrew` is not maintained by us, so we will also not track this installation method. In summary, we'll have the following values in `UpstreamInventoryAgentMetadata.InstallMethods` for now: `dockerfile` `helmkubeagent` `node_script` `systemctl` To determine if the agent is running on Docker, we'll check if the file `/.dockerenv` exists. (Docker itself ). If so, `UpstreamInventoryAgentMetadata.ContainerRuntime` will be set to `docker`. If we're interested in tracking other container runtimes, we could follow the approach by `gopsutil` (). To determine if the agent is running on a Kubernetes pod, we can try to initialize a Kubernetes `client` similar to how does it. If this succeeds, the agent is running on Kubernetes. Afterwards, we'll try to detect in which cloud provider the pod is running on. For this, we'll call `client.ServerVersion()`: in EKS, the git version looks like `"v1.24.8-eks-ffeb93d"` (i.e. contains the substring `"-eks"`) in GPC (), the git version looks like `"1.23.14-gke.1800"` (i.e. contains the substring `"-gke"`) in AKS, the git version looks like `"v1.25.2"`, so it's not possible to detect this environment using this method. (This is also a problem for Helm charts, as reported in .) In the end, `UpstreamInventoryAgentMetadata.ContainerOrchestrator` will be set to `kubernetes-$GIT_VERSION`. Initially we considered setting `UpstreamInventoryAgentMetadata.ContainerOrchestrator` to `kubernetes-eks` if on EKS, `kubernetes-gcp` if on GCP and `kubernetes-unknown` otherwise. However, this will require changing the agent code in order to track AKS (if at some point they decide to include the substring `"-aks"`) or some other container orchestrator that can also be detected using the git version. The only way to determine this seems to be by hitting certain HTTP endpoints specific to each cloud environment: AWS (): http://169.254.169.254/latest/ GCP (): http://metadata.google.internal/computeMetadata/v1/ Azure (): http://169.254.169.254/metadata/instance?api-version=2021-02-01 `UpstreamInventoryAgentMetadata.CloudEnvironment` will be set to: `aws` if on AWS `gcp` if on GCP `azure` if on Azure Detecting the 9. Container orchestrator and 10. Cloud environment requires hitting certain HTTP endpoints. This may be considered too intrusive, so we have to make a decision on whether we really want to track it and argue why it's okay to do so. The host ID will be anonymized as it may not be just a UUID. Nothing special is done regarding sanitization. This will be tackled more holistically in a follow-up project. Data analysis and visualization are not a goal for this RFD, so no UX concerns for now.