Project: Vector (Observability and Analysis / Observability)
File: kubernetes.md

title: Install Vector on Kubernetes short: Kubernetes weight: 2 {{< requirement title="Minimum Kubernetes version" >}} Vector must be installed on Kubernetes version 1.15 or higher. Vector is tested with Kubernetes versions 1.19 or higher. {{< /requirement >}} [Kubernetes], also known as k8s, is an open source container orchestration system for automating application deployment, scaling, and management. This page covers installing and managing Vector on the Kubernetes platform. You can install Vector on Kubernetes using , or {{< jump "/docs/setup/installation/package-managers/helm" >}} to install Vector on Kubernetes The instructions below are for installing Vector in the [Agent] and [Aggregator] roles. The Vector [Agent] lets you collect data from your [sources] and then deliver it to a variety of destinations with [sinks]. We recommend running Vector in its own Kubernetes namespace. In the instructions here we'll use `vector` as a namespace but you're free to choose your own. ```shell kubectl create namespace --dry-run=client -o yaml vector > namespace.yaml ``` This example configuration file deploys Vector as an Agent, the full default configuration can be found . For more information about configuration options, see the [configuration] docs page. ```shell cat <<-'KUSTOMIZATION' > kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: vector bases: github.com/vectordotdev/vector/distribution/kubernetes/vector-agent?ref=master images: name: timberio/vector newName: timberio/vector newTag: {{< version >}}-distroless-libc resources: namespace.yaml KUSTOMIZATION ``` ```shell kubectl kustomize ``` ```shell kubectl apply -k . ``` ```shell kubectl logs -n vector daemonset/vector ``` The Vector [Aggregator] lets you [transform] and ship data collected by other agents. For example, it can insure that the data you are collecting is scrubbed of sensitive information, properly formatted for downstream consumers, sampled to reduce volume, and more. We recommend running Vector in its own Kubernetes namespace. In the instructions here we'll use `vector` as a namespace but you're free to choose your own. ```shell kubectl create namespace --dry-run=client -o yaml vector > namespace.yaml ``` This example configuration deploys Vector as an Aggregator, the full configuration can be found . For more information about configuration options, see the [Configuration] docs page. ```shell cat <<-'KUSTOMIZATION' > kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: vector bases: github.com/vectordotdev/vector/tree/master/distribution/kubernetes/vector-aggregator images: name: timberio/vector newName: timberio/vector newTag: {{< version >}}-distroless-libc resources: namespace.yaml KUSTOMIZATION ``` ```shell kubectl kustomize ``` ```shell kubectl apply -k . ``` ```shell "kubectl logs -n vector statefulset/vector" ``` The is community supported resource. The operator deploys and configures a Vector Agent as a DaemonSet on every Node to collect container and application logs from the Node's file system. For additional information, see the . Vector is an end-to-end observability data pipeline designed to deploy under various

You mix and match these roles to create topologies. The intent is to make Vector as flexible as possible, allowing you to fluidly integrate Vector into your infrastructure over time. The deployment section demonstrates common Vector pipelines: {{< jump "/docs/setup/deployment/topologies" >}} Vector checkpoints the current read position after each successful read. This ensures that Vector resumes where it left off when it's restarted, which prevents data from being read twice. The checkpoint positions are stored in the data directory which is specified via the global option, but can be overridden via the `data_dir` option in the file source directly. The can skip the logs from the individual `container`s of a particular Pod. Add an annotation `vector.dev/exclude-containers` to the Pod and enumerate the names of all the containers to exclude in the value of the annotation like so: ```yaml vector.dev/exclude-containers: "container1,container2" ``` This annotation makes Vector skip logs originating from the `container1` and `container2` of the Pod marked with the annotation, while logs from other containers in the Pod are collected. By default, the source augments events with helpful content keys, as explained in the section. Vector enriches data with Kubernetes context. You can find a comprehensive list of fields in the . Vector provides rich filtering options for Kubernetes log collection: Built-in Pod and container exclusion rules The `excludepathsglob_patterns` option enables you to exclude Kubernetes log files by filename and path. The `extrafieldselector` option specifies the field selector to filter Pods with, to be used in addition to the built-in `Node` filter. The `extralabelselector` option specifies the label selector filter Pods with, to be used in addition to the built-in . The `extranamespacelabel_selector` option specifies the label selector filter Namespaces with, to be used in addition to the built-in . Vector requires access to the Kubernetes API. Specifically, the source uses the `/api/v1/pods` endpoint to "watch" Pods from all namespaces. Modern Kubernetes clusters run with a role-based access control (RBAC) scheme. RBAC-enabled clusters require some configuration to grant Vector the authorization to access Kubernetes API endpoints. As RBAC is currently the standard way of controlling access to the Kubernetes API, we ship the necessary configuration out of the box. See `ClusterRole`, `ClusterRoleBinding`, and a `ServiceAccount` in our kubectl YAML config and the `rbac` configuration in the Helm chart. If your cluster doesn't use any access control scheme and doesn't restrict access to the Kubernetes API, you don't need to provide any extra configuration, as Vector should just

Clusters using a legacy ABAC scheme aren't officially supported, although Vector might work if you configure access properly. We encourage you to switch to RBAC. If you use a custom access control scheme, make sure that Vector is granted access to the `/api/v1/pods` resource. Vector communicates with the Vector API to enrich the data it collects with Kubernetes context. In order to do that, Vector needs access to the . If Vector is running in a Kubernetes cluster, Vector connects to that cluster using the . In addition to access, Vector implements proper desync handling to ensure that communication is safe and reliable. This ensures that Vector doesn't overwhelm the Kubernetes API or compromise its stability. Vector's Helm chart deployments provide quality of life around setup and maintenance of metrics pipelines in Kubernetes. Each of the Helm charts provides an `internalmetrics` source and `prometheus` sink out of the box. Agent deployments also expose `hostmetrics` via the same `prometheus` sink. Charts come with options to enable Prometheus integration via annotations or Prometheus Operator integration via PodMonitor. The Prometheus `nodeexporter` agent isn't required when the `hostmetrics` source is enabled. By default, Vector merges partial messages that are split due to the Docker size limit. For everything else, we recommend that you use the , which enables you to handle custom merging of things like stacktraces. By default, the skips logs from Pods that have a `vector.dev/exclude: "true"` label. You can configure additional exclusion rules via label or field selectors. See the . To ensure that all data is collected, Vector continues to collect logs from Pods for some time after their removal. This ensures that Vector obtains some of the most important data, such as crash details. We recommend the resource limits listed below when running Vector on Kubernetes. If you deploy Vector as an ), we recommend the following limits: ```yaml resources: requests: memory: "64Mi" cpu: "500m" limits: memory: "1024Mi" cpu: "6000m" ``` {{< info >}} As with all Kubernetes resource limit recommendations, use these as a reference point and adjust as necessary. If your configuration Vector pipeline is complex, you may need more resources; if you have a simple pipeline, you may need less. {{< /info >}} The component is stateless, which means that its behavior is consistent across each input. For the [Agent] role, Vector stores its state in the host-mapped directory with a static path. If it's redeployed, it's able to continue from where it was interrupted. Vector is tested extensively against Kubernetes. In addition to Kubernetes being Vector's most popular installation method, Vector implements a comprehensive end-to-end test suite for all minor Kubernetes versions beginning with 1.14.