Project: Opstrace (Observability and Analysis / Observability)
File: instrumenting-a-k8s-cluster.md

This guide will show how you can monitor a (k8s) cluster using your Opstrace cluster. Specifically, the goal is to scrape k8s system metrics, and push them to the Opstrace cluster. collect logs of workloads (containers) running on the k8s cluster, and send them to the Opstrace cluster. To that end, we will deploy a single Prometheus instance in the k8s cluster (as a k8s deployment): it will scrape the various k8s system metric endpoints, and then push all collected data to the remote Opstrace cluster. we will deploy a Promtail instance on each node in the k8s cluster (as a k8s daemon set): it will collect all local container logs and push them into the remote Opstrace cluster. An Opstrace cluster. A decision: for which Opstrace tenant would you like to send data? An Opstrace tenant authentication token file (for the tenant of your choice). Also see . For following this guide step-by-step you will additionally need installed on your computer. `Kind` is a tool for running a local k8s cluster using Docker. ```bash kind create cluster ``` ```bash kubectl cluster-info --context kind-kind ``` Copy the `default` tenant's data API authentication token. Create a new file and name it `secret.yaml` ```yaml apiVersion: v1 kind: Secret metadata: name: tenant-auth-token-default stringData: authToken: <YOUR AUTH TOKEN HERE> ``` Create the secret: ```bash kubectl apply -f secret.yaml ``` Create a file named `prometheus-config.yaml` with the following contents. Replace `${CLUSTER_NAME}` with the name of your Opstrace cluster. Note that the configuration snippets below assume that the Opstrace tenant you would like to send data to is called `default`. ```yaml kind: ConfigMap apiVersion: v1 metadata: name: prometheus-config data: prometheus.yml: |- remote_write: url: https://cortex.${TENANTNAME}.${CLUSTERNAME}.opstrace.io/api/v1/push bearertokenfile: /var/run/${TENANT_NAME}-tenant/authToken scrape_configs: job_name: 'kubernetes-pods' kubernetessdconfigs: role: pod tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: sourcelabels: [metakubernetesservicelabel_component] regex: apiserver action: replace targetlabel: scheme_ replacement: https sourcelabels: [metakubernetesnamespace, metakubernetespodlabel_name] action: replace separator: / target_label: job replacement: $1 sourcelabels: [metakubernetespodname] action: replace target_label: instance sourcelabels: [metakubernetespodnode_name] target_label: node job_name: 'kubernetes-nodes' kubernetessdconfigs: role: node tls_config: insecureskipverify: true bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: targetlabel: scheme_ replacement: https sourcelabels: [metakubernetesnodelabelkubernetesio_hostname] target_label: instance job_name: 'kubernetes-service' kubernetessdconfigs: role: endpoints tls_config: insecureskipverify: true bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: sourcelabels: [metakubernetesservicelabel_component] regex: apiserver action: keep targetlabel: scheme_ replacement: https source_labels: [] target_label: job replacement: default/kubernetes ``` Submit this config map: ```bash kubectl apply -f prometheus-config.yaml ``` Next up, create a file named `prometheus.yaml` ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: apiGroups: [""] resources: nodes nodes/proxy services endpoints pods verbs: ["get", "list", "watch"] nonResourceURLs: ["/metrics"] verbs: ["get"] apiVersion: v1 kind: ServiceAccount metadata: name: prometheus apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup:

kind: ClusterRole name: prometheus subjects: kind: ServiceAccount name: prometheus namespace: default apiVersion: apps/v1 kind: Deployment metadata: name: prometheus spec: replicas: 1 selector: matchLabels: name: prometheus template: metadata: labels: name: prometheus spec: serviceAccountName: prometheus containers: name: retrieval image: prom/prometheus:v2.21.0 imagePullPolicy: IfNotPresent args: --config.file=/etc/prometheus/prometheus.yml ports: containerPort: 9090 volumeMounts: name: config-volume mountPath: /etc/prometheus name: tenant-auth-token-default mountPath: /var/run/default-tenant readOnly: true volumes: name: config-volume configMap: name: prometheus-config name: tenant-auth-token-default secret: secretName: tenant-auth-token-default ``` Then start the Prometheus deployment with the following command: ```bash kubectl apply -f prometheus.yaml ``` Create a file named `promtail-config.yaml` with the following contents. Replace `${CLUSTER_NAME}` with the name of your Opstrace cluster. Note that the configuration snippets below assume that the Opstrace tenant you would like to send data to is called `default`. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: promtail-config data: promtail.yml: | client: url: https://loki.default.#{CLUSTER_NAME}.opstrace.io/loki/api/v1/push bearertokenfile: /var/run/default-tenant/authToken scrape_configs: pipeline_stages: docker: job_name: kubernetes-pods-name kubernetessdconfigs: role: pod relabel_configs: source_labels: metakubernetespodlabelname targetlabel: service_ source_labels: metakubernetespodnodename targetlabel: host_ action: drop regex: ^$ source_labels: service action: replace replacement: $1 separator: / source_labels: metakubernetesnamespace service targetlabel: k8sapp action: replace source_labels: metakubernetesnamespace targetlabel: k8snamespace_name action: replace source_labels: metakubernetespod_name targetlabel: k8spod_name action: replace source_labels: metakubernetespodcontainername targetlabel: k8scontainer_name replacement: /var/log/pods/$1/.log separator: / source_labels: metakubernetespod_uid metakubernetespodcontainername targetlabel: path_ replacement: /var/log/pods/$1/.log separator: / source_labels: metakubernetespod_uid metakubernetespodcontainername targetlabel: path_ pipeline_stages: docker: job_name: kubernetes-pods-static kubernetessdconfigs: role: pod relabel_configs: action: drop regex: ^$ source_labels: metakubernetespodannotationkubernetesioconfig_mirror action: replace source_labels: metakubernetespodlabelcomponent targetlabel: service_ source_labels: metakubernetespodnodename targetlabel: host_ action: drop regex: ^$ source_labels: service action: replace replacement: $1 separator: / source_labels: metakubernetesnamespace service targetlabel: k8sapp action: replace source_labels: metakubernetesnamespace targetlabel: k8snamespace_name action: replace source_labels: metakubernetespod_name targetlabel: k8spod_name action: replace source_labels: metakubernetespodcontainername targetlabel: k8scontainer_name replacement: /var/log/pods/$1/.log separator: / source_labels: metakubernetespodannotationkubernetesioconfig_mirror metakubernetespodcontainername targetlabel: path_ ``` Submit this config map: ```bash kubectl apply -f promtail-config.yaml ``` Create a file `promtail.yaml` with the following contents: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: promtail spec: minReadySeconds: 10 selector: matchLabels: name: promtail template: metadata: labels: name: promtail spec: containers: args: -config.file=/etc/promtail/promtail.yml env: name: HOSTNAME valueFrom: fieldRef: fieldPath: spec.nodeName image: grafana/promtail:1.6.1 imagePullPolicy: Always name: promtail readinessProbe: httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 10 ports: containerPort: 80 name: http-metrics securityContext: privileged: true runAsUser: 0 volumeMounts: mountPath: /etc/promtail name: promtail-config mountPath: /var/log name: varlog mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true mountPath: /var/run/default-tenant name: tenant-auth-token-default readOnly: true serviceAccount: promtail tolerations: effect: NoSchedule operator: Exists volumes: configMap: name: promtail-config name: promtail-config secret: secretName: tenant-auth-token-default name: tenant-auth-token-default hostPath: path: /var/log name: varlog hostPath: path: /var/lib/docker/containers name: varlibdockercontainers updateStrategy: type: RollingUpdate apiVersion: v1 kind: ServiceAccount metadata: name: promtail apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: promtail rules: apiGroups: "" resources: nodes nodes/proxy services endpoints pods verbs: get list watch apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: promtail roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: promtail subjects: kind: ServiceAccount name: promtail namespace: default ``` Start the Promtail deployment with the following command: ```bash kubectl apply -f promtail.yaml ``` You can explore the logs and metrics sent to your Opstrace cluster using the Grafana explore view: ```bash open https://default.${CLUSTER_NAME}.opstrace.io/grafana/explore ```