Project: Vald (App Definition and Development / Database)
File: observability-configuration.md

Observability for the Vald cluster helps to check each Vald component status by monitoring metrics, logs, and traces. By enabling observability, you can monitor and visualize the number of indexes, the timing of creating indexes, the number of RPC, CPU resource usage, Memory resource usage, events, etc. This page shows the best practice for applying observing features by constructing an observability environment and setting the Vald Helm chart. Vald conforms and does NOT depend on any commercial data format. <img src="../../assets/docs/guides/observability-configuration/architecture.png" /> OpenTelemetry Collector handles receiving / processing / exporting the telemetry data, which does NOT depend on the vendors' format. All Vald components can send OTLP-compliant telemetry data, like metrics, traces, or logs, to the OpenTelemerty Collector. The telemetry data is used to monitor or visualize with the observing tools. The recommended observability environment consists of the following: Cert-Manager Jaeger Operator and Jaeger Prometheus Operator and Prometheus Grafana OpenTelemetry Operator and OpenTelemtry Collector Vald provides and for deploying those components. Please refer to the following sections to deploy each component. <div class="notice"> You can change parameters such as the export host by editing the manifest for your demands. </div> A `Cert-Manager` is required to enable to use of operators on the Kubernetes Cluster. ```bash make k8s/external/cert-manager/deploy ``` Jaeger is helpful for monitoring trace data. It can be available by running the following: ```bash make k8s/metrics/jaeger/deploy ``` Vald recommends using Prometheus as a backend service by the following command to monitor the metrics data. ```bash make k8s/metrics/prometheus/operator/deploy ``` Vald recommends using Grafana to visualize metrics data. ```bash make k8s/metrics/grafana/deploy ``` Vald uses OpenTelemetry Collector to get the telemetry data and export it to monitor backend components. The following command deploys OpenTelemtry Collector via OpenTelemerty Collector. <div class="warn"> Before executing the following command, please ensure the Prometheus operator runs healthy. </div> ```bash make k8s/otel/operator/deploy k8s/otel/collector/deploy ``` If you would like to deploy observability components with a simple command, please use the following command. ```bash make k8s/external/cert-manager/deploy k8s/monitoring/deploy ``` This section shows how to set `values.yaml` to enable each Vald component to send its own telemetry data. The setting points are the following: Enable observability feature Enable sending system metrics Enable sending trace data Set OpenTelemetry parameters <div class="notice"> The general settings are described for convenience, but individual settings are possible for each component by editing

</div> To enable the observability feature, you must set `defaults.observability.enabled` as `true`. ```yaml defaults: observability: enabled: true ``` Each Vald component sends the system metrics by editing `defaults.observability.metrics`. All metrics are enabled by default. Vald recommends using default values unless there is a specific reason. ```yaml defaults: observability: metrics: enableversioninfo: true versioninfolabels: "vald_version" "server_name" "git_commit" "build_time" "go_version" "go_os" "go_arch" "algorithm_info" enable_memory: true enable_goroutine: true enable_cgo: true ``` Please set `defaults.observability.trace.enabled` as `true` to enable sending trace data. The default value is `false`. ```yaml defaults: observability: trace: enabled: true ``` This section shows the detailed settings for sending telemetry data. To send the telemetry data, you must set the OpenTelemetry Collector's endpoint to `defaults.observability.otlp.collector_endpoint`. The default value is not set. <div class="warn"> It is required to send telemetry data. </div> ```yaml defaults: observability: otlp: collector_endpoint: "opentelemetry-collector-collector.default.svc.cluster.local:4317" ``` You can add the component information to the attribute of telemetry data by editing `defaults.observability.otlp.attirbute`. E.g., when setting `vald-agent-ngt-0` as `agent.observability.otlp.attribute.podname`, `targetpod: vald-agent-ngt-0` will be added to the attribute. These attributes are set auto by the environment values, so Vald recommends using default values unless there is a specific reason. ```yaml defaults: observability: otlp: attribute: namespace: vald pod_name: vald-agent-ngt-0 node_name: kube-worker01 service_name: vald-agent-ngt ``` You can modify the sending telemetry data behavior by changing the `default.observability.otlp` parameters. ```yaml defaults: observability: otlp: tracebatchtimeout: "1s" traceexporttimeout: "1m" tracemaxexportbatchsize: 1024 tracemaxqueue_size: 256 metricsexportinterval: "1s" metricsexporttimeout: "1m" ``` The interceptor configuration is required to send the metrics and trace data related to gRPC. You can add the interceptor to the server-side and client-side by editing `defaults.server_config.servers.grpc.server.grpc.interceptors` parameters. ```yaml defaults: server_config: servers: grpc: server: grpc: interceptors: TraceInterceptor MetricInterceptor grpc: client: dial_option: interceptors: TraceInterceptor ``` Telemetry data can be monitored using Grafana, Jaeger, etc. Vald publishes , which enables Grafana and Jaeger. You can apply it after creating an observability environment. <div class="notice"> The default manifests don't set ingress host.<BR> You can monitor the browser by port forwarding or define the ingress host by yourself. </div> Lastly, the Vald cluster will be removed by executing the following command. ```bash helm uninstall vald ``` Also, the observability components will be removed by executing the following command. ```bash make k8s/monitoring/delete k8s/external/cert-manager/delete ``` And the observability components can also be removed by executing the following command. ```bash make k8s/otel/collector/delete \ k8s/otel/operator/delete \ k8s/metrics/grafana/delete \ k8s/metrics/jaeger/delete \ k8s/metrics/prometheus/operator/delete \ k8s/external/cert-manager/delete ```