Project: VictoriaMetrics (Observability and Analysis / Observability)
File: getting-started-with-opentelemetry.md

weight: 5 title: How to use OpenTelemetry metrics with VictoriaMetrics menu: docs: parent: "guides" weight: 5 VictoriaMetrics supports metrics ingestion with . This guide covers data ingestion via and direct metrics push from application. Install single-server version: ```sh helm repo add vm https://victoriametrics.github.io/helm-charts/ helm repo update helm install victoria-metrics vm/victoria-metrics-single ``` Verify it's up and running: ```sh kubectl get pods ``` Helm chart provides the following urls for reading and writing data: ```text Write url inside the kubernetes cluster: http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428 Read Data: The following url can be used as the datasource url in Grafana: http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428 ``` <img src="/guides/getting-started-with-opentelemetry-collector.webp"> ```sh helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update cat << EOF > values.yaml presets: clusterMetrics: enabled: true config: exporters: prometheusremotewrite: endpoint: "http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write" service: pipelines: metrics: receivers: [otlp] processors: [] exporters: [prometheusremotewrite] EOF helm upgrade -i otl-collector open-telemetry/opentelemetry-collector --set mode=deployment -f values.yaml kubectl get pod NAME READY STATUS RESTARTS AGE otl-colletor-opentelemetry-collector-7467bbb559-2pq2n 1/1 Running 0 23m kubectl port-forward victoria-metrics-victoria-metrics-single-server-0 8428 ``` The full version of possible configuration options could be found in . Metrics could be ingested into VictoriaMetrics directly with HTTP requests. You can use any compatible OpenTelemetry instrumentation . In our example, we'll create a WEB server in and instrument it with metrics. <img src="/guides/getting-started-with-opentelemetry-direct.webp"> See the full source code of the example . The list of OpenTelemetry dependencies for `go.mod` is the following: ```go go 1.20 require ( go.opentelemetry.io/otel v1.7.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric v0.30.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v0.30.0 go.opentelemetry.io/otel/metric v0.30.0 go.opentelemetry.io/otel/sdk v1.7.0 go.opentelemetry.io/otel/sdk/metric v0.30.0 ) ``` Let's create a new file `main.go` with basic implementation of the WEB server: ```go package main func main() { mux := http.NewServeMux() mux.HandleFunc("/api/fast", func(writer http.ResponseWriter, request *http.Request) { writer.WriteHeader(http.StatusOK) writer.Write([]byte(`fast ok`)) }) mux.HandleFunc("/api/slow", func(writer http.ResponseWriter, request *http.Request) { time.Sleep(time.Second * 2) writer.WriteHeader(http.StatusOK) writer.Write([]byte(`slow ok`)) }) mw, err := newMetricsMiddleware(mux) if err != nil { panic(fmt.Sprintf("cannot build metricMiddleWare: %q", err)) } go func() { http.ListenAndServe("localhost:8081", mw) }() } ``` In the code above, we used `newMetricsMiddleware` function to create a `handler` for our server. Let's define it below: ```go type metricMiddleWare struct { h http.Handler requestsCount syncint64.Counter requestsLatency syncfloat64.Histogram activeRequests int64 } func newMetricsMiddleware(h http.Handler) (*metricMiddleWare, error) { mw := &metricMiddleWare{h: h} mc, err := newMetricsController(ctx) if err != nil { return nil, fmt.Errorf("cannot build metrics collector: %w", err) } global.SetMeterProvider(mc) prov := mc.Meter("") mw.requestsLatency, err = prov.SyncFloat64().Histogram("httprequestlatency_seconds") if err != nil { return nil,

create histogram: %w", err) } mw.requestsCount, err = prov.SyncInt64().Counter("httprequeststotal") if err != nil { return nil, fmt.Errorf("cannot create syncInt64 counter: %w", err) } ar, err := prov.AsyncInt64().Gauge("httpactiverequests") if err != nil { return nil, fmt.Errorf("cannot create AsyncInt64 gauge: %w", err) } if err := prov.RegisterCallback([]instrument.Asynchronous{ar}, func(ctx context.Context) { ar.Observe(ctx, atomic.LoadInt64(&mw.activeRequests)) }); err != nil { return nil, fmt.Errorf("cannot Register int64 gauge: %w", err) } return mw, nil } ``` The new type `metricMiddleWare` is instrumented with 3 initialized in `newMetricsMiddleware` method: counter `httprequeststotal` histogram `httprequestlatency_seconds` gauge `httpactiverequests` Let's implement http.Handler interface for `metricMiddleWare` by adding `ServeHTTP` method: ```go func (m metricMiddleWare) ServeHTTP(w http.ResponseWriter, r http.Request) { t := time.Now() path := r.URL.Path m.requestsCount.Add(nil, 1, attribute.String("path", path)) atomic.AddInt64(&m.activeRequests, 1) defer func() { atomic.AddInt64(&m.activeRequests, -1) m.requestsLatency.Record(nil, time.Since(t).Seconds(), attribute.String("path", path)) }() m.h.ServeHTTP(w, r) } ``` In method above, our middleware processes received HTTP requests and updates metrics with each new request. But for these metrics to be shipped we need to add a new method `newMetricsController` to organize metrics collection: ```go func newMetricsController(ctx context.Context) (*controller.Controller, error) { options := []otlpmetrichttp.Option{ otlpmetrichttp.WithEndpoint("<VictoriaMetrics endpoint - host:port>"), otlpmetrichttp.WithURLPath("/opentelemetry/api/v1/push"), } metricExporter, err := otlpmetrichttp.New(ctx, options...) if err != nil { return nil, fmt.Errorf("cannot create otlphttp exporter: %w", err) } resourceConfig, err := resource.New(ctx, resource.WithAttributes(attribute.String("job", "otlp"), attribute.String("instance", "localhost"))) if err != nil { return nil, fmt.Errorf("cannot create meter resource: %w", err) } meterController := controller.New( processor.NewFactory( selector.NewWithHistogramDistribution( histogram.WithExplicitBoundaries([]float64{0.01, 0.05, 0.1, 0.5, 0.9, 1.0, 5.0, 10.0, 100.0}), ), aggregation.CumulativeTemporalitySelector(), processor.WithMemory(true), ), controller.WithExporter(metricExporter), controller.WithCollectPeriod(time.Second * 10), controller.WithResource(resourceConfig), ) if err := meterController.Start(ctx); err != nil { return nil, fmt.Errorf("cannot start meter controller: %w", err) } return meterController, nil } ``` This controller will collect and push collected metrics to VictoriaMetrics address with interval of `10s`. See the full source code of the example . For metrics from our WEB server to be shipped to VictoriaMetrics we need make it available for ingestion. In previous steps we already deployed a single-server VictoriaMetrics, so let's make it available locally: ```sh kubectl port-forward victoria-metrics-victoria-metrics-single-server-0 8428 ``` Now let's run our WEB server and call its APIs: ```sh go run main.go 2024/03/25 19:27:41 Starting web server... 2024/03/25 19:27:41 web server started at localhost:8081. curl http://localhost:8081/api/fast curl http://localhost:8081/api/slow ``` Open and query `httprequeststotal` or `httpactiverequests` with . <img src= "/guides/getting-started-with-opentelemetry-vmui.webp"> VictoriaMetrics doesn't support experimental JSON encoding . VictoriaMetrics supports only `AggregationTemporalityCumulative` type for and