Project: Elastic APM (Observability and Analysis / Observability)
File: otel.md

APM Server supports receiving traces, metrics, and logs over OTLP/gRPC (OpenTelemetry Protocol over gRPC). APM Server services gRPC requests on the same port as HTTP requests sent by Elastic APM agents. The gRPC server is created in the package, with requests flowing through various gRPC interceptors defined under for authentication, logging, rate limiting, etc. We register OTLP/gRPC services for receiving traces, metrics, and logs with the gRPC server in . The business logic for handling these events lives in the type . The `Consumer` type consumes decoded OTLP events, translates them to Elastic APM's data model, and then finally passes them to the standard event processing pipeline . As both OTLP and Elastic APM are evolving, there will be times when we do not translate from to Elastic APM's data model faithfully. When this occurs, it may be useful to log the original OTLP payload to compare against the indexed documents. To log OTLP payloads, you can enable debug logging in APM Server generally, or enable the "otel" debug logging selector specifically. e.g. `{"log.level":"debug","@timestamp":"2022-03-18T17:14:22.264+0800","log.logger":"otel","log.origin":{"file.name":"otel/traces.go","file.line":129},"message":"{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"unknownservice:sendotlp\"}},{\"key\":\"telemetry.sdk.language\",\"value\":{\"stringValue\":\"go\"}},{\"key\":\"telemetry.sdk.name\",\"value\":{\"stringValue\":\"opentelemetry\"}},{\"key\":\"telemetry.sdk.version\",\"value\":{\"stringValue\":\"1.5.0\"}}]},\"instrumentationLibrarySpans\":[{\"instrumentationLibrary\":{\"name\":\"sendotlp\"},\"spans\":[{\"traceId\":\"ab9f8ae614f70e977356699c9a7c26fa\",\"spanId\":\"16934e932b832ce6\",\"parentSpanId\":\"\",\"name\":\"parent\",\"kind\":\"SPANKIND_INTERNAL\",\"startTimeUnixNano\":\"1647594862240746850\",\"endTimeUnixNano\":\"1647594862263688015\",\"status\":{}}]}],\"schemaUrl\":\"https://opentelemetry.io/schemas/v1.7.0\"}]}","service.name":"apm-server","ecs.version":"1.6.0"}` In system tests we use the OpenTelemetry Go SDK for sending OTLP to APM Server. See also , a standalone program that uses the OpenTelemetry Go SDK to send some basic traces and metrics for manual testing purposes. APM Server uses a small amount of the OpenTelemetry Collector code (`go.opentelemetry.io/collector/...`) to service OTLP/gRPC calls and decode events. None of the OpenTelemetry Collector exporters, processors, configuration, and so on, are used or supported. There is currently no policy for synchronising to the latest OpenTelemetry Collector version; this is done ad hoc. We install a gRPC interceptor that checks for `authorization` in the incoming gRPC metadata, similar to checking for the `Authorization` header in HTTP requests. After extracting this value, authentication and authorization is the same as for HTTP requests from Elastic APM agents: we store an `Authorizer` in the request context, which can be used for authorizing specific actions such as ingesting an event. OpenTelemetry SDKs support sending headers (gRPC metadata) as part of the specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md Client systems (e.g. iOS) frequently have misconfigured or out-of-sync clocks, which, left unchecked, leads to distributed traces where timestamps are wildly misaligned between client/server systems. We address this by looking for an OpenTelemetry resource attribute

which is expected to hold the time at which the events were exported by the OpenTelemetry SDK to APM Server, in nanoseconds since the Unix epoch. This attribute is set by the Elastic APM iOS agent. When the server receives a payload with the export timestamp attribute, it subtracts it from the time the payload was received to calculate a time delta to add to each event in the payload. For example, if the client system sent events with an export timestamp of 1pm, and the server received the events at 2pm, then it would add 1h to each event's timestamp. This has the side effect of increasing timestamps by client/server network latency, which is typically an acceptable tradeoff. As mentioned above, APM Server services gRPC requests on the same port as HTTP requests sent by Elastic APM agents. This requires some gymnastics. First some background information. gRPC is based on top of HTTP/2. Although the `net/http` package can transparently negotiate and serve both HTTP/1 and HTTP/2, gRPC (specifically grpc-go) requires access to the lower-level HTTP/2 framing and is not based on top of the `net/http` API. HTTP/2 typically requires TLS, but can also operate in an insecure mode known as h2c (HTTP/2 Cleartext). Most if not all gRPC clients support this, known as "insecure", and can be useful for testing purposes. When TLS is in use, client and server will typically use ALPN (Application-Layer Protocol Negotiation), a feature of TLS, to negotiate the HTTP/2 protocol. Some clients, such as Go's net/http client, will automatically negotiate HTTP/2 over TLS for plain old HTTP requests. Ideally, we would like to support the following: plain old HTTP/1 requests plain old HTTP/2 requests coming over TLS gRPC requests coming over TLS gRPC requests coming over h2c We support all of these with https://github.com/elastic/gmux. Below is a summary of how it works. gmux installs an ALPN handler for "h2", HTTP/2 ALPN identifier. The handler creates a low-level HTTP/2 "framer", and decodes frames until it finds a settings frame with the Content-Type header. If the Content-Type is "application/grpc", the original payload is forwarded onto a gRPC server. Otherwise, we pass the connection off to the net/http server for plain old HTTP/2 request handling. For h2c, gmux assumes all h2c requests are for gRPC and sends them on to the gRPC server. We could also perform the Content-Type header check there, but we do not support h2c apart from gRPC.