Project: containerd (Runtime / Container Runtime)
File: tracing.md

containerd supports OpenTelemetry tracing since v1.6.0. Tracing currently targets only gRPC calls. By configuring `io.containerd.tracing.processor.v1.otlp` plugin. containerd daemon can send traces to the specified OpenTelemetry endpoint. ```toml version = 2 [plugins."io.containerd.tracing.processor.v1.otlp"] endpoint = "http://localhost:4318" ``` The following options are supported. `endpoint`: The address of a server that receives . `protocol`: OpenTelemetry supports multiple protocols. The default value is "http/protobuf". "grpc" is also supported. `insecure`: Disable transport security when the protocol is "grpc". The default is false. "http/protobuf" always uses the schema provided by the endpoint and the value of this setting being ignored. The sampling ratio and the service name on the traces could be configured by `io.containerd.internal.v1.tracing` plugin. ```toml version = 2 [plugins."io.containerd.internal.v1.tracing"] sampling_ratio = 1.0 service_name = "containerd" ``` By configuring its underlying gRPC client, containerd's Go client can send traces to an OpenTelemetry endpoint. Note that the Go client's methods and gRPC calls are not 1:1. Single method call would issue multiple gRPC calls. ```go func clientWithTrace() error { exp, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint("localhost:4318"), otlptracehttp.WithInsecure(), ) if err != nil { return err } res, err := resource.New(ctx, resource.WithAttributes( semconv.ServiceNameKey.String("CLIENT NAME"), )) if err != nil { return err } provider := trace.NewTracerProvider( trace.WithSampler(trace.AlwaysSample()), trace.WithSpanProcessor(trace.NewSimpleSpanProcessor(exp)), trace.WithResource(res), ) otel.SetTracerProvider(provider) otel.SetTextMapPropagator(propagation.TraceContext{}) ... dialOpts := []grpc.DialOption{ grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()), grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()), } client, ctx, cancel, err := commands.NewClient(context, containerd.WithDialOpts(dialOpts)) if err != nil { return err } defer cancel() ctx, span := tracing.StartSpan(ctx, "OPERATION NAME") defer span.End() ... } ``` OpenTelemetry provides language specific libraries to instrument parts of your application that are not covered by automatic instrumentation. In Containerd, a thin wrapper library defined in `tracing/tracing.go` provides additional functionality and makes it easier to use the OpenTelemetry API for manual instrumentation. To create a new span, use the `tracing.StartSpan()` method. You should already have a global TracerProvider set by configuring `io.containerd.tracing.processor.v1.otlp` plugin, else this will only create an instance of a NoopSpan{}. ```go func CreateContainer(ctx context.Context, r *runtime.CreateContainerRequest) error { ctx, span := tracing.StartSpan(ctx, tracing.Name(criSpanPrefix, "CreateContainer") // name of the span tracing.WithAttribute("sandbox.id",r.GetPodSandboxId(), //attributes to be added to the span ) defer span.End() // end the span once the function returns ... } ``` Mark the span complete at the end of workflow by calling `Span.End()`. In the above example, we use 'defer' to ensure that the span is properly closed and its duration is recorded. You can add additional attributes to the span using `Span.SetAttributes()`. Attributes can be added during span creation (by passing `tracing.WithAttribute()` to tracing.StartSpan()) or at any other time during the lifecycle of a span before it has completed. ```go func CreateContainer(ctx context.Context, r *runtime.CreateContainerRequest) error { ctx, span := tracing.StartSpan(ctx, tracing.Name(criSpanPrefix, "CreateContainer") tracing.WithAttribute("sandbox.id",r.GetPodSandboxId(), ) defer span.End() ... containerId :=

containerName := makeContainerName(metadata, sandboxConfig.GetMetadata()) //Add new attributes to the existing span span.SetAttributes( tracing.Attribute("container.id", containerId), tracing.Attribute("container.name", containerName), ) ... } ``` Use `Span.AddEvent()` to add an event to an existing span. A is a specific occurrence within a span, such as the completion of an operation or the occurrence of an error. Span events can be used to provide additional information about the operation represented by the span, and can be used for debugging or performance analysis. The below example shows how we can add an event to the span to mark the execution of an NRI hook. ```go func CreateContainer(ctx context.Context, r *runtime.CreateContainerRequest) error { span := tracing.SpanFromContext(ctx) // get the current span from context ... ... if c.nri.isEnabled() { // Add an event to mark start of an NRI api call span.AddEvent("start NRI postCreateContainer request") err = c.nri.postCreateContainer(ctx, &sandbox, &container) if err != nil { log.G(ctx).WithError(err).Errorf("NRI post-create notification failed") } // Add an event to mark completion of the request span.AddEvent("finished NRI postCreateContainer request") } ... // You can also add additional attributes to an event span.AddEvent("container created", tracing.Attribute("container.create.duration", time.Since(start).String()), ) return &runtime.CreateContainerResponse{ContainerId: id}, nil } ``` You can use `Span.RecordError()` to record an error as an exception span event for this span. The `RecordError` function does not automatically set a spans status to Error, so if you wish to consider the span tracking a failed operation you should use `Span.SetStatus(err error)` to record the error and also set the span status to Error. To record an error: ```go span := tracing.SpanFromContext(ctx) defer span.End() ... err = c.nri.postCreateContainer(ctx, &sandbox, &container) if err != nil { span.RecordError(err) //record error log.G(ctx).WithError(err).Errorf("NRI post-create notification failed") } ``` To record an error and also set the span status: ```go span := tracing.SpanFromContext(ctx) defer span.End() ... err = c.nri.postCreateContainer(ctx, &sandbox, &container) if err != nil { span.SetStatus(err) //record error and set status log.G(ctx).WithError(err).Errorf("NRI post-create notification failed") } ``` OpenTelemetry maintains a set of recommended for different types of telemetry data, such as traces and metrics, to help users of the OpenTelemetry libraries and tools to collect and use telemetry data in a consistent and interoperable way. Manually instrumented spans in Containerd follow the conventions defined for and Dot-separated notation. Span Names may include relative path to the package. Span Names should include a name that represents the specific component or service performing the operation. For example: "pkg.cri.sbserver.CreateContainer" "pkg.cri.sbserver" - relative path to the package "CreateContainer" - describes the operation that is traced Lower-case. Dot-separated notation. Use a namespace based representation. For example: "http.method.get" , http.method.post". "http" - represents the general category of the attribute. "method" - specific aspect or property of the attribute. "get" - additional detail or context.