Repository: open-telemetry/opentelemetry-rust

Issue #761: Infinite trace generator
Created: 2022-03-14T23:27:35Z
When I use the opentelemetry_datadog library with reqwest as the client, each batch of spans will generate an additional 2 spans. These spans look like this:
[[{'service': 'test', 'name': 'opentelemetry-datadog', 'resource': 'encode_headers', 'trace_id': 9313706354471130378, 'span_id': 2181458161362850175, 'parent_id': 0, 'start': 1647298645595811251, 'duration': 156890, 'error': 0, 'meta': {'busy_ns': '66163', 'code.namespace': 'hyper::proto::h1::role', 'code.filepath': '/home/hamiltop/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.17/src/proto/h1/role.rs', 'code.lineno': '1061', 'idle_ns': '90678'}}], [{'service': 'test', 'name': 'opentelemetry-datadog', 'resource': 'parse_headers', 'trace_id': 11785853278508097401, 'span_id': 9996945950193117364, 'parent_id': 0, 'start': 1647298645596617264, 'duration': 159966, 'error': 0, 'meta': {'code.namespace': 'hyper::proto::h1::role', 'code.lineno': '1183', 'busy_ns': '119852', 'code.filepath': '/home/hamiltop/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.17/src/proto/h1/role.rs', 'idle_ns': '36137'}}]]

I believe this is from the http request to datadog to save the original spans. Now, the request to send these 2 spans also generates 2 new spans, which generate 2 new spans, etc.
To reproduce:

Run this script as a mock datadog agent: https://gist.github.com/kongo2002/a4d560dca89e24bf27c8d59dc63f4b1e This will echo all incoming traces to stdout for easy inspecting.
Run this code:

use opentelemetry_datadog::{new_pipeline, ApiVersion};
use tracing::trace_span;
use tracing_subscriber::layer::SubscriberExt;

#[tokio::main]
async fn main() {
    let _tracer = new_pipeline()
        .with_agent_endpoint("http://localhost:8126")
        .with_service_name("test")
        .with_version(ApiVersion::Version03)
        .install_batch(opentelemetry::runtime::TokioCurrentThread)
        .unwrap();

    let telemetry = tracing_opentelemetry::layer().with_tracer(_tracer);

    let subscriber = tracing_subscriber::Registry::default().with(telemetry);

    tracing::subscriber::set_global_default(subscriber)
        .expect("Failed to install `tracing` subscriber.");

    trace_span!({ "test" }, "test span").in_scope(|| {
        println!("Hello, world!");
    });

    tokio::time::sleep(tokio::time::Duration::from_secs(60)).await;

    opentelemetry::global::shutdown_tracer_provider();
}


Watch output in script (1) to see the encode_headers/parse_headers traces get generated over and over again.

I would expect to have some way to disable tracing of the tracer. Is this a bug? Do I just need to configure trace levels more precisely?

Duplicate issue #1143: SimpleSpanProcessor is delaying traces (and sometimes crashing)
Created: 2023-07-06T10:53:14Z
Given the following server program, that uses install_simple() to export traces,
use std::net::SocketAddr;

use axum::{
    routing::{get, post},
    Router, Server,
};
use color_eyre::{install, Result};
use opentelemetry::global;
use opentelemetry::sdk::trace;
use opentelemetry_otlp::{new_exporter, new_pipeline, WithExportConfig};
use tracing_subscriber::{prelude::*, registry};

#[tokio::main]
async fn main() -> Result<()> {
    install()?;

    let tracer = new_pipeline()
        .tracing()
        .with_trace_config(trace::config())
        .with_exporter(
            new_exporter().tonic(),
        )
        .install_simple()?;

    registry()
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .init();

    let result = run().await;

    global::shutdown_tracer_provider();

    result
}

async fn run() -> Result<()> {
    // build our application with a route
    let app = Router::new()
        // `GET /` goes to `root`
        .route("/", post(root))
        // `POST /users` goes to `create_user`
        .route("/users", get(create_user));

    let addr = SocketAddr::from(([127, 0, 0, 1], 3000));

    println!("listening on {addr}");

    Server::bind(&addr).serve(app.into_make_service()).await?;

    Ok(())
}

#[tracing::instrument]
pub async fn root() -> &'static str {
    hello()
}

#[tracing::instrument]
pub async fn create_user() -> &'static str {
    "User created!"
}

#[tracing::instrument]
fn hello() -> &'static str {
    "Hello, World!"
}
I have jaegar/all-in-one docker container running that takes the traces from the above program and shows me what's happening. These are the following observations:

The first time (or minute), I hit an endpoint and generate traces, they are sent immediately and are visible in Jaegar.
After that, any endpoints I hit, all the traces take up to 2 minutes to show up.

I have reproduced this with both tonic and http.
When I use tonic, sometimes the program crashes with SIGKILL which could be caused by #1048.
When I use tonic, sometimes the traces are dropped. I am not sure what's causing it.

Closing comment (2024-03-04T09:55:52Z): Related to #761