Repository: traefik/traefik (language: go)

Issue:
Apply Trace Context to Logs if Tracing is Enabled
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you expect to see?

Tracing with Datadog works very well from the get-go.  The one item that I find missing is that the tracing context is not attached to any logs related to a particular request.  This includes logs relating to middleware-related actions and access logs.  I could be missing other use-cases, but would be helpful to tie Traefik logs with a particular trace instead of having to manually look for them.  I haven't found any blogs or issues around this support AFAIK.

This request would also apply to other vendors as well. 


Discussion / hints:
Hey @danhlee329 !

Thanks for your suggestion.

We are interested in this issue but are unsure about the use case and the traction it will receive, so we are going to leave the status as "kind/proposal" to give the community time to let us know that they would like this.
We will reevaluate as people respond.

This is a pain point for us as well. All of our other services in our system are able to correlate logs with traces which is vital when trying to diagnose issues or review audit / access logs.

With Traefik, we get the traces showing up in datadog, but you then need to go spelunking in the datadog logs explorer to find the logs you _think_ are the right ones. This becomes a challenge when you see 100s of requests to the same endpoint. As far as I know, all datadog requires is adding a key in the JSON log with the trace ID. We typically use java and this comes for free through auto-instrumentation. With GO, I would imagine you need to ensure you are injecting the trace context manually into logs yourself
> This is a pain point for us as well. All of our other services in our system are able to correlate logs with traces which is vital when trying to diagnose issues or review audit / access logs.
> 
> With Traefik, we get the traces showing up in datadog, but you then need to go spelunking in the datadog logs explorer to find the logs you _think_ are the right ones. This becomes a challenge when you see 100s of requests to the same endpoint. As far as I know, all datadog requires is adding a key in the JSON log with the trace ID. We typically use java and this comes for free through auto-instrumentation. With GO, I would imagine you need to ensure you are injecting the trace context manually into logs yourself

Luckily we're at least able to apply the [trace remapper](https://docs.datadoghq.com/logs/log_configuration/processors/?tab=ui#trace-remapper) for our Traefik access logs, but would be really nice to tie all other Traefik-related logs for a particular trace.  This will help tons around the middlewares and their logs that are generated around a particular trace for better visibility as well as any `DEBUG` logs if we enable it.
Hello,

This would be a good enhancement and we are labeling this issue accordingly.
Related to #8599

@danhlee329 Could you explain how you managed to include a trace ID in the Traefik access logs? From what I've seen, it seems we can't directly add the trace ID into the access log. Typically, the trace remapper functions effectively when the ID is manually inserted into the log, allowing for correlation between traces. Any insights would be appreciated.
> @danhlee329 Could you explain how you managed to include a trace ID in the Traefik access logs? From what I've seen, it seems we can't directly add the trace ID into the access log. Typically, the trace remapper functions effectively when the ID is manually inserted into the log, allowing for correlation between traces. Any insights would be appreciated.

@markrity Initially, I've enabled Datadog tracing in Traefik, which would pass the tracing context via HTTP headers downstream.  To ensure that we have the `X-Datadog-Trace-Id` header in the response (as we need per our API for our customers), we used the [traefik-plugin-add-response-header plugin](https://plugins.traefik.io/plugins/63b899ec3038a467c0ee9cb5/add-response-header) (which we renamed the header to something else + also accepting the caveat in the plugin, which we haven't experienced any issues yet).  With the response headers and their values captured in the access logs, we then grab that value and use the trace-id mapper for the access log.

The one issue we found is that throughout the middleware chain, if any of the middlewares short-circuits and breaks the flow from the chain, we lose the trace-id response header (even if we set the response-header-plugin middleware first).  What we found was that calling the ForwardAuth middleware after the response-header-plugin (as we need for our use-case) helps mitigate this and any middleware-short-circuiting thereafter doesn't lose that header anymore (wondering if any Traefik-ers know why this is the case).  Hope this helps!
I also need this feature, excepting traefik all services logs could be tracked in datadog.

I am glad to work this issue if you don't have enough time. I just only want to add `traceId` and `spanId` into logs, and then each log systems everyone could  parse the fields.
@weijiany Sure, we would gladly welcome a contribution! Thanks!

Fix patch:
diff --git a/docs/content/observability/access-logs.md b/docs/content/observability/access-logs.md
index f5481eb960..eb30e0200e 100644
--- a/docs/content/observability/access-logs.md
+++ b/docs/content/observability/access-logs.md
@@ -250,6 +250,8 @@ accessLog:
     | `TLSVersion`            | The TLS version used by the connection (e.g. `1.2`) (if connection is TLS).                                                                                         |
     | `TLSCipher`             | The TLS cipher used by the connection (e.g. `TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA`) (if connection is TLS)                                                           |
     | `TLSClientSubject`      | The string representation of the TLS client certificate's Subject (e.g. `CN=username,O=organization`)                                                               |
+    | `TraceId`               | A consistent identifier for tracking requests across services, including upstream ones managed by Traefik, shown as a 32-hex digit string                           |
+    | `SpanId`                | A unique identifier for Traefik’s root span (EntryPoint) within a request trace, formatted as a 16-hex digit string.                                                |
 
 ## Log Rotation
 
diff --git a/pkg/middlewares/accesslog/logdata.go b/pkg/middlewares/accesslog/logdata.go
index ef3f3be6cd..19dc43d6c4 100644
--- a/pkg/middlewares/accesslog/logdata.go
+++ b/pkg/middlewares/accesslog/logdata.go
@@ -77,6 +77,11 @@ const (
 	TLSCipher = "TLSCipher"
 	// TLSClientSubject is the string representation of the TLS client certificate's Subject.
 	TLSClientSubject = "TLSClientSubject"
+
+	// TraceID is the consistent identifier for tracking requests across services, including upstream ones managed by Traefik, shown as a 32-hex digit string.
+	TraceID = "TraceId"
+	// SpanID is the unique identifier for Traefik’s root span (EntryPoint) within a request trace, formatted as a 16-hex digit string.
+	SpanID = "SpanId"
 )
 
 // These are written out in the default case when no config is provided to specify keys of interest.
diff --git a/pkg/middlewares/observability/entrypoint.go b/pkg/middlewares/observability/entrypoint.go
index 0de455b92e..4849f25b2d 100644
--- a/pkg/middlewares/observability/entrypoint.go
+++ b/pkg/middlewares/observability/entrypoint.go
@@ -11,6 +11,7 @@ import (
 	"github.com/containous/alice"
 	"github.com/traefik/traefik/v3/pkg/metrics"
 	"github.com/traefik/traefik/v3/pkg/middlewares"
+	"github.com/traefik/traefik/v3/pkg/middlewares/accesslog"
 	"github.com/traefik/traefik/v3/pkg/tracing"
 	"go.opentelemetry.io/otel/attribute"
 	"go.opentelemetry.io/otel/metric"
@@ -69,6 +70,12 @@ func (e *entryPointTracing) ServeHTTP(rw http.ResponseWriter, req *http.Request)
 
 	e.tracer.CaptureServerRequest(span, req)
 
+	if logData := accesslog.GetLogData(req); logData != nil {
+		spanContext := span.SpanContext()
+		logData.Core[accesslog.TraceID] = spanContext.TraceID().String()
+		logData.Core[accesslog.SpanID] = spanContext.SpanID().String()
+	}
+
 	recorder := newStatusCodeRecorder(rw, http.StatusOK)
 	e.next.ServeHTTP(recorder, req)
 


Test patch:
diff --git a/pkg/middlewares/observability/entrypoint_test.go b/pkg/middlewares/observability/entrypoint_test.go
index f32c052c8d..3e7a90870d 100644
--- a/pkg/middlewares/observability/entrypoint_test.go
+++ b/pkg/middlewares/observability/entrypoint_test.go
@@ -11,6 +11,7 @@ import (
 	"github.com/stretchr/testify/require"
 	ptypes "github.com/traefik/paerser/types"
 	"github.com/traefik/traefik/v3/pkg/metrics"
+	"github.com/traefik/traefik/v3/pkg/middlewares/accesslog"
 	"github.com/traefik/traefik/v3/pkg/tracing"
 	"github.com/traefik/traefik/v3/pkg/types"
 	"go.opentelemetry.io/otel/attribute"
@@ -181,3 +182,27 @@ func TestEntryPointMiddleware_metrics(t *testing.T) {
 		})
 	}
 }
+
+func TestEntryPointMiddleware_tracingInfoIntoLog(t *testing.T) {
+	req := httptest.NewRequest(http.MethodGet, "http://www.test.com/", http.NoBody)
+	req = req.WithContext(
+		context.WithValue(
+			req.Context(),
+			accesslog.DataTableKey,
+			&accesslog.LogData{Core: accesslog.CoreLogData{}},
+		),
+	)
+
+	next := http.HandlerFunc(func(rw http.ResponseWriter, _ *http.Request) {})
+
+	tracer := &mockTracer{}
+
+	handler := newEntryPoint(context.Background(), tracing.NewTracer(tracer, []string{}, []string{}, []string{}), nil, "test", next)
+	handler.ServeHTTP(httptest.NewRecorder(), req)
+
+	expectedSpanCtx := tracer.spans[0].SpanContext()
+
+	logData := accesslog.GetLogData(req)
+	assert.Equal(t, expectedSpanCtx.TraceID().String(), logData.Core[accesslog.TraceID])
+	assert.Equal(t, expectedSpanCtx.SpanID().String(), logData.Core[accesslog.SpanID])
+}