Repository: thanos-io/thanos

Issue #5872: query: v0.29.0. Opentracing config create gopanic SIGSEGV
Created: 2022-11-07T11:21:57Z
Thanos, Prometheus and Golang version used:
thanos v0.29.0
What happened:
While upgrading thanos version from 0.28.1 to v0.29.0, all our thanos queriers went in Crashloop because of a Segmentation violation (see logs below)
What you expected to happen:
as per #5411 states, there should be no breaking change.
Removing the tracing config from the specs, fix the pod (but remove tracing :) )
            - |
              --tracing.config=type: JAEGER
              config:
                service_name: "thanos-query-main"
                disabled: false
                rpc_metrics: false
                tags: ""
                sampler_type: ""
                sampler_param: 1
                sampler_manager_host_port: ""
                sampler_max_operations: 0
                sampler_refresh_interval: 0s
                reporter_max_queue_size: 0
                reporter_flush_interval: 0s
                reporter_log_spans: false
                endpoint: ""
                user: ""
                password: ""
                agent_host: "jaeger-main-agent.argos.svc"
                agent_port: 6831
How to reproduce it (as minimally and precisely as possible):
hard to tell, on our setup with the working opentracing config above, we just upgraded the docker image version... :/
I am not sure about the healthiness of my jaeger installation though, maybe it panics when the otel target is down only. Still this should not break the thanos process if this happens :)
Full logs to relevant components:
Logs

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x143d0e8]

goroutine 456 [running]:
go.opentelemetry.io/otel/sdk/trace.parentBased.ShouldSample({{0x0, 0x0}, {{0x270e3f0, 0x3a18158}, {0x270e3c8, 0x3a18158}, {0x270e3f0, 0x3a18158}, {0x270e3c8, 0x3a18158}}}, ...)
	/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.10.0/trace/sampling.go:281 +0x1c8
github.com/thanos-io/thanos/pkg/tracing/migration.samplerWithOverride.ShouldSample(...)
	/app/pkg/tracing/migration/sampler.go:42
go.opentelemetry.io/otel/sdk/trace.(*tracer).newSpan(0xc0005b0640, {0x2719670, 0xc000e97980}, {0xc0001a0d50, 0x16}, 0xc000ec3480)
	/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.10.0/trace/tracer.go:95 +0x456
go.opentelemetry.io/otel/sdk/trace.(*tracer).Start(0xc0005b0640, {0x2719670?, 0xc000e97980?}, {0xc0001a0d50, 0x16}, {0xc000e9af40?, 0x4?, 0xc000ec35b0?})
	/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.10.0/trace/tracer.go:52 +0x153
go.opentelemetry.io/otel/bridge/opentracing.(*WrapperTracer).Start(0xc0006b6228, {0x2719670?, 0xc000e97980?}, {0xc0001a0d50?, 0x1eae540?}, {0xc000e9af40?, 0x1?, 0xc000ea2f70?})
	/go/pkg/mod/go.opentelemetry.io/otel/bridge/opentracing@v1.10.0/wrapper.go:79 +0x4b
go.opentelemetry.io/otel/bridge/opentracing.(*BridgeTracer).StartSpan(0xc0006ce3c0, {0xc0001a0d50, 0x16}, {0xc000e97920, 0x3, 0x3a18158?})
	/go/pkg/mod/go.opentelemetry.io/otel/bridge/opentracing@v1.10.0/bridge.go:430 +0x3f5
github.com/thanos-io/thanos/pkg/tracing/migration.(*bridgeTracerWrapper).StartSpan(0x2009940?, {0xc0001a0d50?, 0x3a18158?}, {0xc000e97920?, 0x39acd40?, 0xc000ec3828?})
	/app/pkg/tracing/migration/bridge.go:89 +0x26
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.newClientSpanFromContext({0x2719638, 0xc000ed88a0}, {0x2714460, 0xc00030a080}, {0xc0001a0d50, 0x16})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/tracing/client.go:92 +0x245
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.(*opentracingClientReportable).ClientReporter(0xc0006b64e0, {0x2719638, 0xc000ed88a0}, {0x0?, 0x0?}, {0x220b8e2, 0x5}, {0x22309c5, 0x10}, {0x22309d6, ...})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/tracing/client.go:51 +0x127
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryClientInterceptor.func1({0x2719638, 0xc000ed88a0}, {0x22309c4, 0x16}, {0x21122a0, 0x3a18158}, {0x21123e0, 0xc000ed89c0}, 0xc000ea0c30?, 0x22f4818, ...)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/client.go:19 +0x195
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryClient.func1.1.1({0x2719638?, 0xc000ed88a0?}, {0x22309c4?, 0x38?}, {0x21122a0?, 0x3a18158?}, {0x21123e0?, 0xc000ed89c0?}, 0x0?, {0xc000eb47c0, ...})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:74 +0x86
github.com/grpc-ecosystem/go-grpc-prometheus.(*ClientMetrics).UnaryClientInterceptor.func1({0x2719638, 0xc000ed88a0}, {0x22309c4, 0x16}, {0x21122a0, 0x3a18158}, {0x21123e0, 0xc000ed89c0}, 0x8?, 0xc000e98a68, ...)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/client_metrics.go:112 +0x117
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryClient.func1.1.1({0x2719638?, 0xc000ed88a0?}, {0x22309c4?, 0x203000?}, {0x21122a0?, 0x3a18158?}, {0x21123e0?, 0xc000ed89c0?}, 0x1?, {0xc000eb47c0, ...})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:74 +0x86
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryClient.func1({0x2719638, 0xc000ed88a0}, {0x22309c4, 0x16}, {0x21122a0, 0x3a18158}, {0x21123e0, 0xc000ed89c0}, 0x0?, 0x22f4818, ...)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:83 +0x157
google.golang.org/grpc.(*ClientConn).Invoke(0xc00085ea00?, {0x2719638?, 0xc000ed88a0?}, {0x22309c4?, 0x16?}, {0x21122a0?, 0x3a18158?}, {0x21123e0?, 0xc000ed89c0?}, {0xc000e94a10, ...})
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35 +0x223
github.com/thanos-io/thanos/pkg/info/infopb.(*infoClient).Info(0xc00030a2c8, {0x2719638, 0xc000ed88a0}, 0x8?, {0xc000e94a10, 0x1, 0x1})
	/app/pkg/info/infopb/rpc.pb.go:422 +0xc9
github.com/thanos-io/thanos/pkg/query.(*endpointRef).Metadata(0xc0006fca80, {0x2719638, 0xc000ed88a0}, {0x2700800, 0xc00030a2c8}, {0x271a8d0, 0xc00030a2d0})
	/app/pkg/query/endpointset.go:61 +0xe3
github.com/thanos-io/thanos/pkg/query.(*EndpointSet).updateEndpoint(0xc000a48300, {0x2719638, 0xc000ed88a0}, 0xc000e987c8, 0xc0006fca80)
	/app/pkg/query/endpointset.go:409 +0x105
github.com/thanos-io/thanos/pkg/query.(*EndpointSet).Update.func2(0xc000e987c8)
	/app/pkg/query/endpointset.go:349 +0x2cb
created by github.com/thanos-io/thanos/pkg/query.(*EndpointSet).Update
	/app/pkg/query/endpointset.go:338 +0x60a

Duplicate issue #5943: panic when starting with tracing config for jaeger
Created: 2022-12-07T16:12:04Z
Thanos, Prometheus and Golang version used: thanos:0.29.0

Object Storage Provider: Minio
What happened:
Thanos crashes with segmentation fault when an API request arrives after starting up
What you expected to happen:
Thanos starts and reports traces to Tempo using Jaeger format
How to reproduce it (as minimally and precisely as possible):
Configured basic tracing settings via argument flag on the pod:
- |
  --tracing.config=type: JAEGER
  config:
    service_name: "thanos"
    endpoint: "http://tempo-distributor.tempo:14268/api/traces"

Full logs to relevant components:
Logs

level=debug ts=2022-12-06T22:47:20.058736343Z caller=main.go:67 msg="maxprocs: Leaving GOMAXPROCS=[4]: CPU quota undefined"
level=info ts=2022-12-06T22:47:20.085074223Z caller=factory.go:43 msg="loading tracing configuration"
level=info ts=2022-12-06T22:47:20.086482782Z caller=client.go:56 msg="enabling client to server TLS"
level=info ts=2022-12-06T22:47:20.086785261Z caller=options.go:114 msg="TLS client using provided certificate pool"
level=info ts=2022-12-06T22:47:20.086854475Z caller=options.go:147 msg="TLS client authentication enabled"
level=debug ts=2022-12-06T22:47:20.087886643Z caller=engine.go:347 msg="Lookback delta is zero, setting to default value" value=5m0s
level=info ts=2022-12-06T22:47:20.093106817Z caller=options.go:26 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-12-06T22:47:20.094581885Z caller=query.go:776 msg="starting query node"
level=info ts=2022-12-06T22:47:20.095158803Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=debug ts=2022-12-06T22:47:20.095577726Z caller=endpointset.go:309 component=endpointset msg="starting to update API endpoints" cachedEndpoints=0
level=debug ts=2022-12-06T22:47:20.096116061Z caller=endpointset.go:384 component=endpointset msg="updated endpoints" activeEndpoints=0
level=info ts=2022-12-06T22:47:20.096911999Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-12-06T22:47:20.098377739Z caller=grpc.go:131 service=gRPC/server component=query msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2022-12-06T22:47:20.09864108Z caller=http.go:73 service=http/server component=query msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2022-12-06T22:47:20.099325109Z caller=tls_config.go:195 service=http/server component=query msg="TLS is disabled." http2=false
level=debug ts=2022-12-06T22:47:25.148605711Z caller=endpointset.go:309 component=endpointset msg="starting to update API endpoints" cachedEndpoints=0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x143d0e8]

goroutine 86 [running]:
go.opentelemetry.io/otel/sdk/trace.parentBased.ShouldSample({{0x0, 0x0}, {{0x270e3f0, 0x3a18158}, {0x270e3c8, 0x3a18158}, {0x270e3f0, 0x3a18158}, {0x270e3c8, 0x3a18158}}}, ...)
    /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.10.0/trace/sampling.go:281 +0x1c8
github.com/thanos-io/thanos/pkg/tracing/migration.samplerWithOverride.ShouldSample(...)
    /app/pkg/tracing/migration/sampler.go:42
go.opentelemetry.io/otel/sdk/trace.(*tracer).newSpan(0xc00081f040, {0x2719670, 0xc0006bafc0}, {0xc0002c40d8, 0x16}, 0xc0006e3480)
    /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.10.0/trace/tracer.go:95 +0x456
go.opentelemetry.io/otel/sdk/trace.(*tracer).Start(0xc00081f040, {0x2719670?, 0xc0006bafc0?}, {0xc0002c40d8, 0x16}, {0xc0006b86c0?, 0x4?, 0xc0006e35b0?})
    /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.10.0/trace/tracer.go:52 +0x153
go.opentelemetry.io/otel/bridge/opentracing.(*WrapperTracer).Start(0xc000863620, {0x2719670?, 0xc0006bafc0?}, {0xc0002c40d8?, 0x1eae540?}, {0xc0006b86c0?, 0x1?, 0xc000117790?})
    /go/pkg/mod/go.opentelemetry.io/otel/bridge/opentracing@v1.10.0/wrapper.go:79 +0x4b
go.opentelemetry.io/otel/bridge/opentracing.(*BridgeTracer).StartSpan(0xc000784fc0, {0xc0002c40d8, 0x16}, {0xc0006baf60, 0x3, 0x3a18158?})
    /go/pkg/mod/go.opentelemetry.io/otel/bridge/opentracing@v1.10.0/bridge.go:430 +0x3f5
github.com/thanos-io/thanos/pkg/tracing/migration.(*bridgeTracerWrapper).StartSpan(0x2009940?, {0xc0002c40d8?, 0x3a18158?}, {0xc0006baf60?, 0x39acd40?, 0xc0006e3828?})
    /app/pkg/tracing/migration/bridge.go:89 +0x26
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.newClientSpanFromContext({0x2719638, 0xc0006b45a0}, {0x2714460, 0xc0008440f0}, {0xc0002c40d8, 0x16})
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/tracing/client.go:92 +0x245
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/tracing.(*opentracingClientReportable).ClientReporter(0xc0008638d8, {0x2719638, 0xc0006b45a0}, {0x0?, 0x0?}, {0x220b8e2, 0x5}, {0x22309c5, 0x10}, {0x22309d6, ...})
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/tracing/client.go:51 +0x127
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryClientInterceptor.func1({0x2719638, 0xc0006b45a0}, {0x22309c4, 0x16}, {0x21122a0, 0x3a18158}, {0x21123e0, 0xc0006b46c0}, 0xc0002903c0?, 0x22f4818, ...)
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/client.go:19 +0x195
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryClient.func1.1.1({0x2719638?, 0xc0006b45a0?}, {0x22309c4?, 0x38?}, {0x21122a0?, 0x3a18158?}, {0x21123e0?, 0xc0006b46c0?}, 0x0?, {0xc0001255a0, ...})
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:74 +0x86
github.com/grpc-ecosystem/go-grpc-prometheus.(*ClientMetrics).UnaryClientInterceptor.func1({0x2719638, 0xc0006b45a0}, {0x22309c4, 0x16}, {0x21122a0, 0x3a18158}, {0x21123e0, 0xc0006b46c0}, 0x8?, 0xc00011c348, ...)
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/client_metrics.go:112 +0x117
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryClient.func1.1.1({0x2719638?, 0xc0006b45a0?}, {0x22309c4?, 0x203000?}, {0x21122a0?, 0x3a18158?}, {0x21123e0?, 0xc0006b46c0?}, 0x1?, {0xc0001255a0, ...})
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:74 +0x86
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryClient.func1({0x2719638, 0xc0006b45a0}, {0x22309c4, 0x16}, {0x21122a0, 0x3a18158}, {0x21123e0, 0xc0006b46c0}, 0x0?, 0x22f4818, ...)
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:83 +0x157
google.golang.org/grpc.(*ClientConn).Invoke(0xc000798000?, {0x2719638?, 0xc0006b45a0?}, {0x22309c4?, 0x16?}, {0x21122a0?, 0x3a18158?}, {0x21123e0?, 0xc0006b46c0?}, {0xc0001ea7d0, ...})
    /go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35 +0x223
github.com/thanos-io/thanos/pkg/info/infopb.(*infoClient).Info(0xc0002ba1b8, {0x2719638, 0xc0006b45a0}, 0x8?, {0xc0001ea7d0, 0x1, 0x1})
    /app/pkg/info/infopb/rpc.pb.go:422 +0xc9
github.com/thanos-io/thanos/pkg/query.(*endpointRef).Metadata(0xc00061c180, {0x2719638, 0xc0006b45a0}, {0x2700800, 0xc0002ba1b8}, {0x271a8d0, 0xc0002ba478})
    /app/pkg/query/endpointset.go:61 +0xe3
github.com/thanos-io/thanos/pkg/query.(*EndpointSet).updateEndpoint(0xc0008a6500, {0x2719638, 0xc0006b45a0}, 0xc00011c2d0, 0xc00061c180)
    /app/pkg/query/endpointset.go:409 +0x105
github.com/thanos-io/thanos/pkg/query.(*EndpointSet).Update.func2(0xc00011c2d0)
    /app/pkg/query/endpointset.go:349 +0x2cb
created by github.com/thanos-io/thanos/pkg/query.(*EndpointSet).Update
    /app/pkg/query/endpointset.go:338 +0x60a
Stream closed EOF for thanos/thanos-query-5db87c6989-lvvl5 (query)




Anything else we need to know:
The above issue does not happen when OTLP grpc tracing is configured
            - |
              --tracing.config=type: OTLP
              config:
                client_type: "grpc"
                insecure: true
                endpoint: "tempo-distributor.tempo:4317"

Closing comment (2022-12-07T16:46:17Z): Duplicate of #5872