Repository: jaegertracing/jaeger (Go)

Issue:
[Bug]: kafka: cannot connect to TLS managed kafka
### What happened?

Using v1.64.0.

The PR #6270, change the way to load TLS on kafka.

My TLS kafka configuration on 1.64:
`"TLS":{"CAFile":"","CAPem":"[REDACTED]","IncludeSystemCACertsPool":false,"CertFile":"","CertPem":"[REDACTED]","KeyFile":"","KeyPem":"[REDACTED]","MinVersion":"","MaxVersion":"","CipherSuites":null,"ReloadInterval":0,"Insecure":false,"InsecureSkipVerify":false,"ServerName":""}`

My TLS kafka option before 1.64:
`"TLS":{"Enabled":true,"CAPath":"","CertPath":"","KeyPath":"","ServerName":"","ClientCAPath":"","CipherSuites":null,"MinVersion":"","MaxVersion":"","SkipHostVerify":false,"ReloadInterval":0}`

The new configuration has `IncludeSystemCACertsPool:false` by default, it should be possible to set IncludeSystemCACertsPool to true, or issue related to the new code loading TLS kafka.

### Steps to reproduce

1. Run jaeger-collector/jaeger-ingester with kafka + TLS producer/consumer enabled without ca/cert loaded.
2. Crash to connect to remote kafka with signed certificate.
3. See Relevant log output


### Expected behavior

It should work as before.

### Relevant log output

```shell
{"level":"fatal","ts":1733844390.6674302,"caller":"collector/main.go:73","msg":"Failed to init storage factory","error":"kafka: client has run out of available brokers to talk to: 21 errors occurred:\n\t* unexpected EOF\n\t* unexpected EOF\n\t* EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:73\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.8.1/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.8.1/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.8.1/command.go:1041\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:150\nruntime.main\n\truntime/proc.go:272"}

{"level":"fatal","ts":1733842843.1473405,"caller":"ingester/main.go:71","msg":"Unable to create consumer","error":"kafka: client has run out of available brokers to talk to: 21 errors occurred:\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/ingester/main.go:71\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.8.1/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.8.1/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.8.1/command.go:1041\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/ingester/main.go:107\nruntime.main\n\truntime/proc.go:272"}
```

### Screenshot

_No response_

### Additional context

_No response_

### Jaeger backend version

v1.64.0

### SDK

_No response_

### Pipeline

_No response_

### Stogage backend

_No response_

### Operating system

_No response_

### Deployment model

_No response_

### Deployment configs

```shell

```


Discussion:
@mahadzaryab1 we missed something in the upgrade, perhaps we need to set that flag manually if OTEL struct doesn't expose it. 
@edefaria thanks for reporting. Restoring previous behavior in #6335 



Patch:
diff --git a/pkg/config/tlscfg/options.go b/pkg/config/tlscfg/options.go
index 4604313e157..c67dcb2b2ef 100644
--- a/pkg/config/tlscfg/options.go
+++ b/pkg/config/tlscfg/options.go
@@ -139,6 +139,10 @@ func (o *Options) ToOtelClientConfig() configtls.ClientConfig {
 			MinVersion:     o.MinVersion,
 			MaxVersion:     o.MaxVersion,
 			ReloadInterval: o.ReloadInterval,
+
+			// when no truststore given, use SystemCertPool
+			// https://github.com/jaegertracing/jaeger/issues/6334
+			IncludeSystemCACertsPool: o.Enabled && (len(o.CAPath) == 0),
 		},
 	}
 }


Test patch:
diff --git a/cmd/ingester/app/flags_test.go b/cmd/ingester/app/flags_test.go
index d390b57481f..65ce83a6b7b 100644
--- a/cmd/ingester/app/flags_test.go
+++ b/cmd/ingester/app/flags_test.go
@@ -10,6 +10,7 @@ import (
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"go.opentelemetry.io/collector/config/configtls"
 
 	"github.com/jaegertracing/jaeger/pkg/config"
 	"github.com/jaegertracing/jaeger/pkg/kafka/auth"
@@ -66,12 +67,18 @@ func TestTLSFlags(t *testing.T) {
 			expected: auth.AuthenticationConfig{Authentication: "kerberos", Kerberos: kerb, PlainText: plain},
 		},
 		{
-			flags:    []string{"--kafka.consumer.authentication=tls"},
-			expected: auth.AuthenticationConfig{Authentication: "tls", Kerberos: kerb, PlainText: plain},
-		},
-		{
-			flags:    []string{"--kafka.consumer.authentication=tls", "--kafka.consumer.tls.enabled=false"},
-			expected: auth.AuthenticationConfig{Authentication: "tls", Kerberos: kerb, PlainText: plain},
+			flags: []string{"--kafka.consumer.authentication=tls"},
+			expected: auth.AuthenticationConfig{
+				Authentication: "tls",
+				Kerberos:       kerb,
+				// TODO this test is unclear - if tls.enabled != true, why is it not tls.Insecure=true?
+				TLS: configtls.ClientConfig{
+					Config: configtls.Config{
+						IncludeSystemCACertsPool: true,
+					},
+				},
+				PlainText: plain,
+			},
 		},
 	}
 
diff --git a/pkg/kafka/auth/config_test.go b/pkg/kafka/auth/config_test.go
index f5354c67d2f..56d248a7e4a 100644
--- a/pkg/kafka/auth/config_test.go
+++ b/pkg/kafka/auth/config_test.go
@@ -37,20 +37,18 @@ func Test_InitFromViper(t *testing.T) {
 		"--kafka.auth.kerberos.keytab-file=/path/to/keytab",
 		"--kafka.auth.kerberos.disable-fast-negotiation=true",
 		"--kafka.auth.tls.enabled=false",
+		"--kafka.auth.tls.ca=/not/allowed/if/tls/is/disabled",
 		"--kafka.auth.plaintext.username=user",
 		"--kafka.auth.plaintext.password=password",
 		"--kafka.auth.plaintext.mechanism=SCRAM-SHA-256",
-		"--kafka.auth.tls.ca=failing",
 	})
 
 	authConfig := &AuthenticationConfig{}
 	err := authConfig.InitFromViper(configPrefix, v)
-	require.EqualError(t, err, "failed to process Kafka TLS options: kafka.auth.tls.* options cannot be used when kafka.auth.tls.enabled is false")
+	require.ErrorContains(t, err, "kafka.auth.tls.* options cannot be used when kafka.auth.tls.enabled is false")
 
-	command.ParseFlags([]string{"--kafka.auth.tls.ca="})
-	v.BindPFlags(command.Flags())
-	err = authConfig.InitFromViper(configPrefix, v)
-	require.NoError(t, err)
+	command.ParseFlags([]string{"--kafka.auth.tls.ca="}) // incrementally update authConfig
+	require.NoError(t, authConfig.InitFromViper(configPrefix, v))
 
 	expectedConfig := &AuthenticationConfig{
 		Authentication: "tls",
@@ -64,7 +62,11 @@ func Test_InitFromViper(t *testing.T) {
 			KeyTabPath:      "/path/to/keytab",
 			DisablePAFXFast: true,
 		},
-		TLS: configtls.ClientConfig{},
+		TLS: configtls.ClientConfig{
+			Config: configtls.Config{
+				IncludeSystemCACertsPool: true,
+			},
+		},
 		PlainText: PlainTextConfig{
 			Username:  "user",
 			Password:  "password",
diff --git a/plugin/storage/kafka/options_test.go b/plugin/storage/kafka/options_test.go
index cb41caac0a3..c02823fc89c 100644
--- a/plugin/storage/kafka/options_test.go
+++ b/plugin/storage/kafka/options_test.go
@@ -184,12 +184,31 @@ func TestTLSFlags(t *testing.T) {
 			expected: auth.AuthenticationConfig{Authentication: "kerberos", Kerberos: kerb, TLS: configtls.ClientConfig{}, PlainText: plain},
 		},
 		{
-			flags:    []string{"--kafka.producer.authentication=tls"},
-			expected: auth.AuthenticationConfig{Authentication: "tls", Kerberos: kerb, TLS: configtls.ClientConfig{}, PlainText: plain},
+			flags: []string{"--kafka.producer.authentication=tls"},
+			expected: auth.AuthenticationConfig{
+				Authentication: "tls",
+				Kerberos:       kerb,
+				TLS: configtls.ClientConfig{
+					Config: configtls.Config{
+						IncludeSystemCACertsPool: true,
+					},
+				},
+				PlainText: plain,
+			},
 		},
 		{
-			flags:    []string{"--kafka.producer.authentication=tls", "--kafka.producer.tls.enabled=false"},
-			expected: auth.AuthenticationConfig{Authentication: "tls", Kerberos: kerb, TLS: configtls.ClientConfig{}, PlainText: plain},
+			flags: []string{"--kafka.producer.authentication=tls", "--kafka.producer.tls.enabled=false"},
+			expected: auth.AuthenticationConfig{
+				Authentication: "tls",
+				Kerberos:       kerb,
+				// TODO this test is unclear - if tls.enabled=false, why is it not tls.Insecure=true?
+				TLS: configtls.ClientConfig{
+					Config: configtls.Config{
+						IncludeSystemCACertsPool: true,
+					},
+				},
+				PlainText: plain,
+			},
 		},
 	}