Repository: pingcap/ticdc
Pull request #1359: kafka: option should be adjust first and then new the kafka factory
Author: login: 3AceShowHand
Labels: name: lgtm
name: approved
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: release-note-none
description: Denotes a PR that doesn't merit a release note.
Created: 2025-05-22T11:22:55Z | Merged: 2025-05-23T05:35:55Z

<!--
Thank you for contributing to TiCDC! 
Please read MD's [CONTRIBUTING](https://github.com/pingcap/ticdc/blob/master/CONTRIBUTING.md) document **BEFORE** filing this PR.
-->

### What problem does this PR solve?
<!--
Please create an issue first to describe the problem.

There MUST be one line starting with "Issue Number:  " and 
linking the relevant issues via the "close" or "ref".

For more info, check https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/contribute-code.html#referring-to-an-issue.
 -->

Issue Number: close #1360 

### What is changed and how it works?

### Check List <!--REMOVE the items that are not applicable-->

#### Tests <!-- At least one of them must be included. -->

- Unit test
- Integration test
- Manual test (add detailed scripts or steps below)
- No code

#### Questions <!-- Authors should answer these questions and reviewers should consider these questions. -->

##### Will it cause performance regression or break compatibility?

##### Do you need to update user documentation, design documentation or monitoring documentation?

### Release note <!-- bugfixes or new features need a release note -->

```release-note
`None`
```


Linked issue #1360: kafka sink adjust options before create the factory cause the message too large
adjust kafka option should be called before the kafka factory created.

Diff:
diff --git a/api/v2/changefeed.go b/api/v2/changefeed.go
index 1d49f8b106..9dbf9870e8 100644
--- a/api/v2/changefeed.go
+++ b/api/v2/changefeed.go
@@ -168,9 +168,8 @@ func (h *OpenAPIV2) CreateChangefeed(c *gin.Context) {
 	}
 
 	// verify sinkURI
-	tempChangefeedID := common.NewChangeFeedIDWithName("sink-uri-verify-changefeed-id")
 	cfConfig := info.ToChangefeedConfig()
-	err = sink.Verify(ctx, cfConfig, tempChangefeedID)
+	err = sink.Verify(ctx, cfConfig, changefeedID)
 	if err != nil {
 		_ = c.Error(errors.WrapError(errors.ErrSinkURIInvalid, err))
 		return
@@ -181,7 +180,7 @@ func (h *OpenAPIV2) CreateChangefeed(c *gin.Context) {
 		if !needRemoveGCSafePoint {
 			return
 		}
-		err := gc.UndoEnsureChangefeedStartTsSafety(
+		err = gc.UndoEnsureChangefeedStartTsSafety(
 			ctx,
 			pdClient,
 			createGcServiceID,
@@ -618,14 +617,13 @@ func (h *OpenAPIV2) UpdateChangefeed(c *gin.Context) {
 	}
 
 	// verify sink
-	tempChangefeedID := common.NewChangeFeedIDWithName("sink-uri-verify-changefeed-id")
-	err = sink.Verify(ctx, oldCfInfo.ToChangefeedConfig(), tempChangefeedID)
+	err = sink.Verify(ctx, oldCfInfo.ToChangefeedConfig(), oldCfInfo.ChangefeedID)
 	if err != nil {
 		_ = c.Error(errors.WrapError(errors.ErrSinkURIInvalid, err))
 		return
 	}
 
-	if err := coordinator.UpdateChangefeed(ctx, oldCfInfo); err != nil {
+	if err = coordinator.UpdateChangefeed(ctx, oldCfInfo); err != nil {
 		_ = c.Error(err)
 		return
 	}
diff --git a/downstreamadapter/sink/kafka/helper.go b/downstreamadapter/sink/kafka/helper.go
index 8b5a6b1ddd..85ec229b64 100644
--- a/downstreamadapter/sink/kafka/helper.go
+++ b/downstreamadapter/sink/kafka/helper.go
@@ -61,46 +61,22 @@ func newKafkaSinkComponentWithFactory(ctx context.Context,
 		return kafkaComponent, config.ProtocolUnknown, errors.Trace(err)
 	}
 
+	topic, err := helper.GetTopic(sinkURI)
+	if err != nil {
+		return kafkaComponent, protocol, errors.Trace(err)
+	}
+
 	options := kafka.NewOptions()
 	if err = options.Apply(changefeedID, sinkURI, sinkConfig); err != nil {
 		return kafkaComponent, protocol, errors.WrapError(errors.ErrKafkaInvalidConfig, err)
 	}
+	options.Topic = topic
 
 	kafkaComponent.factory, err = factoryCreator(ctx, options, changefeedID)
 	if err != nil {
 		return kafkaComponent, protocol, errors.WrapError(errors.ErrKafkaNewProducer, err)
 	}
 
-	kafkaComponent.adminClient, err = kafkaComponent.factory.AdminClient()
-	if err != nil {
-		return kafkaComponent, protocol, errors.WrapError(errors.ErrKafkaNewProducer, err)
-	}
-
-	// We must close adminClient when this func return cause by an error
-	// otherwise the adminClient will never be closed and lead to a goroutine leak.
-	defer func() {
-		if err != nil && kafkaComponent.adminClient != nil {
-			kafkaComponent.adminClient.Close()
-		}
-	}()
-
-	topic, err := helper.GetTopic(sinkURI)
-	if err != nil {
-		return kafkaComponent, protocol, errors.Trace(err)
-	}
-	// adjust the option configuration before creating the kafka client
-	if err = kafka.AdjustOptions(ctx, kafkaComponent.adminClient, options, topic); err != nil {
-		return kafkaComponent, protocol, errors.WrapError(errors.ErrKafkaNewProducer, err)
-	}
-
-	kafkaComponent.topicManager, err = topicmanager.GetTopicManagerAndTryCreateTopic(
-		ctx,
-		changefeedID,
-		topic,
-		options.DeriveTopicConfig(),
-		kafkaComponent.adminClient,
-	)
-
 	kafkaComponent.eventRouter, err = eventrouter.NewEventRouter(
 		sinkConfig, topic, false, protocol == config.ProtocolAvro)
 	if err != nil {
@@ -126,6 +102,27 @@ func newKafkaSinkComponentWithFactory(ctx context.Context,
 	if err != nil {
 		return kafkaComponent, protocol, errors.Trace(err)
 	}
+
+	kafkaComponent.adminClient, err = kafkaComponent.factory.AdminClient()
+	if err != nil {
+		return kafkaComponent, protocol, errors.WrapError(errors.ErrKafkaNewProducer, err)
+	}
+
+	// We must close adminClient when this func return cause by an error
+	// otherwise the adminClient will never be closed and lead to a goroutine leak.
+	defer func() {
+		if err != nil && kafkaComponent.adminClient != nil {
+			kafkaComponent.adminClient.Close()
+		}
+	}()
+
+	kafkaComponent.topicManager, err = topicmanager.GetTopicManagerAndTryCreateTopic(
+		ctx,
+		changefeedID,
+		topic,
+		options.DeriveTopicConfig(),
+		kafkaComponent.adminClient,
+	)
 	return kafkaComponent, protocol, nil
 }
 
diff --git a/pkg/sink/kafka/admin.go b/pkg/sink/kafka/admin.go
index 914a5c3811..08db95ca97 100644
--- a/pkg/sink/kafka/admin.go
+++ b/pkg/sink/kafka/admin.go
@@ -19,10 +19,9 @@ import (
 	"strings"
 
 	"github.com/IBM/sarama"
-	"github.com/pingcap/errors"
 	"github.com/pingcap/log"
 	"github.com/pingcap/ticdc/pkg/common"
-	cerror "github.com/pingcap/ticdc/pkg/errors"
+	"github.com/pingcap/ticdc/pkg/errors"
 	"go.uber.org/zap"
 )
 
@@ -33,7 +32,7 @@ type saramaAdminClient struct {
 	admin  sarama.ClusterAdmin
 }
 
-func (a *saramaAdminClient) GetAllBrokers(_ context.Context) ([]Broker, error) {
+func (a *saramaAdminClient) GetAllBrokers(_ context.Context) []Broker {
 	brokers := a.client.Brokers()
 	result := make([]Broker, 0, len(brokers))
 	for _, broker := range brokers {
@@ -41,8 +40,7 @@ func (a *saramaAdminClient) GetAllBrokers(_ context.Context) ([]Broker, error) {
 			ID: broker.ID(),
 		})
 	}
-
-	return result, nil
+	return result
 }
 
 func (a *saramaAdminClient) GetBrokerConfig(
@@ -76,7 +74,7 @@ func (a *saramaAdminClient) GetBrokerConfig(
 		zap.String("namespace", a.changefeed.Namespace()),
 		zap.String("changefeed", a.changefeed.Name()),
 		zap.String("configName", configName))
-	return "", cerror.ErrKafkaConfigNotFound.GenWithStack(
+	return "", errors.ErrKafkaConfigNotFound.GenWithStack(
 		"cannot find the `%s` from the broker's configuration", configName)
 }
 
@@ -110,7 +108,7 @@ func (a *saramaAdminClient) GetTopicConfig(
 		zap.String("namespace", a.changefeed.Namespace()),
 		zap.String("changefeed", a.changefeed.Name()),
 		zap.String("configName", configName))
-	return "", cerror.ErrKafkaConfigNotFound.GenWithStack(
+	return "", errors.ErrKafkaConfigNotFound.GenWithStack(
 		"cannot find the `%s` from the topic's configuration", configName)
 }
 
@@ -126,6 +124,9 @@ func (a *saramaAdminClient) GetTopicsMeta(
 
 	for _, meta := range metaList {
 		if meta.Err != sarama.ErrNoError {
+			if meta.Err == sarama.ErrUnknownTopicOrPartition {
+				continue
+			}
 			if !ignoreTopicError {
 				return nil, meta.Err
 			}
diff --git a/pkg/sink/kafka/cluster_admin_client.go b/pkg/sink/kafka/cluster_admin_client.go
index 2277540d1e..2367a2b905 100644
--- a/pkg/sink/kafka/cluster_admin_client.go
+++ b/pkg/sink/kafka/cluster_admin_client.go
@@ -33,7 +33,7 @@ type Broker struct {
 // which supports managing and inspecting topics, brokers, configurations and ACLs.
 type ClusterAdminClient interface {
 	// GetAllBrokers return all brokers among the cluster
-	GetAllBrokers(ctx context.Context) ([]Broker, error)
+	GetAllBrokers(ctx context.Context) []Broker
 
 	// GetBrokerConfig return the broker level configuration with the `configName`
 	GetBrokerConfig(ctx context.Context, configName string) (string, error)
diff --git a/pkg/sink/kafka/factory.go b/pkg/sink/kafka/factory.go
index 3498126de5..e737deddc1 100644
--- a/pkg/sink/kafka/factory.go
+++ b/pkg/sink/kafka/factory.go
@@ -40,7 +40,7 @@ type Factory interface {
 }
 
 // FactoryCreator defines the type of factory creator.
-type FactoryCreator func(context.Context, *Options, commonType.ChangeFeedID) (Factory, error)
+type FactoryCreator func(context.Context, *options, commonType.ChangeFeedID) (Factory, error)
 
 // SyncProducer is the kafka sync producer
 type SyncProducer interface {
diff --git a/pkg/sink/kafka/metrics_collector.go b/pkg/sink/kafka/metrics_collector.go
index 8f1905e0cd..b3383ce88d 100644
--- a/pkg/sink/kafka/metrics_collector.go
+++ b/pkg/sink/kafka/metrics_collector.go
@@ -30,8 +30,8 @@ type MetricsCollector interface {
 }
 
 const (
-	// RefreshMetricsInterval specifies the interval of refresh kafka client metrics.
-	RefreshMetricsInterval = 5 * time.Second
+	// refreshMetricsInterval specifies the interval of refresh kafka client metrics.
+	refreshMetricsInterval = 5 * time.Second
 	// refreshClusterMetaInterval specifies the interval of refresh kafka cluster meta.
 	// Do not set it too small, because it will cause too many requests to kafka cluster.
 	// Every request will get all topics and all brokers information.
@@ -67,7 +67,7 @@ func (m *saramaMetricsCollector) Run(ctx context.Context) {
 	// Initialize brokers.
 	m.updateBrokers(ctx)
 
-	refreshMetricsTicker := time.NewTicker(RefreshMetricsInterval)
+	refreshMetricsTicker := time.NewTicker(refreshMetricsInterval)
 	refreshClusterMetaTicker := time.NewTicker(refreshClusterMetaInterval)
 	defer func() {
 		refreshMetricsTicker.Stop()
@@ -92,18 +92,7 @@ func (m *saramaMetricsCollector) Run(ctx context.Context) {
 }
 
 func (m *saramaMetricsCollector) updateBrokers(ctx context.Context) {
-	start := time.Now()
-	brokers, err := m.adminClient.GetAllBrokers(ctx)
-	if err != nil {
-		log.Warn("Get Kafka brokers failed, "+
-			"use historical brokers to collect kafka broker level metrics",
-			zap.String("namespace", m.changefeedID.Namespace()),
-			zap.String("changefeed", m.changefeedID.Name()),
-			zap.Duration("duration", time.Since(start)),
-			zap.Error(err))
-		return
-	}
-
+	brokers := m.adminClient.GetAllBrokers(ctx)
 	for _, b := range brokers {
 		m.brokers[b.ID] = struct{}{}
 	}
diff --git a/pkg/sink/kafka/mock_cluster_admin_client.go b/pkg/sink/kafka/mock_cluster_admin_client.go
index ef1dc104f4..cadcdbacd4 100644
--- a/pkg/sink/kafka/mock_cluster_admin_client.go
+++ b/pkg/sink/kafka/mock_cluster_admin_client.go
@@ -96,8 +96,8 @@ func NewClusterAdminClientMockImpl() *ClusterAdminClientMockImpl {
 }
 
 // GetAllBrokers implement the ClusterAdminClient interface
-func (c *ClusterAdminClientMockImpl) GetAllBrokers(context.Context) ([]Broker, error) {
-	return nil, nil
+func (c *ClusterAdminClientMockImpl) GetAllBrokers(context.Context) []Broker {
+	return nil
 }
 
 // GetBrokerConfig implement the ClusterAdminClient interface
diff --git a/pkg/sink/kafka/mock_factory.go b/pkg/sink/kafka/mock_factory.go
index 892b5d3c2b..d241665b31 100644
--- a/pkg/sink/kafka/mock_factory.go
+++ b/pkg/sink/kafka/mock_factory.go
@@ -33,9 +33,9 @@ type mockFactory struct {
 // NewMockFactory constructs a Factory with mock implementation.
 func NewMockFactory(
 	_ context.Context,
-	o *Options, changefeedID commonType.ChangeFeedID,
+	o *options, changefeedID commonType.ChangeFeedID,
 ) (Factory, error) {
-	config, err := NewSaramaConfig(context.Background(), o)
+	config, err := newSaramaConfig(context.Background(), o)
 	if err != nil {
 		return nil, errors.Trace(err)
 	}
diff --git a/pkg/sink/kafka/oauth2_token_provider.go b/pkg/sink/kafka/oauth2_token_provider.go
index c447cd4702..dd25b3ff31 100644
--- a/pkg/sink/kafka/oauth2_token_provider.go
+++ b/pkg/sink/kafka/oauth2_token_provider.go
@@ -51,7 +51,7 @@ func (t *tokenProvider) Token() (*sarama.AccessToken, error) {
 	return &sarama.AccessToken{Token: token.AccessToken}, nil
 }
 
-func newTokenProvider(ctx context.Context, o *Options) (sarama.AccessTokenProvider, error) {
+func newTokenProvider(ctx context.Context, o *options) (sarama.AccessTokenProvider, error) {
 	// grant_type is by default going to be set to 'client_credentials' by the
 	// client credentials library as defined by the spec, however non-compliant
 	// auth server implementations may want a custom type
diff --git a/pkg/sink/kafka/options.go b/pkg/sink/kafka/options.go
index 37aeb9a654..39fe52524a 100644
--- a/pkg/sink/kafka/options.go
+++ b/pkg/sink/kafka/options.go
@@ -140,8 +140,9 @@ type urlConfig struct {
 	InsecureSkipVerify           *bool   `form:"insecure-skip-verify"`
 }
 
-// Options stores user specified configurations
-type Options struct {
+// options stores user specified configurations
+type options struct {
+	Topic           string
 	BrokerEndpoints []string
 
 	// control whether to create topic
@@ -173,8 +174,8 @@ type Options struct {
 }
 
 // NewOptions returns a default Kafka configuration
-func NewOptions() *Options {
-	return &Options{
+func NewOptions() *options {
+	return &options{
 		Version: "2.4.0",
 		// MaxMessageBytes will be used to initialize producer
 		MaxMessageBytes:    config.DefaultMaxMessageBytes,
@@ -191,8 +192,8 @@ func NewOptions() *Options {
 	}
 }
 
-// SetPartitionNum set the partition-num by the topic's partition count.
-func (o *Options) SetPartitionNum(realPartitionCount int32) error {
+// setPartitionNum set the partition-num by the topic's partition count.
+func (o *options) setPartitionNum(realPartitionCount int32) error {
 	// user does not specify the `partition-num` in the sink-uri
 	if o.PartitionNum == 0 {
 		o.PartitionNum = realPartitionCount
@@ -220,8 +221,8 @@ func (o *Options) SetPartitionNum(realPartitionCount int32) error {
 	return nil
 }
 
-// Apply the sinkURI to update Options
-func (o *Options) Apply(changefeedID common.ChangeFeedID,
+// Apply the sinkURI to update options
+func (o *options) Apply(changefeedID common.ChangeFeedID,
 	sinkURI *url.URL, sinkConfig *config.SinkConfig,
 ) error {
 	o.BrokerEndpoints = strings.Split(sinkURI.Host, ",")
@@ -362,7 +363,7 @@ func mergeConfig(
 	return dest, nil
 }
 
-func (o *Options) applyTLS(params *urlConfig) error {
+func (o *options) applyTLS(params *urlConfig) error {
 	if params.CA != nil && *params.CA != "" {
 		o.Credential.CAPath = *params.CA
 	}
@@ -409,7 +410,7 @@ func (o *Options) applyTLS(params *urlConfig) error {
 	return nil
 }
 
-func (o *Options) applySASL(urlParameter *urlConfig, sinkConfig *config.SinkConfig) error {
+func (o *options) applySASL(urlParameter *urlConfig, sinkConfig *config.SinkConfig) error {
 	if urlParameter.SASLUser != nil && *urlParameter.SASLUser != "" {
 		o.SASL.SASLUser = *urlParameter.SASLUser
 	}
@@ -534,7 +535,7 @@ type AutoCreateTopicConfig struct {
 	ReplicationFactor int16
 }
 
-func (o *Options) DeriveTopicConfig() *AutoCreateTopicConfig {
+func (o *options) DeriveTopicConfig() *AutoCreateTopicConfig {
 	return &AutoCreateTopicConfig{
 		AutoCreate:        o.AutoCreate,
 		PartitionNum:      o.PartitionNum,
@@ -565,11 +566,11 @@ func NewKafkaClientID(captureAddr string,
 	return
 }
 
-// AdjustOptions adjust the `Options` and `sarama.Config` by condition.
-func AdjustOptions(
+// adjustOptions adjust the `options` and `sarama.Config` by condition.
+func adjustOptions(
 	ctx context.Context,
 	admin ClusterAdminClient,
-	options *Options,
+	options *options,
 	topic string,
 ) error {
 	topics, err := admin.GetTopicsMeta(ctx, []string{topic}, true)
@@ -626,7 +627,7 @@ func AdjustOptions(
 				zap.String("topic", topic), zap.Any("detail", info))
 		}
 
-		if err := options.SetPartitionNum(info.NumPartitions); err != nil {
+		if err = options.setPartitionNum(info.NumPartitions); err != nil {
 			return errors.Trace(err)
 		}
 
@@ -731,7 +732,7 @@ func validateMinInsyncReplicas(
 		log.Error(msg, zap.Int("replication-factor", replicationFactor),
 			zap.Int("min.insync.replicas", minInsyncReplicas))
 		return cerror.ErrKafkaInvalidConfig.GenWithStack(
-			"TiCDC Kafka producer's `request.required.acks` defaults to -1, "+
+			"TiCDC Kafka sink's `request.required.acks` defaults to -1, "+
 				"TiCDC cannot deliver messages when the `replication-factor` %d "+
 				"is smaller than the `min.insync.replicas` %d of %s",
 			replicationFactor, minInsyncReplicas, configFrom,
@@ -756,7 +757,7 @@ func getTopicConfig(
 		return c, nil
 	}
 
-	log.Info("TiCDC cannot find the configuration from topic, try to get it from broker",
+	log.Info("kafka sink cannot find the configuration from topic, try to get it from broker",
 		zap.String("topic", topicName), zap.String("config", topicConfigName))
 	return admin.GetBrokerConfig(ctx, brokerConfigName)
 }
diff --git a/pkg/sink/kafka/sarama.go b/pkg/sink/kafka/sarama.go
index 2e89b5f004..c06a3fce21 100644
--- a/pkg/sink/kafka/sarama.go
+++ b/pkg/sink/kafka/sarama.go
@@ -21,9 +21,8 @@ import (
 	"time"
 
 	"github.com/IBM/sarama"
-	"github.com/pingcap/errors"
 	"github.com/pingcap/log"
-	cerror "github.com/pingcap/ticdc/pkg/errors"
+	"github.com/pingcap/ticdc/pkg/errors"
 	"github.com/pingcap/ticdc/pkg/security"
 	"go.uber.org/zap"
 )
@@ -33,8 +32,8 @@ var (
 	maxKafkaVersion     = sarama.V2_8_0_0
 )
 
-// NewSaramaConfig return the default config and set the according version and metrics
-func NewSaramaConfig(ctx context.Context, o *Options) (*sarama.Config, error) {
+// newSaramaConfig return the default config and set the according version and metrics
+func newSaramaConfig(ctx context.Context, o *options) (*sarama.Config, error) {
 	config := sarama.NewConfig()
 	config.ClientID = o.ClientID
 	var err error
@@ -118,33 +117,12 @@ func NewSaramaConfig(ctx context.Context, o *Options) (*sarama.Config, error) {
 
 	err = completeSaramaSASLConfig(ctx, config, o)
 	if err != nil {
-		return nil, cerror.WrapError(cerror.ErrKafkaInvalidConfig, err)
-	}
-
-	kafkaVersion, err := getKafkaVersion(config, o)
-	if err != nil {
-		log.Warn("Can't get Kafka version by broker. ticdc will use default version",
-			zap.String("defaultVersion", kafkaVersion.String()))
-	}
-	config.Version = kafkaVersion
-
-	if o.IsAssignedVersion {
-		version, err := sarama.ParseKafkaVersion(o.Version)
-		if err != nil {
-			return nil, cerror.WrapError(cerror.ErrKafkaInvalidVersion, err)
-		}
-		config.Version = version
-		if !version.IsAtLeast(maxKafkaVersion) && version.String() != kafkaVersion.String() {
-			log.Warn("The Kafka version you assigned may not be correct. "+
-				"Please assign a version equal to or less than the specified version",
-				zap.String("assignedVersion", version.String()),
-				zap.String("desiredVersion", kafkaVersion.String()))
-		}
+		return nil, errors.WrapError(errors.ErrKafkaInvalidConfig, err)
 	}
 	return config, nil
 }
 
-func completeSaramaSASLConfig(ctx context.Context, config *sarama.Config, o *Options) error {
+func completeSaramaSASLConfig(ctx context.Context, config *sarama.Config, o *options) error {
 	if o.SASL != nil && o.SASL.SASLMechanism != "" {
 		config.Net.SASL.Enable = true
 		config.Net.SASL.Mechanism = sarama.SASLMechanism(o.SASL.SASLMechanism)
@@ -187,9 +165,7 @@ func completeSaramaSASLConfig(ctx context.Context, config *sarama.Config, o *Opt
 	return nil
 }
 
-func getKafkaVersion(config *sarama.Config, o *Options) (sarama.KafkaVersion, error) {
-	var err error
-	version := defaultKafkaVersion
+func getKafkaVersion(config *sarama.Config, o *options) (sarama.KafkaVersion, error) {
 	addrs := o.BrokerEndpoints
 	if len(addrs) > 1 {
 		// Shuffle the list of addresses to randomize the order in which
@@ -199,13 +175,37 @@ func getKafkaVersion(config *sarama.Config, o *Options) (sarama.KafkaVersion, er
 			addrs[i], addrs[j] = addrs[j], addrs[i]
 		})
 	}
+
+	var (
+		err           error
+		targetVersion sarama.KafkaVersion
+	)
 	for i := range addrs {
-		version, err := getKafkaVersionFromBroker(config, o.RequestVersion, addrs[i])
+		targetVersion, err = getKafkaVersionFromBroker(config, o.RequestVersion, addrs[i])
 		if err == nil {
-			return version, err
+			break
+		}
+	}
+	if err != nil {
+		log.Warn("kafka sink use the default kafka version since cannot find it from the brokers",
+			zap.String("defaultVersion", defaultKafkaVersion.String()))
+		targetVersion = defaultKafkaVersion
+	}
+
+	if o.IsAssignedVersion {
+		assignedVersion, err := sarama.ParseKafkaVersion(o.Version)
+		if err != nil {
+			return assignedVersion, errors.WrapError(errors.ErrKafkaInvalidVersion, err)
+		}
+		if !assignedVersion.IsAtLeast(maxKafkaVersion) && assignedVersion.String() != targetVersion.String() {
+			log.Warn("The Kafka version you assigned may not be correct. "+
+				"Please assign a version equal to or less than the specified version",
+				zap.String("assignedVersion", assignedVersion.String()),
+				zap.String("desiredVersion", targetVersion.String()))
 		}
+		targetVersion = assignedVersion
 	}
-	return version, err
+	return targetVersion, nil
 }
 
 func getKafkaVersionFromBroker(config *sarama.Config, requestVersion int16, addr string) (sarama.KafkaVersion, error) {
@@ -213,7 +213,7 @@ func getKafkaVersionFromBroker(config *sarama.Config, requestVersion int16, addr
 	broker := sarama.NewBroker(addr)
 	err := broker.Open(config)
 	defer func() {
-		broker.Close()
+		_ = broker.Close()
 	}()
 	if err != nil {
 		log.Warn("Kafka fail to open broker", zap.String("addr", addr), zap.Error(err))
diff --git a/pkg/sink/kafka/sarama_factory.go b/pkg/sink/kafka/sarama_factory.go
index 4e6d31e456..ca4c061fcc 100644
--- a/pkg/sink/kafka/sarama_factory.go
+++ b/pkg/sink/kafka/sarama_factory.go
@@ -35,11 +35,11 @@ type saramaFactory struct {
 // NewSaramaFactory constructs a Factory with sarama implementation.
 func NewSaramaFactory(
 	ctx context.Context,
-	o *Options,
+	o *options,
 	changefeedID common.ChangeFeedID,
 ) (Factory, error) {
 	start := time.Now()
-	saramaConfig, err := NewSaramaConfig(ctx, o)
+	config, err := newSaramaConfig(ctx, o)
 	duration := time.Since(start).Seconds()
 	if duration > 2 {
 		log.Warn("new sarama config cost too much time",
@@ -48,7 +48,37 @@ func NewSaramaFactory(
 	if err != nil {
 		return nil, errors.Trace(err)
 	}
+
+	version, err := getKafkaVersion(config, o)
+	if err != nil {
+		return nil, err
+	}
+	config.Version = version
+
+	admin, err := newAdminClient(changefeedID, o.BrokerEndpoints, config)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	defer func() {
+		admin.Close()
+	}()
+
+	if err = adjustOptions(ctx, admin, o, o.Topic); err != nil {
+		return nil, errors.Trace(err)
+	}
+
+	start = time.Now()
+	saramaConfig, err := newSaramaConfig(ctx, o)
+	duration = time.Since(start).Seconds()
+	if duration > 2 {
+		log.Warn("new sarama config cost too much time",
+			zap.Any("duration", duration), zap.Stringer("changefeedID", changefeedID))
+	}
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
 	saramaConfig.MetricRegistry = metrics.NewRegistry()
+	saramaConfig.Version = version
 
 	return &saramaFactory{
 		changefeedID: changefeedID,
@@ -57,13 +87,13 @@ func NewSaramaFactory(
 	}, nil
 }
 
-func (f *saramaFactory) AdminClient() (ClusterAdminClient, error) {
+func newAdminClient(changefeedID common.ChangeFeedID, endpoints []string, config *sarama.Config) (ClusterAdminClient, error) {
 	start := time.Now()
-	client, err := sarama.NewClient(f.endpoints, f.config)
+	client, err := sarama.NewClient(endpoints, config)
 	duration := time.Since(start).Seconds()
 	if duration > 2 {
 		log.Warn("new sarama client cost too much time",
-			zap.Any("duration", duration), zap.Stringer("changefeedID", f.changefeedID))
+			zap.Any("duration", duration), zap.Stringer("changefeedID", changefeedID))
 	}
 	if err != nil {
 		return nil, errors.Trace(err)
@@ -74,7 +104,7 @@ func (f *saramaFactory) AdminClient() (ClusterAdminClient, error) {
 	duration = time.Since(start).Seconds()
 	if duration > 2 {
 		log.Warn("new sarama cluster admin cost too much time",
-			zap.Any("duration", duration), zap.Stringer("changefeedID", f.changefeedID))
+			zap.Any("duration", duration), zap.Stringer("changefeedID", changefeedID))
 	}
 	if err != nil {
 		return nil, errors.Trace(err)
@@ -82,10 +112,14 @@ func (f *saramaFactory) AdminClient() (ClusterAdminClient, error) {
 	return &saramaAdminClient{
 		client:     client,
 		admin:      admin,
-		changefeed: f.changefeedID,
+		changefeed: changefeedID,
 	}, nil
 }
 
+func (f *saramaFactory) AdminClient() (ClusterAdminClient, error) {
+	return newAdminClient(f.changefeedID, f.endpoints, f.config)
+}
+
 // SyncProducer returns a Sync SyncProducer,
 // it should be the caller's responsibility to close the producer
 func (f *saramaFactory) SyncProducer() (SyncProducer, error) {


Reviews:
user: login: gemini-code-assist[bot]
body: Hello @3AceShowHand, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

## Summary of Changes
Hello team, gemini-code-assist here with a summary of this pull request. The primary goal of this PR, as indicated by the title and patch content, is to refactor the Kafka sink initialization logic. Specifically, it aims to ensure that Kafka options, which might need adjustment based on the Kafka cluster's state (like topic partition counts), are finalized *before* the main Kafka client factory is created with the final configuration. This involves reordering some steps in the sink creation process and moving the option adjustment logic into the factory creation itself. Additionally, there are some minor cleanups in the API handlers and Kafka admin client interface.

### Highlights
* **Kafka Sink Initialization Reordering**: The process for creating a Kafka sink component has been reordered. Basic options are now applied and the topic is determined before the Kafka factory is created. The admin client and topic manager are now initialized after the factory is available.
* **Kafka Option Adjustment Logic Moved**: The logic for adjusting Kafka options (like setting partition numbers based on the topic) has been moved from the main sink component creation function (`newKafkaSinkComponentWithFactory`) into the Kafka factory creation function (`NewSaramaFactory`). A temporary admin client is created within the factory creation process solely for the purpose of option adjustment, and then closed.
* **Kafka Admin Client Interface Change**: The `GetAllBrokers` method in the `ClusterAdminClient` interface and its implementations (`saramaAdminClient`, `ClusterAdminClientMockImpl`) have been updated to no longer return an error. It now simply returns the slice of `Broker` structs.
* **API Handler Cleanup**: In the `CreateChangefeed` and `UpdateChangefeed` API handlers, the creation of a temporary `ChangeFeedID` for `sink.Verify` has been removed. The actual changefeed ID is now used directly for verification.
* **Kafka Topic Metadata Error Handling**: The `GetTopicsMeta` method in the `saramaAdminClient` now explicitly checks for `sarama.ErrUnknownTopicOrPartition` and continues processing other topics if `ignoreTopicError` is true, preventing an error return solely for non-existent topics when ignoring errors.

### Changelog
<details>
<summary>Click here to see the changelog</summary>

* **api/v2/changefeed.go**
  * Removed temporary changefeed ID creation for `sink.Verify` in `CreateChangefeed` and `UpdateChangefeed`, using the actual changefeed ID instead.
  * Minor syntax change for error assignment (`err :=` to `err =`) in `CreateChangefeed` and `UpdateChangefeed`.
* **downstreamadapter/sink/kafka/helper.go**
  * Reordered Kafka sink component initialization: get topic and apply basic options before creating the factory.
  * Moved the `AdjustOptions` call out of this function.
  * Added the topic to the `Options` struct before creating the factory.
  * Moved admin client and topic manager initialization to occur after factory creation.
* **pkg/sink/kafka/admin.go**
  * Changed `GetAllBrokers` signature to not return an error.
  * Added handling in `GetTopicsMeta` to ignore `sarama.ErrUnknownTopicOrPartition` when `ignoreTopicError` is true.
* **pkg/sink/kafka/cluster_admin_client.go**
  * Updated `ClusterAdminClient` interface to reflect `GetAllBrokers` no longer returning an error.
* **pkg/sink/kafka/metrics_collector.go**
  * Updated `updateBrokers` to use the new `GetAllBrokers` signature (no error return) and removed associated error handling.
* **pkg/sink/kafka/mock_cluster_admin_client.go**
  * Updated mock implementation of `GetAllBrokers` to match the new signature (no error return).
* **pkg/sink/kafka/options.go**
  * Added a `Topic` field to the `Options` struct.
  * Renamed `AdjustOptions` to `adjustOptions`, making it unexported.
  * Minor syntax change for error assignment (`err :=` to `err =`) in `adjustOptions`.
* **pkg/sink/kafka/sarama_factory.go**
  * Moved the `adjustOptions` call into the `NewSaramaFactory` function.
  * Created a temporary admin client within `NewSaramaFactory` to perform option adjustment before creating the final `sarama.Config`.
  * Extracted admin client creation logic into a new unexported function `newAdminClient`.

</details>


<details>

<summary><b>Using Gemini Code Assist</b></summary>
<br>

The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.

<b>Invoking Gemini</b>

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands.

Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments.
Help | `/gemini help` | Displays a list of available commands.

<b>Customization</b>

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).

<b>Limitations & Feedback</b>

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments to provide feedback.

</details>

<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>


[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/review-github-code). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).

---

> Kafka streams flow free,
> Options set before the key,
> Order now makes sense.


state: COMMENTED
author_association: NONE
submitted_at: 2025-05-22T11:23:29Z
user: login: gemini-code-assist[bot]
body: ## Code Review
This pull request successfully refactors the Kafka option initialization logic, ensuring that options are adjusted *before* the Sarama factory is fully created. This is a key improvement and aligns with the PR's stated goal. The changes also include simplifications in API handling for sink verification and fixes for variable redeclarations, which are good. 

However, there are a couple of areas in `pkg/sink/kafka/sarama_factory.go` that could be improved for robustness and potential performance, as detailed in the specific comments.

### Summary of Findings
* **Kafka Option Adjustment Order**: The PR correctly reorders Kafka option adjustment to happen before the Sarama factory is finalized, using a temporary admin client. This is the main improvement and aligns with the PR title.
* **Potential Double Fetch of Kafka Version**: `NewSaramaConfig` (which internally calls `getKafkaVersion` to detect broker version) appears to be invoked twice in `NewSaramaFactory` in a way that might cause `getKafkaVersion` to perform network operations twice if the version isn't pre-specified. This could impact setup performance.
* **Ignored Error on Temporary Admin Client Closure**: In `NewSaramaFactory`, the error returned by `admin.Close()` for the temporary admin client (used for option adjustment) is currently ignored. Logging this error could be beneficial for diagnostics.
* **API Simplification and Fixes**: The changes in `api/v2/changefeed.go` to use actual changefeed IDs for verification and fix variable redeclarations are positive. The robustness improvement in `pkg/sink/kafka/admin.go` for `GetTopicsMeta` is also good.

### Merge Readiness
This PR introduces a valuable refactoring to the Kafka sink's initialization sequence. The core logic change appears sound. However, there are two medium-severity concerns highlighted in `pkg/sink/kafka/sarama_factory.go` regarding potential performance implications of fetching the Kafka version twice and the handling of errors when closing the temporary admin client. 

I recommend addressing these points to enhance the robustness and efficiency of the change. As I am an AI assistant, I am not authorized to approve pull requests. Please ensure these changes are reviewed and approved by other maintainers before merging.
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-05-22T11:26:00Z
user: login: hongyunyan
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-23T03:53:09Z

Review comments:
diff_hunk: @@ -48,6 +48,29 @@ func NewSaramaFactory(
 	if err != nil {
 		return nil, errors.Trace(err)
 	}
+
+	admin, err := newAdminClient(changefeedID, o.BrokerEndpoints, config)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	defer func() {
+		_ = admin.Close
+	}()
path: pkg/sink/kafka/sarama_factory.go
user: login: gemini-code-assist[bot]
body: ![medium](https://img.shields.io/badge/priority-medium-yellow)

The error from `admin.Close()` is currently ignored. While this is a temporary admin client used for option adjustment, its failure to close properly could indicate underlying issues (e.g., network problems, resource leaks) or lead to them. 

Would it be better to log this error if it occurs? This would aid in diagnosing problems during changefeed setup.

```go
	defer func() {
		if err := admin.Close(); err != nil {
			log.Warn("Failed to close temporary Kafka admin client", zap.Error(err), zap.Stringer("changefeedID", changefeedID))
		}
	}()
```
created_at: 2025-05-22T11:26:00Z
author_association: NONE
diff_hunk: @@ -48,6 +48,29 @@ func NewSaramaFactory(
 	if err != nil {
 		return nil, errors.Trace(err)
 	}
+
+	admin, err := newAdminClient(changefeedID, o.BrokerEndpoints, config)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	defer func() {
+		_ = admin.Close
+	}()
+
+	if err = adjustOptions(ctx, admin, o, o.Topic); err != nil {
+		return nil, errors.Trace(err)
+	}
+
+	start = time.Now()
+	saramaConfig, err := NewSaramaConfig(ctx, o)
+	duration = time.Since(start).Seconds()
+	if duration > 2 {
+		log.Warn("new sarama config cost too much time",
+			zap.Any("duration", duration), zap.Stringer("changefeedID", changefeedID))
+	}
+	if err != nil {
+		return nil, errors.Trace(err)
path: pkg/sink/kafka/sarama_factory.go
user: login: gemini-code-assist[bot]
body: ![medium](https://img.shields.io/badge/priority-medium-yellow)

`NewSaramaConfig` is called here for the second time after `adjustOptions` has potentially modified `o`. Inside `NewSaramaConfig`, `getKafkaVersion` is called to determine the Kafka broker version if `o.Version` isn't already set. Since `adjustOptions` doesn't appear to set `o.Version`, and if the user hasn't specified a Kafka version in the sink URI, `getKafkaVersion` (which can involve network calls to brokers) might be executed twice: once for the temporary `admin` client setup (L42) and again here.

Could this lead to redundant network calls and a slight performance overhead during changefeed initialization, especially if there are many changefeeds or slow broker responses? 

Perhaps `getKafkaVersion` could be called explicitly once, its result stored (e.g., in `o.Version` if appropriate and not user-overridden), and then both `NewSaramaConfig` calls could leverage this predetermined version, avoiding a second round of broker queries for version detection.
created_at: 2025-05-22T11:26:00Z
author_association: NONE

PR comments:
user: login: ti-chi-bot[bot]
created_at: 2025-05-22T11:26:03Z
body: @gemini-code-assist[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

<details>

In response to [this](https://github.com/pingcap/ticdc/pull/1359#pullrequestreview-2860860382):

>## Code Review
>This pull request successfully refactors the Kafka option initialization logic, ensuring that options are adjusted *before* the Sarama factory is fully created. This is a key improvement and aligns with the PR's stated goal. The changes also include simplifications in API handling for sink verification and fixes for variable redeclarations, which are good. 
>
>However, there are a couple of areas in `pkg/sink/kafka/sarama_factory.go` that could be improved for robustness and potential performance, as detailed in the specific comments.
>
>### Summary of Findings
>* **Kafka Option Adjustment Order**: The PR correctly reorders Kafka option adjustment to happen before the Sarama factory is finalized, using a temporary admin client. This is the main improvement and aligns with the PR title.
>* **Potential Double Fetch of Kafka Version**: `NewSaramaConfig` (which internally calls `getKafkaVersion` to detect broker version) appears to be invoked twice in `NewSaramaFactory` in a way that might cause `getKafkaVersion` to perform network operations twice if the version isn't pre-specified. This could impact setup performance.
>* **Ignored Error on Temporary Admin Client Closure**: In `NewSaramaFactory`, the error returned by `admin.Close()` for the temporary admin client (used for option adjustment) is currently ignored. Logging this error could be beneficial for diagnostics.
>* **API Simplification and Fixes**: The changes in `api/v2/changefeed.go` to use actual changefeed IDs for verification and fix variable redeclarations are positive. The robustness improvement in `pkg/sink/kafka/admin.go` for `GetTopicsMeta` is also good.
>
>### Merge Readiness
>This PR introduces a valuable refactoring to the Kafka sink's initialization sequence. The core logic change appears sound. However, there are two medium-severity concerns highlighted in `pkg/sink/kafka/sarama_factory.go` regarding potential performance implications of fetching the Kafka version twice and the handling of errors when closing the temporary admin client. 
>
>I recommend addressing these points to enhance the robustness and efficiency of the change. As I am an AI assistant, I am not authorized to approve pull requests. Please ensure these changes are reviewed and approved by other maintainers before merging.


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-23T03:53:13Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/pingcap/ticdc/pull/1359#pullrequestreview-2863070674" title="Approved">hongyunyan</a>*

The full list of commands accepted by this bot can be found [here](https://prow.tidb.net/command-help?repo=pingcap%2Fticdc).

The pull request process is described [here](https://book.prow.tidb.net/#/workflows/pr)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/pingcap/ticdc/blob/master/OWNERS)~~ [hongyunyan]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-23T03:53:13Z
body: [LGTM Timeline notifier]
---
Timeline:

- `2025-05-23 03:53:13.133944916 +0000 UTC m=+5019.079209726`: :ballot_box_with_check: agreed by [hongyunyan](https://github.com/hongyunyan).
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-23T05:05:24Z
body: @3AceShowHand: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cdc-pulsar-integration-light | a26f2c7922a71ca8a5236611e9c1b9d0bf105d15 | [link](https://do.pingcap.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_pulsar_integration_light/634/display/redirect) | false | `/test pull-cdc-pulsar-integration-light`

[Full PR test history](https://prow.tidb.net/pr-history?org=pingcap&repo=ticdc&pr=1359). [Your PR dashboard](https://prow.tidb.net/pr?query=is%3Apr%20state%3Aopen%20author%3A3AceShowHand).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
author_association: NONE