Repository: PingCAP-QE/ci
Pull request #3522: ticdc: add integration test and make it ready
Author: login: wk989898
Labels: name: approved
name: lgtm
name: size/XXL
name: area/jobs/jenkins-DSL
name: area/jenkins-pipelines
name: area/jobs/prow
Created: 2025-05-15T10:26:06Z | Merged: 2025-05-16T07:18:31Z



Diff:
diff --git a/jobs/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy b/jobs/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
new file mode 100644
index 0000000000..c72d8e4d62
--- /dev/null
+++ b/jobs/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
@@ -0,0 +1,38 @@
+// REF: https://<your-jenkins-server>/plugin/job-dsl/api-viewer/index.html
+// For trunk and latest release branches.
+pipelineJob('pingcap/ticdc/pull_cdc_kafka_integration_heavy') {
+    logRotator {
+        daysToKeep(30)
+    }
+    parameters {
+        // Ref: https://docs.prow.k8s.io/docs/jobs/#job-environment-variables
+        stringParam("BUILD_ID")
+        stringParam("PROW_JOB_ID")
+        stringParam("JOB_SPEC")
+    }
+    properties {
+        githubProjectUrl("https://github.com/pingcap/ticdc")
+    }
+ 
+    definition {
+        cpsScm {
+            lightweight(true)
+            scriptPath("pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy")
+            scm {
+                git{
+                    remote {
+                        url('https://github.com/PingCAP-QE/ci.git')
+                    }
+                    branch('main')
+                    extensions {
+                        cloneOptions {
+                            depth(1)
+                            shallow(true)
+                            timeout(5)
+                        } 
+                    }
+                }
+            }
+        }
+    }
+}
diff --git a/jobs/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy b/jobs/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy
new file mode 100644
index 0000000000..5a01761fde
--- /dev/null
+++ b/jobs/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy
@@ -0,0 +1,38 @@
+// REF: https://<your-jenkins-server>/plugin/job-dsl/api-viewer/index.html
+// For trunk and latest release branches.
+pipelineJob('pingcap/ticdc/pull_cdc_storage_integration_heavy') {
+    logRotator {
+        daysToKeep(30)
+    }
+    parameters {
+        // Ref: https://docs.prow.k8s.io/docs/jobs/#job-environment-variables
+        stringParam("BUILD_ID")
+        stringParam("PROW_JOB_ID")
+        stringParam("JOB_SPEC")
+    }
+    properties {
+        githubProjectUrl("https://github.com/pingcap/ticdc")
+    }
+ 
+    definition {
+        cpsScm {
+            lightweight(true)
+            scriptPath("pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy")
+            scm {
+                git{
+                    remote {
+                        url('https://github.com/PingCAP-QE/ci.git')
+                    }
+                    branch('main')
+                    extensions {
+                        cloneOptions {
+                            depth(1)
+                            shallow(true)
+                            timeout(5)
+                        } 
+                    }
+                }
+            }
+        }
+    }
+}
diff --git a/pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml b/pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml
new file mode 100644
index 0000000000..568cc54c00
--- /dev/null
+++ b/pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml
@@ -0,0 +1,159 @@
+apiVersion: v1
+kind: Pod
+spec:
+  securityContext:
+    fsGroup: 1000
+  containers:
+    - image: wurstmeister/zookeeper
+      imagePullPolicy: IfNotPresent
+      name: zookeeper
+      resources:
+        requests:
+          cpu: 1000m
+          memory: 4Gi
+        limits:
+          cpu: 1000m
+          memory: 4Gi
+      tty: true
+      volumeMounts:
+        - mountPath: /tmp
+          name: volume-0
+    - args:
+        - cat
+      image: hub.pingcap.net/jenkins/rocky8_golang-1.23:tini
+      imagePullPolicy: Always
+      name: golang
+      resources:
+        limits:
+          cpu: "12"
+          memory: 16Gi
+      tty: true
+      volumeMounts:
+        - mountPath: /tmp
+          name: volume-0
+    - env:
+        - name: KAFKA_CREATE_TOPICS
+          value: big-message-test:1:1
+        - name: KAFKA_BROKER_ID
+          value: "1"
+        - name: KAFKA_SSL_KEYSTORE_PASSWORD
+          value: test1234
+        - name: KAFKA_ZOOKEEPER_CONNECT
+          value: localhost:2181
+        - name: KAFKA_MESSAGE_MAX_BYTES
+          value: "11534336"
+        - name: KAFKA_REPLICA_FETCH_MAX_BYTES
+          value: "11534336"
+        - name: KAFKA_ADVERTISED_LISTENERS
+          value: SSL://127.0.0.1:9093,PLAINTEXT://127.0.0.1:9092
+        - name: ZK
+          value: zk
+        - name: KAFKA_SSL_KEYSTORE_LOCATION
+          value: /tmp/kafka.server.keystore.jks
+        - name: KAFKA_SSL_KEY_PASSWORD
+          value: test1234
+        - name: KAFKA_SSL_TRUSTSTORE_PASSWORD
+          value: test1234
+        - name: KAFKA_LISTENERS
+          value: SSL://127.0.0.1:9093,PLAINTEXT://127.0.0.1:9092
+        - name: KAFKA_SSL_TRUSTSTORE_LOCATION
+          value: /tmp/kafka.server.truststore.jks
+        - name: RACK_COMMAND
+          value: curl -sfL https://github.com/pingcap/tiflow/raw/6e62afcfecc4e3965d8818784327d4bf2600d9fa/tests/_certificates/kafka.server.keystore.jks -o /tmp/kafka.server.keystore.jks && curl -sfL https://github.com/pingcap/tiflow/raw/6e62afcfecc4e3965d8818784327d4bf2600d9fa/tests/_certificates/kafka.server.truststore.jks -o /tmp/kafka.server.truststore.jks
+      image: wurstmeister/kafka:2.12-2.4.1
+      imagePullPolicy: IfNotPresent
+      name: kafka
+      resources:
+        requests:
+          cpu: 2000m
+          memory: 6Gi
+        limits:
+          cpu: 2000m
+          memory: 6Gi
+      tty: true
+      volumeMounts:
+        - mountPath: /tmp
+          name: volume-0
+    - env:
+        - name: KAFKA_SERVER
+          value: 127.0.0.1:9092
+        - name: ZOOKEEPER_SERVER
+          value: 127.0.0.1:2181
+        - name: DOWNSTREAM_DB_HOST
+          value: 127.0.0.1
+        - name: USE_FLAT_MESSAGE
+          value: "true"
+        - name: DOWNSTREAM_DB_PORT
+          value: "3306"
+        - name: DB_NAME
+          value: test
+      image: rustinliu/ticdc-canal-json-adapter:latest
+      imagePullPolicy: IfNotPresent
+      name: canal-adapter
+      # TODO: add resource limit
+      # can't add resource limit now, because canal-adapter will OOM
+      # issue: https://github.com/PingCAP-QE/ci/issues/1893
+      resources:
+        requests:
+          cpu: 2000m
+          memory: 4Gi
+      tty: true
+      volumeMounts:
+        - mountPath: /tmp
+          name: volume-0
+    - name: net-tool
+      image: hub.pingcap.net/jenkins/network-multitool
+      tty: true
+      resources:
+        limits:
+          memory: 128Mi
+          cpu: 600m
+    - name: mysql
+      image: quay.io/debezium/example-mysql:2.4
+      imagePullPolicy: IfNotPresent
+      env:
+        - name: MYSQL_ROOT_PASSWORD
+          value: ""
+        - name: MYSQL_USER
+          value: mysqluser
+        - name: MYSQL_PASSWORD
+          value: mysqlpw
+        - name: MYSQL_ALLOW_EMPTY_PASSWORD
+          value: "yes"
+        - name: MYSQL_TCP_PORT
+          value: "3310"
+      resources:
+        requests:
+          cpu: 2000m
+          memory: 4Gi
+    - name: connect
+      image: quay.io/debezium/connect:2.4
+      env:
+        - name: BOOTSTRAP_SERVERS
+          value: "127.0.0.1:9092"
+        - name: GROUP_ID
+          value: "1"
+        - name: CONFIG_STORAGE_TOPIC
+          value: "my_connect_configs"
+        - name: OFFSET_STORAGE_TOPIC
+          value: "my_connect_offsets"
+        - name: STATUS_STORAGE_TOPIC
+          value: "my_connect_statuses"
+        - name: LANG
+          value: "C.UTF-8"
+      resources:
+        requests:
+          cpu: 2000m
+          memory: 4Gi
+  volumes:
+    - emptyDir: {}
+      name: volume-0
+  affinity:
+    nodeAffinity:
+      requiredDuringSchedulingIgnoredDuringExecution:
+        nodeSelectorTerms:
+          - matchExpressions:
+              - key: kubernetes.io/arch
+                operator: In
+                values:
+                  - amd64
diff --git a/pipelines/pingcap/ticdc/latest/pod-pull_cdc_storage_integration_heavy.yaml b/pipelines/pingcap/ticdc/latest/pod-pull_cdc_storage_integration_heavy.yaml
new file mode 100644
index 0000000000..5703efbd29
--- /dev/null
+++ b/pipelines/pingcap/ticdc/latest/pod-pull_cdc_storage_integration_heavy.yaml
@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Pod
+spec:
+  securityContext:
+    fsGroup: 1000
+  containers:
+    - name: golang
+      image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest"
+      tty: true
+      resources:
+        limits:
+          memory: 16Gi
+          cpu: "12"
+    - name: net-tool
+      image: hub.pingcap.net/jenkins/network-multitool
+      tty: true
+      resources:
+        limits:
+          memory: 128Mi
+          cpu: 600m
+  affinity:
+    nodeAffinity:
+      requiredDuringSchedulingIgnoredDuringExecution:
+        nodeSelectorTerms:
+          - matchExpressions:
+              - key: kubernetes.io/arch
+                operator: In
+                values:
+                  - amd64
diff --git a/pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy b/pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
new file mode 100644
index 0000000000..c850dacd64
--- /dev/null
+++ b/pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
@@ -0,0 +1,166 @@
+// REF: https://www.jenkins.io/doc/book/pipeline/syntax/#declarative-pipeline
+// Keep small than 400 lines: https://issues.jenkins.io/browse/JENKINS-37984
+// should triggerd for master branches
+@Library('tipipeline') _
+
+final K8S_NAMESPACE = "jenkins-tiflow"
+final GIT_FULL_REPO_NAME = 'pingcap/ticdc'
+final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
+final POD_TEMPLATE_FILE = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml'
+final POD_TEMPLATE_FILE_BUILD = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_integration_build.yaml'
+final REFS = readJSON(text: params.JOB_SPEC).refs
+
+pipeline {
+    agent {
+        kubernetes {
+            namespace K8S_NAMESPACE
+            yamlFile POD_TEMPLATE_FILE_BUILD
+            defaultContainer 'golang'
+        }
+    }
+    environment {
+        FILE_SERVER_URL = 'http://fileserver.pingcap.net'
+    }
+    options {
+        timeout(time: 65, unit: 'MINUTES')
+        parallelsAlwaysFailFast()
+    }
+    stages {
+        stage('Debug info') {
+            steps {
+                sh label: 'Debug info', script: """
+                    printenv
+                    echo "-------------------------"
+                    go env
+                    echo "-------------------------"
+                    echo "debug command: kubectl -n ${K8S_NAMESPACE} exec -ti ${NODE_NAME} bash"
+                """
+                container(name: 'net-tool') {
+                    sh 'dig github.com'
+                    script {
+                        prow.setPRDescription(REFS)
+                    }
+                }
+            }
+        }
+        stage('Checkout') {
+            options { timeout(time: 10, unit: 'MINUTES') }
+            steps {
+                dir("ticdc") {
+                    cache(path: "./", includes: '**/*', key: prow.getCacheKey('git', REFS), restoreKeys: prow.getRestoreKeys('git', REFS)) {
+                        retry(2) {
+                            script {
+                                prow.checkoutRefs(REFS)
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        stage("prepare") {
+            options { timeout(time: 20, unit: 'MINUTES') }
+            steps {
+                dir("third_party_download") {
+                    script {
+                        def tidbBranch = component.computeBranchFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def pdBranch = component.computeBranchFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tikvBranch = component.computeBranchFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tiflashBranch = component.computeBranchFromPR('tiflash', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        retry(2) {
+                            sh label: "download third_party", script: """
+                                export TIDB_BRANCH=${tidbBranch}
+                                export PD_BRANCH=${pdBranch}
+                                export TIKV_BRANCH=${tikvBranch}
+                                export TIFLASH_BRANCH=${tiflashBranch}
+                                cd ../ticdc && ./tests/scripts/download-integration-test-binaries.sh ${REFS.base_ref} && ls -alh ./bin
+                                make check_third_party_binary
+                                cd - && mkdir -p bin && mv ../ticdc/bin/* ./bin/
+                                ls -alh ./bin
+                                ./bin/tidb-server -V
+                                ./bin/pd-server -V
+                                ./bin/tikv-server -V
+                                ./bin/tiflash --version
+                            """
+                        } 
+                    }
+                }
+                dir("ticdc") {
+                    cache(path: "./bin", includes: '**/*', key: prow.getCacheKey('binary', REFS, 'cdc-integration-test')) {
+                        // build cdc, kafka_consumer, storage_consumer, cdc.test for integration test
+                        // only build binarys if not exist, use the cached binarys if exist
+                        sh label: "prepare", script: """
+                            ls -alh ./bin
+                            [ -f ./bin/cdc ] || make cdc
+                            [ -f ./bin/cdc_kafka_consumer ] || make kafka_consumer
+                            [ -f ./bin/cdc_storage_consumer ] || make storage_consumer
+                            [ -f ./bin/cdc.test ] || make integration_test_build
+                            ls -alh ./bin
+                            ./bin/cdc version
+                        """
+                    }
+                    cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") { 
+                        sh label: "prepare", script: """
+                            cp -r ../third_party_download/bin/* ./bin/
+                            ls -alh ./bin
+                        """
+                    }
+                }
+            }
+        }
+
+        stage('Tests') {
+            matrix {
+                axes {
+                    axis {
+                        name 'TEST_GROUP'
+                        values 'G00', 'G01', 'G02', 'G03', 'G04', 'G05', 'G06',  'G07', 'G08',
+                            'G09', 'G10', 'G11', 'G12', 'G13', 'G14', 'G15'
+                    }
+                }
+                agent{
+                    kubernetes {
+                        namespace K8S_NAMESPACE
+                        yamlFile POD_TEMPLATE_FILE
+                        defaultContainer 'golang'
+                    }
+                } 
+                stages {
+                    stage("Test") {
+                        options { timeout(time: 45, unit: 'MINUTES') }
+                        steps {
+                            dir('ticdc') {
+                                cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") {
+                                    container("kafka") {
+                                        timeout(time: 6, unit: 'MINUTES') {
+                                            sh label: "Waiting for kafka ready", script: """
+                                                echo "Waiting for zookeeper to be ready..."
+                                                while ! nc -z localhost 2181; do sleep 10; done
+                                                echo "Waiting for kafka to be ready..."
+                                                while ! nc -z localhost 9092; do sleep 10; done
+                                                echo "Waiting for kafka-broker to be ready..."
+                                                while ! echo dump | nc localhost 2181 | grep brokers | awk '{\$1=\$1;print}' | grep -F -w "/brokers/ids/1"; do sleep 10; done
+                                            """
+                                        }
+                                    }
+                                    sh label: "${TEST_GROUP}", script: """
+                                        ./tests/integration_tests/run_heavy_it_in_ci.sh kafka ${TEST_GROUP}
+                                    """
+                                }
+                            }
+                        }
+                        post {
+                            failure {
+                                sh label: "collect logs", script: """
+                                    ls /tmp/tidb_cdc_test/
+                                    tar --warning=no-file-changed  -cvzf log-${TEST_GROUP}.tar.gz \$(find /tmp/tidb_cdc_test/ -type f -name "*.log")    
+                                    ls -alh  log-${TEST_GROUP}.tar.gz  
+                                """
+                                archiveArtifacts artifacts: "log-${TEST_GROUP}.tar.gz", fingerprint: true 
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    }
+}
diff --git a/pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy b/pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy
new file mode 100644
index 0000000000..22e3287e39
--- /dev/null
+++ b/pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy
@@ -0,0 +1,154 @@
+// REF: https://www.jenkins.io/doc/book/pipeline/syntax/#declarative-pipeline
+// Keep small than 400 lines: https://issues.jenkins.io/browse/JENKINS-37984
+// should triggerd for master branches
+@Library('tipipeline') _
+
+final K8S_NAMESPACE = "jenkins-tiflow"
+final GIT_FULL_REPO_NAME = 'pingcap/ticdc'
+final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
+final POD_TEMPLATE_FILE = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_storage_integration_heavy.yaml'
+final POD_TEMPLATE_FILE_BUILD = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_integration_build.yaml'
+final REFS = readJSON(text: params.JOB_SPEC).refs
+
+pipeline {
+    agent {
+        kubernetes {
+            namespace K8S_NAMESPACE
+            yamlFile POD_TEMPLATE_FILE_BUILD
+            defaultContainer 'golang'
+        }
+    }
+    environment {
+        FILE_SERVER_URL = 'http://fileserver.pingcap.net'
+    }
+    options {
+        timeout(time: 60, unit: 'MINUTES')
+        parallelsAlwaysFailFast()
+    }
+    stages {
+        stage('Debug info') {
+            steps {
+                sh label: 'Debug info', script: """
+                    printenv
+                    echo "-------------------------"
+                    go env
+                    echo "-------------------------"
+                    echo "debug command: kubectl -n ${K8S_NAMESPACE} exec -ti ${NODE_NAME} bash"
+                """
+                container(name: 'net-tool') {
+                    sh 'dig github.com'
+                    script {
+                        prow.setPRDescription(REFS)
+                    }
+                }
+            }
+        }
+        stage('Checkout') {
+            options { timeout(time: 10, unit: 'MINUTES') }
+            steps {
+                dir("ticdc") {
+                    cache(path: "./", includes: '**/*', key: prow.getCacheKey('git', REFS), restoreKeys: prow.getRestoreKeys('git', REFS)) {
+                        retry(2) {
+                            script {
+                                prow.checkoutRefs(REFS)
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        stage("prepare") {
+            options { timeout(time: 20, unit: 'MINUTES') }
+            steps {
+                dir("third_party_download") {
+                    script {
+                        def tidbBranch = component.computeBranchFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def pdBranch = component.computeBranchFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tikvBranch = component.computeBranchFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tiflashBranch = component.computeBranchFromPR('tiflash', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        retry(2) {
+                            sh label: "download third_party", script: """
+                                export TIDB_BRANCH=${tidbBranch}
+                                export PD_BRANCH=${pdBranch}
+                                export TIKV_BRANCH=${tikvBranch}
+                                export TIFLASH_BRANCH=${tiflashBranch}
+                                cd ../ticdc && ./tests/scripts/download-integration-test-binaries.sh ${REFS.base_ref} && ls -alh ./bin
+                                make check_third_party_binary
+                                cd - && mkdir -p bin && mv ../ticdc/bin/* ./bin/
+                                ls -alh ./bin
+                                ./bin/tidb-server -V
+                                ./bin/pd-server -V
+                                ./bin/tikv-server -V
+                                ./bin/tiflash --version
+                            """
+                        }
+                    }
+                }
+                dir("ticdc") {
+                    cache(path: "./bin", includes: '**/*', key: prow.getCacheKey('binary', REFS, 'cdc-storage-integration')) { 
+                        // build cdc, kafka_consumer, storage_consumer, cdc.test for integration test
+                        // only build binarys if not exist, use the cached binarys if exist
+                        sh label: "prepare", script: """
+                            ls -alh ./bin
+                            [ -f ./bin/cdc ] || make cdc
+                            [ -f ./bin/cdc_kafka_consumer ] || make kafka_consumer
+                            [ -f ./bin/cdc_storage_consumer ] || make storage_consumer
+                            [ -f ./bin/cdc.test ] || make integration_test_build
+                            ls -alh ./bin
+                            ./bin/cdc version
+                        """
+                    }
+                    cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") { 
+                        sh label: "prepare", script: """
+                            cp -r ../third_party_download/bin/* ./bin/
+                            ls -alh ./bin
+                        """
+                    }
+                }
+            }
+        }
+
+        stage('Tests') {
+            matrix {
+                axes {
+                    axis {
+                        name 'TEST_GROUP'
+                        values 'G00', 'G01', 'G02', 'G03', 'G04', 'G05', 'G06',  'G07', 'G08', 'G09',
+                            'G10', 'G11', 'G12', 'G13', 'G14', 'G15'
+                    }
+                }
+                agent{
+                    kubernetes {
+                        namespace K8S_NAMESPACE
+                        yamlFile POD_TEMPLATE_FILE
+                        defaultContainer 'golang'
+                    }
+                } 
+                stages {
+                    stage("Test") {
+                        options { timeout(time: 40, unit: 'MINUTES') }
+                        steps {
+                            dir('ticdc') {
+                                cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") {
+                                    sh label: "${TEST_GROUP}", script: """
+                                        ./tests/integration_tests/run_heavy_it_in_ci.sh storage ${TEST_GROUP}
+                                    """
+                                }
+                            }
+                        }
+                        post {
+                            failure {
+                                sh label: "collect logs", script: """
+                                    ls /tmp/tidb_cdc_test/
+                                    tar --warning=no-file-changed  -cvzf log-${TEST_GROUP}.tar.gz \$(find /tmp/tidb_cdc_test/ -type f -name "*.log")    
+                                    ls -alh  log-${TEST_GROUP}.tar.gz  
+                                """
+                                archiveArtifacts artifacts: "log-${TEST_GROUP}.tar.gz", fingerprint: true 
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    }
+}
diff --git a/prow-jobs/pingcap/ticdc/latest-presubmits.yaml b/prow-jobs/pingcap/ticdc/latest-presubmits.yaml
index 31c842a696..2db6a3f0f6 100644
--- a/prow-jobs/pingcap/ticdc/latest-presubmits.yaml
+++ b/prow-jobs/pingcap/ticdc/latest-presubmits.yaml
@@ -38,12 +38,24 @@ presubmits:
       always_run: false
       skip_if_only_changed: *skip_if_only_changed
       context: pull-cdc-kafka-integration-light
-      optional: true # TODO: change this after the job is ready
+      optional: false
       skip_report: false
       trigger: "(?m)^/test (?:.*? )?(pull-cdc-kafka-integration-light)(?: .*?)?$"
       rerun_command: "/test pull-cdc-kafka-integration-light"
       branches: *branches
 
+    - name: pingcap/ticdc/pull_cdc_kafka_integration_heavy
+      agent: jenkins
+      decorate: false # need add this.
+      always_run: false
+      skip_if_only_changed: *skip_if_only_changed
+      context: pull-cdc-kafka-integration-heavy
+      optional: false
+      skip_report: false
+      trigger: "(?m)^/test (?:.*? )?(pull-cdc-kafka-integration-heavy)(?: .*?)?$"
+      rerun_command: "/test pull-cdc-kafka-integration-heavy"
+      branches: *branches
+
     - name: pingcap/ticdc/pull_cdc_pulsar_integration_light
       agent: jenkins
       decorate: false # need add this.
@@ -68,6 +80,18 @@ presubmits:
       rerun_command: "/test pull-cdc-storage-integration-light"
       branches: *branches
 
+    - name: pingcap/ticdc/pull_cdc_storage_integration_heavy
+      agent: jenkins
+      decorate: false # need add this.
+      always_run: false
+      skip_if_only_changed: *skip_if_only_changed
+      context: pull-cdc-storage-integration-heavy
+      optional: false
+      skip_report: false
+      trigger: "(?m)^/test (?:.*? )?(pull-cdc-storage-integration-heavy)(?: .*?)?$"
+      rerun_command: "/test pull-cdc-storage-integration-heavy"
+      branches: *branches
+
     - name: pull-unit-test
       cluster: gcp-prow-ksyun
       decorate: true # need add this.


Reviews:
user: login: gemini-code-assist[bot]
body: Hello @wk989898, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

## Summary of Changes
Hello team, gemini-code-assist here to provide a summary of this pull request. This PR, titled "ticdc: add kafka integration test and make it ready", focuses on integrating and enabling Kafka integration tests within the TiCDC Prow CI system. The primary change involves modifying the Prow job configuration file to enable an existing 'light' Kafka integration test (by changing its `optional` status from `true` to `false`) and adding a new configuration for a 'heavy' Kafka integration test, also marked as non-optional. This makes both light and heavy Kafka integration tests mandatory checks for relevant PRs in the future, ensuring better test coverage for the Kafka sink.

### Highlights
* **CI Configuration Update**: The Prow job configuration file for TiCDC presubmits (`latest-presubmits.yaml`) has been updated to include and enable Kafka integration tests.
* **Enable Light Kafka Test**: The existing `pull-cdc-kafka-integration-light` job is changed from `optional: true` to `optional: false`, making it a required check.
* **Add Heavy Kafka Test**: A new Prow job configuration, `pull-cdc-kafka-integration-heavy`, is added and configured as a non-optional (`optional: false`) test.

### Changelog
* **prow-jobs/pingcap/ticdc/latest-presubmits.yaml**
  * Changed the `optional` field for the `pull-cdc-kafka-integration-light` job from `true` to `false` (line 41).
  * Added a new job definition for `pingcap/ticdc/pull_cdc_kafka_integration_heavy`, including its agent, run conditions, context, optional status (`false`), trigger, rerun command, and branches (lines 47-57).


<details>

<summary><b>Using Gemini Code Assist</b></summary>
<br>

The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.

<b>Invoking Gemini</b>

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands.

Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments.
Help | `/gemini help` | Displays a list of available commands.

<b>Customization</b>

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).

<b>Limitations & Feedback</b>

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments to provide feedback.

</details>

<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>


[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/review-github-code). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).

---

> CI runs tests,
> Kafka streams flow fast,
> Green checks appear.


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T10:26:24Z
user: login: gemini-code-assist[bot]
body: ## Code Review
This pull request updates the Prow presubmit configuration for TiCDC. It makes the `pull-cdc-kafka-integration-light` job mandatory by changing `optional: true` to `optional: false` and removing the associated `TODO` comment. Additionally, it introduces a new job, `pingcap/ticdc/pull_cdc_kafka_integration_heavy`.

The changes are generally well-structured and follow existing patterns in the configuration file. The primary concern revolves around the new heavy integration test being set as mandatory, which could impact CI performance. The PR description is currently empty; adding context for these changes, especially the rationale for making a heavy test mandatory, would be very helpful.

### Summary of Findings
* **Mandatory Heavy Integration Test**: The newly added `pull_cdc_kafka_integration_heavy` test is configured as mandatory (`optional: false`). This is a potential concern as heavy tests can significantly increase CI validation times and resource consumption for every pull request. Clarification on this decision is recommended.
* **PR Description Context**: The pull request description is empty. For changes that affect CI configuration, especially those introducing new mandatory heavy tests, providing context and rationale in the PR description is highly beneficial for reviewers and future maintainers. (Severity: low, not commented due to review settings)
* **Informal Comment Style**: The comment `# need add this.` associated with `decorate: false` is present in the new job definition, consistent with other existing jobs. While consistent, this comment is informal and doesn't add much clarity. Consider standardizing comments or removing them if `decorate: false` is a well-understood default for these Jenkins agent jobs. (Severity: low, not commented due to review settings)

### Merge Readiness
The pull request introduces a new heavy Kafka integration test and makes the light version mandatory. While the configuration is structurally sound, the decision to make the `pull_cdc_kafka_integration_heavy` test mandatory (`optional: false`) raises concerns about potential impacts on CI performance and PR turnaround times. 

I recommend that this high-severity concern be addressed by either providing a clear justification for this setting or by changing it to `optional: true` if appropriate. Therefore, I suggest that changes are made before this PR is merged. As a reviewer, I am not authorized to approve the pull request; further review and approval from authorized maintainers will be necessary.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T10:28:49Z
user: login: purelind
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T06:19:18Z
user: login: purelind
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T06:51:31Z
user: login: purelind
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T06:54:52Z
user: login: wk989898
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T07:05:54Z
user: login: purelind
body: /lgtm
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T07:11:39Z

Review comments:
diff_hunk: @@ -38,12 +38,24 @@ presubmits:
       always_run: false
       skip_if_only_changed: *skip_if_only_changed
       context: pull-cdc-kafka-integration-light
-      optional: true # TODO: change this after the job is ready
+      optional: false
       skip_report: false
       trigger: "(?m)^/test (?:.*? )?(pull-cdc-kafka-integration-light)(?: .*?)?$"
       rerun_command: "/test pull-cdc-kafka-integration-light"
       branches: *branches
 
+    - name: pingcap/ticdc/pull_cdc_kafka_integration_heavy
+      agent: jenkins
+      decorate: false # need add this.
+      always_run: false
+      skip_if_only_changed: *skip_if_only_changed
+      context: pull-cdc-kafka-integration-heavy
+      optional: false
path: prow-jobs/pingcap/ticdc/latest-presubmits.yaml
user: login: gemini-code-assist[bot]
body: ![high](https://img.shields.io/badge/priority-high-orange)

The new `pull_cdc_kafka_integration_heavy` job is configured with `optional: false`, making it a mandatory check for pull requests.

Heavy integration tests are typically resource-intensive and can significantly increase the time it takes to validate a PR. Could you please clarify the reasoning behind making this heavy test mandatory by default? 

If this test is intended for specific scenarios or manual triggering, setting `optional: true` might be more appropriate. This would allow it to be run when needed (e.g., via `/test pull-cdc-kafka-integration-heavy`) without impacting all PRs.
created_at: 2025-05-15T10:28:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -38,12 +38,24 @@ presubmits:
       always_run: false
       skip_if_only_changed: *skip_if_only_changed
       context: pull-cdc-kafka-integration-light
-      optional: true # TODO: change this after the job is ready
+      optional: false
       skip_report: false
       trigger: "(?m)^/test (?:.*? )?(pull-cdc-kafka-integration-light)(?: .*?)?$"
       rerun_command: "/test pull-cdc-kafka-integration-light"
       branches: *branches
 
+    - name: pingcap/ticdc/pull_cdc_kafka_integration_heavy
path: prow-jobs/pingcap/ticdc/latest-presubmits.yaml
user: login: purelind
body: Please add the following files based on pull_cdc_kafka_integration_light testing.
1. job DSL file https://github.com/PingCAP-QE/ci/blob/main/jobs/pingcap/ticdc/latest/pull_cdc_kafka_integration_light.groovy
2. pod file https://github.com/PingCAP-QE/ci/blob/main/pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_light.yaml
3. pipeline file https://github.com/PingCAP-QE/ci/blob/main/pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_light.groovy
created_at: 2025-05-16T06:19:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,166 @@
+// REF: https://www.jenkins.io/doc/book/pipeline/syntax/#declarative-pipeline
+// Keep small than 400 lines: https://issues.jenkins.io/browse/JENKINS-37984
+// should triggerd for master branches
+@Library('tipipeline') _
+
+final K8S_NAMESPACE = "jenkins-tiflow"
+final GIT_FULL_REPO_NAME = 'pingcap/ticdc'
+final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
+final POD_TEMPLATE_FILE = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml'
+final POD_TEMPLATE_FILE_BUILD = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_integration_build.yaml'
+final REFS = readJSON(text: params.JOB_SPEC).refs
+
+pipeline {
+    agent {
+        kubernetes {
+            namespace K8S_NAMESPACE
+            yamlFile POD_TEMPLATE_FILE_BUILD
+            defaultContainer 'golang'
+        }
+    }
+    environment {
+        FILE_SERVER_URL = 'http://fileserver.pingcap.net'
+    }
+    options {
+        timeout(time: 65, unit: 'MINUTES')
+        parallelsAlwaysFailFast()
+    }
+    stages {
+        stage('Debug info') {
+            steps {
+                sh label: 'Debug info', script: """
+                    printenv
+                    echo "-------------------------"
+                    go env
+                    echo "-------------------------"
+                    echo "debug command: kubectl -n ${K8S_NAMESPACE} exec -ti ${NODE_NAME} bash"
+                """
+                container(name: 'net-tool') {
+                    sh 'dig github.com'
+                    script {
+                        prow.setPRDescription(REFS)
+                    }
+                }
+            }
+        }
+        stage('Checkout') {
+            options { timeout(time: 10, unit: 'MINUTES') }
+            steps {
+                dir("ticdc") {
+                    cache(path: "./", includes: '**/*', key: prow.getCacheKey('git', REFS), restoreKeys: prow.getRestoreKeys('git', REFS)) {
+                        retry(2) {
+                            script {
+                                prow.checkoutRefs(REFS)
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        stage("prepare") {
+            options { timeout(time: 20, unit: 'MINUTES') }
+            steps {
+                dir("third_party_download") {
+                    script {
+                        def tidbBranch = component.computeBranchFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def pdBranch = component.computeBranchFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tikvBranch = component.computeBranchFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tiflashBranch = component.computeBranchFromPR('tiflash', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        retry(2) {
+                            sh label: "download third_party", script: """
+                                export TIDB_BRANCH=${tidbBranch}
+                                export PD_BRANCH=${pdBranch}
+                                export TIKV_BRANCH=${tikvBranch}
+                                export TIFLASH_BRANCH=${tiflashBranch}
+                                cd ../ticdc && ./tests/scripts/download-integration-test-binaries.sh ${REFS.base_ref} && ls -alh ./bin
+                                make check_third_party_binary
+                                cd - && mkdir -p bin && mv ../ticdc/bin/* ./bin/
+                                ls -alh ./bin
+                                ./bin/tidb-server -V
+                                ./bin/pd-server -V
+                                ./bin/tikv-server -V
+                                ./bin/tiflash --version
+                            """
+                        } 
+                    }
+                }
+                dir("ticdc") {
+                    cache(path: "./bin", includes: '**/*', key: prow.getCacheKey('binary', REFS, 'cdc-integration-test')) {
+                        // build cdc, kafka_consumer, storage_consumer, cdc.test for integration test
+                        // only build binarys if not exist, use the cached binarys if exist
+                        sh label: "prepare", script: """
+                            ls -alh ./bin
+                            [ -f ./bin/cdc ] || make cdc
+                            [ -f ./bin/cdc_kafka_consumer ] || make kafka_consumer
+                            [ -f ./bin/cdc_storage_consumer ] || make storage_consumer
+                            [ -f ./bin/cdc.test ] || make integration_test_build
+                            ls -alh ./bin
+                            ./bin/cdc version
+                        """
+                    }
+                    cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") { 
+                        sh label: "prepare", script: """
+                            cp -r ../third_party_download/bin/* ./bin/
+                            ls -alh ./bin
+                        """
+                    }
+                }
+            }
+        }
+
+        stage('Tests') {
+            matrix {
+                axes {
+                    axis {
+                        name 'TEST_GROUP'
path: pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
user: login: purelind
body: Is the heavy-related test also 16 groups? 
created_at: 2025-05-16T06:51:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,166 @@
+// REF: https://www.jenkins.io/doc/book/pipeline/syntax/#declarative-pipeline
+// Keep small than 400 lines: https://issues.jenkins.io/browse/JENKINS-37984
+// should triggerd for master branches
+@Library('tipipeline') _
+
+final K8S_NAMESPACE = "jenkins-tiflow"
+final GIT_FULL_REPO_NAME = 'pingcap/ticdc'
+final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
+final POD_TEMPLATE_FILE = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml'
+final POD_TEMPLATE_FILE_BUILD = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_integration_build.yaml'
+final REFS = readJSON(text: params.JOB_SPEC).refs
+
+pipeline {
+    agent {
+        kubernetes {
+            namespace K8S_NAMESPACE
+            yamlFile POD_TEMPLATE_FILE_BUILD
+            defaultContainer 'golang'
+        }
+    }
+    environment {
+        FILE_SERVER_URL = 'http://fileserver.pingcap.net'
+    }
+    options {
+        timeout(time: 65, unit: 'MINUTES')
+        parallelsAlwaysFailFast()
+    }
+    stages {
+        stage('Debug info') {
+            steps {
+                sh label: 'Debug info', script: """
+                    printenv
+                    echo "-------------------------"
+                    go env
+                    echo "-------------------------"
+                    echo "debug command: kubectl -n ${K8S_NAMESPACE} exec -ti ${NODE_NAME} bash"
+                """
+                container(name: 'net-tool') {
+                    sh 'dig github.com'
+                    script {
+                        prow.setPRDescription(REFS)
+                    }
+                }
+            }
+        }
+        stage('Checkout') {
+            options { timeout(time: 10, unit: 'MINUTES') }
+            steps {
+                dir("ticdc") {
+                    cache(path: "./", includes: '**/*', key: prow.getCacheKey('git', REFS), restoreKeys: prow.getRestoreKeys('git', REFS)) {
+                        retry(2) {
+                            script {
+                                prow.checkoutRefs(REFS)
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        stage("prepare") {
+            options { timeout(time: 20, unit: 'MINUTES') }
+            steps {
+                dir("third_party_download") {
+                    script {
+                        def tidbBranch = component.computeBranchFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def pdBranch = component.computeBranchFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tikvBranch = component.computeBranchFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tiflashBranch = component.computeBranchFromPR('tiflash', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        retry(2) {
+                            sh label: "download third_party", script: """
+                                export TIDB_BRANCH=${tidbBranch}
+                                export PD_BRANCH=${pdBranch}
+                                export TIKV_BRANCH=${tikvBranch}
+                                export TIFLASH_BRANCH=${tiflashBranch}
+                                cd ../ticdc && ./tests/scripts/download-integration-test-binaries.sh ${REFS.base_ref} && ls -alh ./bin
+                                make check_third_party_binary
+                                cd - && mkdir -p bin && mv ../ticdc/bin/* ./bin/
+                                ls -alh ./bin
+                                ./bin/tidb-server -V
+                                ./bin/pd-server -V
+                                ./bin/tikv-server -V
+                                ./bin/tiflash --version
+                            """
+                        } 
+                    }
+                }
+                dir("ticdc") {
+                    cache(path: "./bin", includes: '**/*', key: prow.getCacheKey('binary', REFS, 'cdc-integration-test')) {
+                        // build cdc, kafka_consumer, storage_consumer, cdc.test for integration test
+                        // only build binarys if not exist, use the cached binarys if exist
+                        sh label: "prepare", script: """
+                            ls -alh ./bin
+                            [ -f ./bin/cdc ] || make cdc
+                            [ -f ./bin/cdc_kafka_consumer ] || make kafka_consumer
+                            [ -f ./bin/cdc_storage_consumer ] || make storage_consumer
+                            [ -f ./bin/cdc.test ] || make integration_test_build
+                            ls -alh ./bin
+                            ./bin/cdc version
+                        """
+                    }
+                    cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") { 
+                        sh label: "prepare", script: """
+                            cp -r ../third_party_download/bin/* ./bin/
+                            ls -alh ./bin
+                        """
+                    }
+                }
+            }
+        }
+
+        stage('Tests') {
+            matrix {
+                axes {
+                    axis {
+                        name 'TEST_GROUP'
+                        values 'G00', 'G01', 'G02', 'G03', 'G04', 'G05', 'G06',  'G07', 'G08',
+                            'G09', 'G10', 'G11', 'G12', 'G13', 'G14', 'G15'
+                    }
+                }
+                agent{
+                    kubernetes {
+                        namespace K8S_NAMESPACE
+                        yamlFile POD_TEMPLATE_FILE
+                        defaultContainer 'golang'
+                    }
+                } 
+                stages {
+                    stage("Test") {
+                        options { timeout(time: 45, unit: 'MINUTES') }
+                        steps {
+                            dir('ticdc') {
+                                cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") {
+                                    container("kafka") {
+                                        timeout(time: 6, unit: 'MINUTES') {
+                                            sh label: "Waiting for kafka ready", script: """
+                                                echo "Waiting for zookeeper to be ready..."
+                                                while ! nc -z localhost 2181; do sleep 10; done
+                                                echo "Waiting for kafka to be ready..."
+                                                while ! nc -z localhost 9092; do sleep 10; done
+                                                echo "Waiting for kafka-broker to be ready..."
+                                                while ! echo dump | nc localhost 2181 | grep brokers | awk '{\$1=\$1;print}' | grep -F -w "/brokers/ids/1"; do sleep 10; done
+                                            """
+                                        }
+                                    }
+                                    sh label: "${TEST_GROUP}", script: """
+                                        ./tests/integration_tests/run_light_it_in_ci.sh kafka ${TEST_GROUP}
path: pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
user: login: purelind
body: Is the script name here `run_light_it_in_ci ` correct?
created_at: 2025-05-16T06:54:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,166 @@
+// REF: https://www.jenkins.io/doc/book/pipeline/syntax/#declarative-pipeline
+// Keep small than 400 lines: https://issues.jenkins.io/browse/JENKINS-37984
+// should triggerd for master branches
+@Library('tipipeline') _
+
+final K8S_NAMESPACE = "jenkins-tiflow"
+final GIT_FULL_REPO_NAME = 'pingcap/ticdc'
+final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
+final POD_TEMPLATE_FILE = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml'
+final POD_TEMPLATE_FILE_BUILD = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_integration_build.yaml'
+final REFS = readJSON(text: params.JOB_SPEC).refs
+
+pipeline {
+    agent {
+        kubernetes {
+            namespace K8S_NAMESPACE
+            yamlFile POD_TEMPLATE_FILE_BUILD
+            defaultContainer 'golang'
+        }
+    }
+    environment {
+        FILE_SERVER_URL = 'http://fileserver.pingcap.net'
+    }
+    options {
+        timeout(time: 65, unit: 'MINUTES')
+        parallelsAlwaysFailFast()
+    }
+    stages {
+        stage('Debug info') {
+            steps {
+                sh label: 'Debug info', script: """
+                    printenv
+                    echo "-------------------------"
+                    go env
+                    echo "-------------------------"
+                    echo "debug command: kubectl -n ${K8S_NAMESPACE} exec -ti ${NODE_NAME} bash"
+                """
+                container(name: 'net-tool') {
+                    sh 'dig github.com'
+                    script {
+                        prow.setPRDescription(REFS)
+                    }
+                }
+            }
+        }
+        stage('Checkout') {
+            options { timeout(time: 10, unit: 'MINUTES') }
+            steps {
+                dir("ticdc") {
+                    cache(path: "./", includes: '**/*', key: prow.getCacheKey('git', REFS), restoreKeys: prow.getRestoreKeys('git', REFS)) {
+                        retry(2) {
+                            script {
+                                prow.checkoutRefs(REFS)
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        stage("prepare") {
+            options { timeout(time: 20, unit: 'MINUTES') }
+            steps {
+                dir("third_party_download") {
+                    script {
+                        def tidbBranch = component.computeBranchFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def pdBranch = component.computeBranchFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tikvBranch = component.computeBranchFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        def tiflashBranch = component.computeBranchFromPR('tiflash', REFS.base_ref, REFS.pulls[0].title, 'master')
+                        retry(2) {
+                            sh label: "download third_party", script: """
+                                export TIDB_BRANCH=${tidbBranch}
+                                export PD_BRANCH=${pdBranch}
+                                export TIKV_BRANCH=${tikvBranch}
+                                export TIFLASH_BRANCH=${tiflashBranch}
+                                cd ../ticdc && ./tests/scripts/download-integration-test-binaries.sh ${REFS.base_ref} && ls -alh ./bin
+                                make check_third_party_binary
+                                cd - && mkdir -p bin && mv ../ticdc/bin/* ./bin/
+                                ls -alh ./bin
+                                ./bin/tidb-server -V
+                                ./bin/pd-server -V
+                                ./bin/tikv-server -V
+                                ./bin/tiflash --version
+                            """
+                        } 
+                    }
+                }
+                dir("ticdc") {
+                    cache(path: "./bin", includes: '**/*', key: prow.getCacheKey('binary', REFS, 'cdc-integration-test')) {
+                        // build cdc, kafka_consumer, storage_consumer, cdc.test for integration test
+                        // only build binarys if not exist, use the cached binarys if exist
+                        sh label: "prepare", script: """
+                            ls -alh ./bin
+                            [ -f ./bin/cdc ] || make cdc
+                            [ -f ./bin/cdc_kafka_consumer ] || make kafka_consumer
+                            [ -f ./bin/cdc_storage_consumer ] || make storage_consumer
+                            [ -f ./bin/cdc.test ] || make integration_test_build
+                            ls -alh ./bin
+                            ./bin/cdc version
+                        """
+                    }
+                    cache(path: "./", includes: '**/*', key: "ws/${BUILD_TAG}/ticdc") { 
+                        sh label: "prepare", script: """
+                            cp -r ../third_party_download/bin/* ./bin/
+                            ls -alh ./bin
+                        """
+                    }
+                }
+            }
+        }
+
+        stage('Tests') {
+            matrix {
+                axes {
+                    axis {
+                        name 'TEST_GROUP'
path: pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
user: login: wk989898
body: yes
created_at: 2025-05-16T07:05:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: ti-chi-bot[bot]
created_at: 2025-05-15T10:26:13Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**
The pull request adds a Kafka integration test and prepares it for execution in the `latest-presubmits.yaml` file for `ticdc` repository. The changes set the `optional` field to `false` for the Kafka integration test job and add a new job for heavy Kafka integration testing. Overall, the PR seems to enhance the testing capabilities of the `ticdc` repository.

**Critical Issues:**
- No critical issues observed in the provided diff.

**Code Improvements:**
- In the `latest-presubmits.yaml` file, the comment "# need add this." is not clear and can be improved for better readability:
  ```diff
  -      decorate: false # need add this.
  +      decorate: false # Add decorator for better visualization
  ```
  
**Best Practices:**
- Add a comment to explain the purpose of the new job `pull_cdc_kafka_integration_heavy` for clarity:
  ```diff
  +      # Heavy Kafka integration test job
  +      name: pingcap/ticdc/pull_cdc_kafka_integration_heavy
  ```

Overall, the changes look good with minor improvements for clarity and readability. It's recommended to address the above suggestions for better maintainability and understanding of the codebase.
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-15T10:28:53Z
body: @gemini-code-assist[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

<details>

In response to [this](https://github.com/PingCAP-QE/ci/pull/3522#pullrequestreview-2843165763):

>## Code Review
>This pull request updates the Prow presubmit configuration for TiCDC. It makes the `pull-cdc-kafka-integration-light` job mandatory by changing `optional: true` to `optional: false` and removing the associated `TODO` comment. Additionally, it introduces a new job, `pingcap/ticdc/pull_cdc_kafka_integration_heavy`.
>
>The changes are generally well-structured and follow existing patterns in the configuration file. The primary concern revolves around the new heavy integration test being set as mandatory, which could impact CI performance. The PR description is currently empty; adding context for these changes, especially the rationale for making a heavy test mandatory, would be very helpful.
>
>### Summary of Findings
>* **Mandatory Heavy Integration Test**: The newly added `pull_cdc_kafka_integration_heavy` test is configured as mandatory (`optional: false`). This is a potential concern as heavy tests can significantly increase CI validation times and resource consumption for every pull request. Clarification on this decision is recommended.
>* **PR Description Context**: The pull request description is empty. For changes that affect CI configuration, especially those introducing new mandatory heavy tests, providing context and rationale in the PR description is highly beneficial for reviewers and future maintainers. (Severity: low, not commented due to review settings)
>* **Informal Comment Style**: The comment `# need add this.` associated with `decorate: false` is present in the new job definition, consistent with other existing jobs. While consistent, this comment is informal and doesn't add much clarity. Consider standardizing comments or removing them if `decorate: false` is a well-understood default for these Jenkins agent jobs. (Severity: low, not commented due to review settings)
>
>### Merge Readiness
>The pull request introduces a new heavy Kafka integration test and makes the light version mandatory. While the configuration is structurally sound, the decision to make the `pull_cdc_kafka_integration_heavy` test mandatory (`optional: false`) raises concerns about potential impacts on CI performance and PR turnaround times. 
>
>I recommend that this high-severity concern be addressed by either providing a clear justification for this setting or by changing it to `optional: true` if appropriate. Therefore, I suggest that changes are made before this PR is merged. As a reviewer, I am not authorized to approve the pull request; further review and approval from authorized maintainers will be necessary.


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T06:42:20Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**
The pull request adds a Kafka integration test for TiCDC and prepares the job configurations related to this test. Changes include adding a new Jenkins pipeline job and updating the `latest-presubmits.yaml` file. Overall, the PR aims to enhance testing capabilities for TiCDC with Kafka integration tests.

**Critical Issues:**
- **File:** `prow-jobs/pingcap/ticdc/latest-presubmits.yaml`
  - **Line 38:**
    - The `optional` field is changed to `false` without proper validation or confirmation that the job is indeed ready. This change can potentially force the job to run even if it's not fully prepared, causing unexpected failures or issues.
    - **Suggested Solution:** Ensure the job is truly ready before setting `optional` to `false` to avoid running incomplete or unstable tests. Consider adding a validation step or confirming readiness elsewhere in the process.

**Code Improvements:**
- **File:** `jobs/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy`
  - Consider adding more descriptive comments or documentation within the Jenkins pipeline job configuration to explain the purpose and functionality of different parameters and settings for better maintainability and understanding by other team members.

**Best Practices:**
- **File:** `jobs/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy`
  - Add detailed comments explaining the purpose of each section of the pipeline job configuration, such as parameters, properties, and SCM settings, to improve readability and maintainability.
- **Testing Coverage:**
  - Ensure that sufficient testing is in place for the newly added Kafka integration test to cover different scenarios and edge cases thoroughly. Update testing documentation if necessary to reflect the new test cases.
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T06:45:19Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**
This PR adds a Kafka integration test for `ticdc` and prepares the necessary configurations for the test. The changes include adding a Jenkins pipeline job, a Kubernetes pod configuration, and updating the presubmit configuration for the test. The overall quality of the changes seems good, focusing on integration testing improvements.

**Critical Issues:**
- No critical issues found.

**Code Improvements:**
- **Error Handling in pod-pull_cdc_kafka_integration_heavy.yaml:**
  - File: pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml
  - There is a potential issue with error handling in the yaml file. It's recommended to include proper error handling mechanisms or fallback strategies in case any of the containers fail to start or encounter issues during execution.

**Best Practices:**
- **Documentation for Pipeline Job:**
  - File: jobs/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
  - Add comments to describe the purpose of the pipeline job, its stages, and any important configurations. This will help maintainers and developers understand the job's functionality without having to dive deep into the code.
- **Testing Coverage for New Integration Test:**
  - Ensure that the new Kafka integration test is adequately covered by automated tests to validate its functionality and catch any regressions in the future.
- **Naming Convention in Presubmit Configuration:**
  - File: prow-jobs/pingcap/ticdc/latest-presubmits.yaml
  - Consider using consistent naming conventions for the presubmit contexts and jobs to maintain clarity and organization within the configuration file.

These suggestions aim to enhance the maintainability, reliability, and clarity of the added components related to the Kafka integration test for `ticdc`.
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T06:47:10Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**
This pull request introduces heavy integration tests for the `ticdc` project, specifically for Kafka integration. It does so by creating Groovy scripts for Jenkins pipeline jobs and corresponding Kubernetes pod configurations. The pipeline jobs perform steps such as checking out code, preparing binaries, and running tests. The quality of the changes seems generally good, but there are a few areas that could be improved.

1. **Critical Issues:**
   - None identified

2. **Code Improvements:**
   - Hardcoded values: There are numerous hardcoded values in the yaml files (e.g., CPU and memory requests/limits for the containers in the pod, the Kafka configurations, and the containers' images). Making these values configurable would increase flexibility and reusability.
   
   For example, in `pull_cdc_kafka_integration_heavy.groovy`:
   ```groovy
   final K8S_NAMESPACE = "jenkins-tiflow"
   final GIT_FULL_REPO_NAME = 'pingcap/ticdc'
   final GIT_CREDENTIALS_ID = 'github-sre-bot-ssh'
   final POD_TEMPLATE_FILE = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml'
   final POD_TEMPLATE_FILE_BUILD = 'pipelines/pingcap/ticdc/latest/pod-pull_cdc_integration_build.yaml'
   ```
   These values could be passed as parameters or environment variables.

3. **Best Practices:**
   - Error handling: There are several instances where commands are executed without any error handling. This could lead to hard-to-debug failures. Consider adding error handling or at least logging potential error messages.

   For example, in `pull_cdc_kafka_integration_heavy.groovy`:
   ```groovy
   sh label: "Waiting for kafka ready", script: """
       echo "Waiting for zookeeper to be ready..."
       while ! nc -z localhost 2181; do sleep 10; done
       echo "Waiting for kafka to be ready..."
       while ! nc -z localhost 9092; do sleep 10; done
       echo "Waiting for kafka-broker to be ready..."
       while ! echo dump | nc localhost 2181 | grep brokers | awk '{\$1=\$1;print}' | grep -F -w "/brokers/ids/1"; do sleep 10; done
   """
   ```
   These commands could fail for various reasons and it would be useful to have some error messages to go off of.

   - Documentation: The PR lacks sufficient comments explaining the purpose and functionality of the scripts and configuration files. This makes it harder for others to understand and maintain the code. Adding detailed comments would greatly improve the maintainability of the code.
   
   For example, in `pull_cdc_kafka_integration_heavy.groovy`:
   ```groovy
   pipeline {
       agent {
           kubernetes {
               namespace K8S_NAMESPACE
               yamlFile POD_TEMPLATE_FILE_BUILD
               defaultContainer 'golang'
           }
       }
       //...
   }
   ```
   A comment explaining what this pipeline does, why it's needed, and how it functions would be helpful for anyone who needs to maintain or modify this script in the future.
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T06:48:40Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**
This pull request provides a new Kafka integration test for the `ticdc` project, which is part of the PingCAP ecosystem. It includes the creation of new Groovy scripts and Kubernetes configuration files that define the test environment and its execution. The PR also modifies a YAML file that defines the pipeline of presubmit checks to include the new test. The overall quality of the PR is good; however, there are some areas where improvements can be made.

1. **Code Improvements:**

   - `pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy`
     - Line 166: The script has grown quite large and is difficult to understand due to the depth and complexity of the nested stages and steps. This can be improved by breaking down the script into smaller, more manageable functions.
   - `pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml`
     - Lines 156-160: The resource limits for the `canal-adapter` container are commented out due to an Out of Memory issue, which is tracked in a GitHub issue. This should be addressed so that appropriate resource limits can be set.
   - `prow-jobs/pingcap/ticdc/latest-presubmits.yaml`
     - Lines 46-58: The new Kafka integration test is set to always run, which might not be necessary for all changes and could put unnecessary load on the CI system. Consider making it optional or only running it when certain files are changed.

2. **Best Practices:**
   
   - `pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy`
     - There's a comment on line 1 suggesting to keep the file less than 400 lines due to a Jenkins issue. This script is currently 166 lines long, so it's far from this limit, but it's a good practice to keep in mind for future updates. Consider breaking down the script into smaller functions or separate scripts to keep it manageable.
   - `pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml`
     - There is no documentation in this file which makes it hard for others to understand the purpose of each container and the overall design of the test environment. Consider adding comments to explain the role and configuration of each container.

Suggested changes:

```groovy
// pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy
// Break down the 'Tests' stage into smaller functions
def runTest(String group) {
    ...
}

pipeline {
    ...
    stages {
        ...
        stage('Tests') {
            matrix {
                ...
                stages {
                    stage("Test") {
                        steps {
                            runTest(TEST_GROUP)
                        }
                    }
                }
            }
        }
    }
}
```

```yaml
# pipelines/pingcap/ticdc/latest/pod-pull_cdc_kafka_integration_heavy.yaml
# Add documentation for each container
...
containers:
  - image: wurstmeister/zookeeper
    ...
    # This container runs Zookeeper, which is used by Kafka for coordination.
    name: zookeeper
  - args:
      - cat
    image: hub.pingcap.net/jenkins/rocky8_golang-1.23:tini
    ...
    # This container provides a Golang environment for the tests.
    name: golang
...
```
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T06:58:41Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**

The PR aims to add an integration test for the `ticdc` component with Kafka and makes it ready for use. It introduces new Jenkins pipeline jobs and associated Kubernetes configurations for the test environment, including Kafka and other related services. The PR is generally well-structured and doesn't appear to introduce any significant issues.

1. **Critical Issues:**
   - None found.

2. **Code Improvements:**
   - In the newly added `pod-pull_cdc_kafka_integration_heavy.yaml` file, there's a `TODO` comment regarding resource limits for the `canal-adapter` container (lines 89-91). Address this issue as described or remove the comment if it's no longer relevant.
   - In the `pull_cdc_kafka_integration_heavy.groovy` file, there are multiple hardcoded test group values (lines 142-144). It's better to externalize these into a configuration or environment variables for easier maintenance and flexibility.
   
3. **Best Practices:**
   - In `pull_cdc_kafka_integration_heavy.groovy`, it would be helpful to add more comments explaining the different stages of the pipeline and why certain steps are necessary (e.g., why the 'net-tool' container is used).
   - In the `pod-pull_cdc_kafka_integration_heavy.yaml` file, consider parameterizing some of the hard-coded values (like CPU and memory requests/limits) using environment variables or config maps. This will make it easier to adjust these settings in different environments or for different scenarios.
   - In both of the `.groovy` files, it’s unclear what the `// REF: https://<your-jenkins-server>/plugin/job-dsl/api-viewer/index.html` comment refers to. If it’s necessary, replace `<your-jenkins-server>` with the actual URL or provide more context.
   
Suggested changes:
```groovy
// pull_cdc_kafka_integration_heavy.groovy
axis {
    name 'TEST_GROUP'
    values System.getenv('TEST_GROUPS').split(',')
}
```
```yaml
# pod-pull_cdc_kafka_integration_heavy.yaml
resources:
    requests:
        cpu: ${CPU_REQUEST}
        memory: ${MEMORY_REQUEST}
    limits:
        cpu: ${CPU_LIMIT}
        memory: ${MEMORY_LIMIT}
```
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T07:00:46Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**
The PR introduces integration tests for the PingCAP's TiCDC project, specifically for Kafka and storage components. It adds new Jenkins jobs and Kubernetes templates to set up and run the tests. The quality of the PR seems to be good, with no visible critical issues or errors. However, there are areas for improvements and potential issues that need to be addressed.

1. **Critical Issues:** None identified.

2. **Code Improvements:**

- In the `pull_cdc_kafka_integration_heavy.groovy` and `pull_cdc_storage_integration_heavy.groovy` Jenkins pipeline scripts, there's a section that downloads third-party binaries. The branches for these third-party projects are determined by certain rules in `computeBranchFromPR()`. It may lead to problems if the branches computed do not exist or if they are not compatible with the current version of TiCDC being tested. It would be better if the versions/branches of these third-party projects are explicitly defined for each version of TiCDC.
  ```groovy
  def tidbBranch = component.computeBranchFromPR('tidb', REFS.base_ref, REFS.pulls[0].title, 'master')
  def pdBranch = component.computeBranchFromPR('pd', REFS.base_ref, REFS.pulls[0].title, 'master')
  def tikvBranch = component.computeBranchFromPR('tikv', REFS.base_ref, REFS.pulls[0].title, 'master')
  def tiflashBranch = component.computeBranchFromPR('tiflash', REFS.base_ref, REFS.pulls[0].title, 'master')
  ```
- In the `pod-pull_cdc_kafka_integration_heavy.yaml` Kubernetes template, the Pod specification contains multiple containers each with its own resource requests and limits. This may lead to resource over-commitment if the total resources requested by all containers exceed the resources available on the node. The resources requested by each container should be carefully analyzed and adjusted to avoid potential resource allocation issues.
- In the `pod-pull_cdc_kafka_integration_heavy.yaml` file, there's a TODO comment regarding the resource limit for the `canal-adapter` container. This needs to be addressed as leaving a container without resource limits can lead to resource starvation for other containers or even bring down the node if the container starts consuming excessive resources.

3. **Best Practices:**

- The PR lacks detailed comments explaining the purpose and working of each section of the code, especially in the Jenkins pipeline scripts and the Kubernetes templates. Adding comments would greatly improve the readability and maintainability of the code.
- The PR could benefit from more comprehensive testing. For instance, testing how the system behaves when the third-party projects' branches computed do not exist or are not compatible with the TiCDC version being tested.
- There's a deviation from the style guideline in the Kubernetes templates where multiple containers are defined in the same Pod. These should ideally be split into individual deployment or statefulset objects, especially if they aren't tightly coupled, for better scalability and management.
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T07:01:46Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**
The pull request's aim is to add a Kafka integration test to the `ticdc` project and make it ready for use. This is achieved by introducing new files for Jenkins pipeline jobs, integration tests, and Kubernetes pod specifications. The quality of the code in the PR appears to be high overall, with good use of Jenkins' Declarative Pipeline syntax, Kubernetes YAML, and Groovy scripting. However, there are a few areas where improvements can be made.

1. **Code Improvements:**
   - In `pull_cdc_kafka_integration_heavy.groovy` and `pull_cdc_storage_integration_heavy.groovy` files, there is a code duplication in the `prepare` stage, where the same logic is used to download and validate third-party binaries. This could be refactored into a function to avoid repetition. 
   - In `pod-pull_cdc_kafka_integration_heavy.yaml` and `pod-pull_cdc_storage_integration_heavy.yaml` files, the same resource limits and requests are set for multiple containers. If these values need to be changed, they would need to be updated in multiple places. It would be more efficient to place these common resource configurations in a variable or a template to avoid duplication. 
   - The `Test` stage in `pull_cdc_kafka_integration_heavy.groovy` and `pull_cdc_storage_integration_heavy.groovy` files contains duplicate code. The `script` command could be refactored into a function with parameters for the test group and type of test.

2. **Best Practices:**
   - There is a lack of comments explaining what the code accomplishes in `pull_cdc_kafka_integration_heavy.groovy` and `pull_cdc_storage_integration_heavy.groovy` files. This makes the code harder to understand for people who are not familiar with this codebase. Adding comments that explain the purpose of each stage and step would be beneficial.
   - In the `pod-pull_cdc_kafka_integration_heavy.yaml` and `pod-pull_cdc_storage_integration_heavy.yaml` files, the image versions for the containers are not specified. This could lead to unexpected behavior if the latest image version introduces breaking changes. It would be better to specify a fixed version for each image. 
   - There are hard-coded values in the container environment variables in `pod-pull_cdc_kafka_integration_heavy.yaml` and `pod-pull_cdc_storage_integration_heavy.yaml` files. These should be replaced with Kubernetes ConfigMaps or Secrets for better security and flexibility.

Here are some suggested code changes:

```groovy
// pull_cdc_kafka_integration_heavy.groovy and pull_cdc_storage_integration_heavy.groovy
// Refactor the download and validation of third-party binaries into a function
void prepareBinaries(String baseRef) {
    sh label: "download third_party", script: """
        export TIDB_BRANCH=${component.computeBranchFromPR('tidb', baseRef, REFS.pulls[0].title, 'master')}
        export PD_BRANCH=${component.computeBranchFromPR('pd', baseRef, REFS.pulls[0].title, 'master')}
        export TIKV_BRANCH=${component.computeBranchFromPR('tikv', baseRef, REFS.pulls[0].title, 'master')}
        export TIFLASH_BRANCH=${component.computeBranchFromPR('tiflash', baseRef, REFS.pulls[0].title, 'master')}
        cd ../ticdc && ./tests/scripts/download-integration-test-binaries.sh ${baseRef} && ls -alh ./bin
        make check_third_party_binary
        cd - && mkdir -p bin && mv ../ticdc/bin/* ./bin/
        ls -alh ./bin
        ./bin/tidb-server -V
        ./bin/pd-server -V
        ./bin/tikv-server -V
        ./bin/tiflash --version
    """
}

// call the function
prepareBinaries(REFS.base_ref)

// pull_cdc_kafka_integration_heavy.groovy and pull_cdc_storage_integration_heavy.groovy
// Refactor the test command into a function
void runTest(String testType, String testGroup) {
    sh label: "${testGroup}", script: """
        ./tests/integration_tests/run_heavy_it_in_ci.sh ${testType} ${testGroup}
    """
}

// call the function
runTest("storage", TEST_GROUP)
```

```yaml
# pod-pull_cdc_kafka_integration_heavy.yaml and pod-pull_cdc_storage_integration_heavy.yaml
# Use Kubernetes ConfigMaps or Secrets for environment variables
env:
  - name: KAFKA_SERVER
    valueFrom:
      configMapKeyRef:
        name: kafka-config
        key: server
```

```yaml
# pod-pull_cdc_kafka_integration_heavy.yaml and pod-pull_cdc_storage_integration_heavy.yaml
# Specify a fixed version for the image
image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:1.0.0"
```
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T07:03:24Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**

This PR primarily introduces two new integration tests for the `ticdc` project: `pull_cdc_kafka_integration_heavy` and `pull_cdc_storage_integration_heavy`. These tests are defined in Groovy scripts and YAML files for the Jenkins pipeline and Kubernetes pod specifications, respectively. Overall, the PR appears to be well-structured, adhering to best practices for Jenkins job and Kubernetes pod configurations.

1. **Critical Issues:**

   None found.

2. **Code Improvements:**

   - In `pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy.groovy` and `pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy.groovy`, there are several hard-coded values for test group names. These could be externalized into a configuration or fetched dynamically to reduce maintenance effort or scale test groups.
   
3. **Best Practices:**

   - Documentation: The PR lacks a detailed description, which is crucial for understanding the context and purpose of the changes. Include a brief explanation of the design decisions and trade-offs.
   
   Suggestion:
   ```markdown
   This PR introduces two new heavy integration tests: `pull_cdc_kafka_integration_heavy` and `pull_cdc_storage_integration_heavy`. These tests use Jenkins pipelines and Kubernetes pods to run the `ticdc` project against various test groups. The test group names are currently hard-coded in the Groovy scripts, and future work may involve fetching these dynamically or from a configuration source.
   ```
   
   - Testing: It would be helpful to see if these new tests have been validated in a test environment before integration into the main branch. If not, consider running them in a controlled environment to ensure they behave as expected.
   
   - Naming conventions: The names for the test groups (e.g. 'G00', 'G01', etc.) in the Groovy scripts are not very descriptive. Consider using more descriptive names for these groups or adding comments to describe what each group is for.
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T07:05:17Z
body: > **I have already done a preliminary review for you, and I hope to help you do a better job.**

**Summary:**

This pull request adds Kafka and storage integration tests to the `ticdc` project. It includes the addition of new Groovy, YAML, and configuration files to set up the tests and manage their execution in a Jenkins pipeline. The pull request appears to be well-structured and thorough, but could benefit from some improvements related to best practices and code quality.

**Code Improvements:**

- In `pull_cdc_kafka_integration_heavy.groovy` and `pull_cdc_storage_integration_heavy.groovy`, there are multiple places where strings and commands are hardcoded. These could be extracted into constants or configuration files to make the code more maintainable and easier to update.

**Best Practices:**

- There are no comments in the Groovy and YAML files. Adding comments would help understand the purpose of each section and make the code easier to maintain and debug.
- The Groovy scripts could benefit from some error handling. Currently, if an error occurs during the execution of the scripts, it may not be clear where the error came from or why it occurred.
- In the YAML files, there are multiple places where resource limits are defined for different containers, but these values seem arbitrary. It would be helpful to document why these specific values were chosen, and if there are any considerations for adjusting them.
- The pull request lacks a description, which makes it harder for reviewers and future developers to understand the reasons for these changes and how they should function. It would be beneficial to include a brief summary of the changes, why they were made, and any important information about how they work.
- Naming conventions: The names of the files and parameters are mostly clear, but some might be improved. For example, `pull_cdc_kafka_integration_heavy.groovy` and `pull_cdc_storage_integration_heavy.groovy` might be renamed to something like `cdc_kafka_heavy_integration_test.groovy` and `cdc_storage_heavy_integration_test.groovy` respectively, to make it clear that these are test files.

**Suggested Changes:**

```groovy
// In pull_cdc_kafka_integration_heavy.groovy and pull_cdc_storage_integration_heavy.groovy
// Consider extracting hardcoded strings into constants or configuration
final String KAFKA_CONTAINER = "kafka"
sh label: "Waiting for ${KAFKA_CONTAINER} ready", script: """
    echo "Waiting for zookeeper to be ready..."
    while ! nc -z localhost 2181; do sleep 10; done
    echo "Waiting for ${KAFKA_CONTAINER} to be ready..."
    while ! nc -z localhost 9092; do sleep 10; done
    echo "Waiting for ${KAFKA_CONTAINER}-broker to be ready..."
    while ! echo dump | nc localhost 2181 | grep brokers | awk '{\$1=\$1;print}' | grep -F -w "/brokers/ids/1"; do sleep 10; done
"""
```

```groovy
// In pull_cdc_kafka_integration_heavy.groovy and pull_cdc_storage_integration_heavy.groovy
// Consider adding error handling
try {
    sh label: "prepare", script: """
        ls -alh ./bin
        [ -f ./bin/cdc ] || make cdc
        [ -f ./bin/cdc_kafka_consumer ] || make kafka_consumer
        [ -f ./bin/cdc_storage_consumer ] || make storage_consumer
        [ -f ./bin/cdc.test ] || make integration_test_build
        ls -alh ./bin
        ./bin/cdc version
    """
} catch (Exception e) {
    echo "An error occurred during preparation: ${e.message}"
    throw e
}
```

```yaml
# In pod-pull_cdc_kafka_integration_heavy.yaml and pod-pull_cdc_storage_integration_heavy.yaml
# Document why specific resource limits are chosen
resources:
  limits:
    memory: 16Gi # Chosen because...
    cpu: "12"   # Chosen because...
```

In the PR description, consider providing a brief summary of the changes, why they were made, and any important information about how they work. This could include an overview of the new tests, any dependencies or prerequisites, and how the tests can be run and results interpreted. 

For the naming conventions, consider renaming files and parameters to better reflect their purpose. For example, `pull_cdc_kafka_integration_heavy.groovy` and `pull_cdc_storage_integration_heavy.groovy` might be renamed to something like `cdc_kafka_heavy_integration_test.groovy` and `cdc_storage_heavy_integration_test.groovy` respectively, to make it clear that these are test files.
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T07:11:43Z
body: [LGTM Timeline notifier]
---
Timeline:

- `2025-05-16 07:11:43.617745944 +0000 UTC m=+96669.873643342`: :ballot_box_with_check: agreed by [purelind](https://github.com/purelind).
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-05-16T07:11:44Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/PingCAP-QE/ci/pull/3522#pullrequestreview-2845717935" title="LGTM">purelind</a>*

The full list of commands accepted by this bot can be found [here](https://prow.tidb.net/command-help?repo=PingCAP-QE%2Fci).

The pull request process is described [here](https://book.prow.tidb.net/#/workflows/pr)

<details >
Needs approval from an approver in each of these files:

- ~~[jobs/OWNERS](https://github.com/PingCAP-QE/ci/blob/main/jobs/OWNERS)~~ [purelind]
- ~~[pipelines/pingcap/ticdc/OWNERS](https://github.com/PingCAP-QE/ci/blob/main/pipelines/pingcap/ticdc/OWNERS)~~ [purelind]
- ~~[prow-jobs/OWNERS](https://github.com/PingCAP-QE/ci/blob/main/prow-jobs/OWNERS)~~ [purelind]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE