Repository: pingcap/tidb
Pull request #60543: localbackend: add cloud region job worker | tidb-test=ccf9f5de6817ed2e54c18f2c1757056d2c8b4e4c
Author: login: D3Hunter
Labels: name: size/XXL
description: Denotes a PR that changes 1000+ lines, ignoring generated files.
name: release-note-none
description: Denotes a PR that doesn't merit a release note.
name: approved
name: lgtm
Created: 2025-04-14T10:34:33Z | Merged: 2025-04-17T04:46:02Z

<!--

Thank you for contributing to TiDB!

PR Title Format:
1. pkg [, pkg2, pkg3]: what's changed
2. *: what's changed

-->

### What problem does this PR solve?
<!--

Please create an issue first to describe the problem.

There MUST be one line starting with "Issue Number:  " and
linking the relevant issues via the "close" or "ref".

For more info, check https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/contribute-code.html#referring-to-an-issue.

-->

Issue Number: ref #60418 

Problem Summary:

### What changed and how does it work?
- add cloud region job worker
- move executeJob from `Backend` to baseJobWorker
### Check List

Tests <!-- At least one of them must be included. -->

- [x] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test
  > - [ ] I checked and no code files have been changed.
  > <!-- Or your custom  "No need to test" reasons -->

Side effects

- [ ] Performance regression: Consumes more CPU
- [ ] Performance regression: Consumes more Memory
- [ ] Breaking backward compatibility

Documentation

- [ ] Affects user behaviors
- [ ] Contains syntax changes
- [ ] Contains variable changes
- [ ] Contains experimental features
- [ ] Changes MySQL compatibility

### Release note

<!-- compatibility change, improvement, bugfix, and new feature need a release note -->

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

```release-note
None
```



Diff:
diff --git a/Makefile b/Makefile
index 37dcda1c426a9..c7995b3c58f34 100644
--- a/Makefile
+++ b/Makefile
@@ -515,6 +515,7 @@ gen_mock: mockgen
 	tools/bin/mockgen -package mockstorage github.com/pingcap/tidb/br/pkg/storage ExternalStorage > br/pkg/mock/storage/storage.go
 	tools/bin/mockgen -package mock github.com/pingcap/tidb/pkg/ddl SchemaLoader > pkg/ddl/mock/schema_loader_mock.go
 	tools/bin/mockgen -package mock github.com/pingcap/tidb/pkg/ddl/systable Manager > pkg/ddl/mock/systable_manager_mock.go
+	tools/bin/mockgen -package mock github.com/pingcap/tidb/pkg/ingestor/ingestcli Client,WriteClient > pkg/ingestor/ingestcli/mock/client_mock.go
 
 # There is no FreeBSD environment for GitHub actions. So cross-compile on Linux
 # but that doesn't work with CGO_ENABLED=1, so disable cgo. The reason to have
diff --git a/pkg/ingestor/ingestcli/BUILD.bazel b/pkg/ingestor/ingestcli/BUILD.bazel
index 4dd4d2ed3fa44..44780f33d382c 100644
--- a/pkg/ingestor/ingestcli/BUILD.bazel
+++ b/pkg/ingestor/ingestcli/BUILD.bazel
@@ -5,5 +5,8 @@ go_library(
     srcs = ["interface.go"],
     importpath = "github.com/pingcap/tidb/pkg/ingestor/ingestcli",
     visibility = ["//visibility:public"],
-    deps = ["//dumpling/context"],
+    deps = [
+        "//br/pkg/restore/split",
+        "@com_github_pingcap_kvproto//pkg/import_sstpb",
+    ],
 )
diff --git a/pkg/ingestor/ingestcli/interface.go b/pkg/ingestor/ingestcli/interface.go
index 1de1446a7d6a2..c9e4356fab553 100644
--- a/pkg/ingestor/ingestcli/interface.go
+++ b/pkg/ingestor/ingestcli/interface.go
@@ -14,11 +14,17 @@
 
 package ingestcli
 
-import "github.com/pingcap/tidb/dumpling/context"
+import (
+	"context"
+
+	"github.com/pingcap/kvproto/pkg/import_sstpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+)
 
 // WriteRequest is the request to write KV to storage layer.
 type WriteRequest struct {
 	ClusterID string
+	Pairs     []*import_sstpb.Pair
 }
 
 // WriteResponse is the response of Write.
@@ -28,10 +34,8 @@ type WriteResponse struct {
 
 // IngestRequest is the request to ingest SST to storage layer.
 type IngestRequest struct {
-}
-
-// IngestResponse is the response of Ingest.
-type IngestResponse struct {
+	Region  *split.RegionInfo
+	SSTFile string
 }
 
 // WriteClient is the client for writing KV to storage layer.
@@ -40,9 +44,12 @@ type IngestResponse struct {
 type WriteClient interface {
 	// Write KV to storage layer.
 	Write(ctx context.Context, in *WriteRequest) error
-	// Close the client, get the response and close the client.
-	// the response should contain the SST info
-	Close(ctx context.Context) (*WriteResponse, error)
+	// CloseAndRecv the client, get the response and close the client.
+	// the response should contain the SST info.
+	// we name it this way to mimic GRPC stream, and to avoid confusion with normal
+	// Close method which should always be called after creation, but in here,
+	// it's not a must.
+	CloseAndRecv(ctx context.Context) (*WriteResponse, error)
 }
 
 // Client is the interface to write KV and ingest SST to storage layer.
@@ -60,5 +67,5 @@ type Client interface {
 	// WriteClient returns a WriteClient to write KV to storage layer.
 	WriteClient(ctx context.Context) (WriteClient, error)
 	// Ingest ingests the SST to storage layer.
-	Ingest(ctx context.Context, in *IngestRequest) (*IngestResponse, error)
+	Ingest(ctx context.Context, in *IngestRequest) error
 }
diff --git a/pkg/ingestor/ingestcli/mock/BUILD.bazel b/pkg/ingestor/ingestcli/mock/BUILD.bazel
new file mode 100644
index 0000000000000..34327d7193836
--- /dev/null
+++ b/pkg/ingestor/ingestcli/mock/BUILD.bazel
@@ -0,0 +1,12 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "mock",
+    srcs = ["client_mock.go"],
+    importpath = "github.com/pingcap/tidb/pkg/ingestor/ingestcli/mock",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//pkg/ingestor/ingestcli",
+        "@org_uber_go_mock//gomock",
+    ],
+)
diff --git a/pkg/ingestor/ingestcli/mock/client_mock.go b/pkg/ingestor/ingestcli/mock/client_mock.go
new file mode 100644
index 0000000000000..7ea29946c891c
--- /dev/null
+++ b/pkg/ingestor/ingestcli/mock/client_mock.go
@@ -0,0 +1,132 @@
+// Code generated by MockGen. DO NOT EDIT.
+// Source: github.com/pingcap/tidb/pkg/ingestor/ingestcli (interfaces: Client,WriteClient)
+//
+// Generated by this command:
+//
+//	mockgen -package mock github.com/pingcap/tidb/pkg/ingestor/ingestcli Client,WriteClient
+//
+
+// Package mock is a generated GoMock package.
+package mock
+
+import (
+	context "context"
+	reflect "reflect"
+
+	ingestcli "github.com/pingcap/tidb/pkg/ingestor/ingestcli"
+	gomock "go.uber.org/mock/gomock"
+)
+
+// MockClient is a mock of Client interface.
+type MockClient struct {
+	ctrl     *gomock.Controller
+	recorder *MockClientMockRecorder
+}
+
+// MockClientMockRecorder is the mock recorder for MockClient.
+type MockClientMockRecorder struct {
+	mock *MockClient
+}
+
+// NewMockClient creates a new mock instance.
+func NewMockClient(ctrl *gomock.Controller) *MockClient {
+	mock := &MockClient{ctrl: ctrl}
+	mock.recorder = &MockClientMockRecorder{mock}
+	return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockClient) EXPECT() *MockClientMockRecorder {
+	return m.recorder
+}
+
+// ISGOMOCK indicates that this struct is a gomock mock.
+func (m *MockClient) ISGOMOCK() struct{} {
+	return struct{}{}
+}
+
+// Ingest mocks base method.
+func (m *MockClient) Ingest(arg0 context.Context, arg1 *ingestcli.IngestRequest) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Ingest", arg0, arg1)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// Ingest indicates an expected call of Ingest.
+func (mr *MockClientMockRecorder) Ingest(arg0, arg1 any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Ingest", reflect.TypeOf((*MockClient)(nil).Ingest), arg0, arg1)
+}
+
+// WriteClient mocks base method.
+func (m *MockClient) WriteClient(arg0 context.Context) (ingestcli.WriteClient, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "WriteClient", arg0)
+	ret0, _ := ret[0].(ingestcli.WriteClient)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// WriteClient indicates an expected call of WriteClient.
+func (mr *MockClientMockRecorder) WriteClient(arg0 any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "WriteClient", reflect.TypeOf((*MockClient)(nil).WriteClient), arg0)
+}
+
+// MockWriteClient is a mock of WriteClient interface.
+type MockWriteClient struct {
+	ctrl     *gomock.Controller
+	recorder *MockWriteClientMockRecorder
+}
+
+// MockWriteClientMockRecorder is the mock recorder for MockWriteClient.
+type MockWriteClientMockRecorder struct {
+	mock *MockWriteClient
+}
+
+// NewMockWriteClient creates a new mock instance.
+func NewMockWriteClient(ctrl *gomock.Controller) *MockWriteClient {
+	mock := &MockWriteClient{ctrl: ctrl}
+	mock.recorder = &MockWriteClientMockRecorder{mock}
+	return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockWriteClient) EXPECT() *MockWriteClientMockRecorder {
+	return m.recorder
+}
+
+// ISGOMOCK indicates that this struct is a gomock mock.
+func (m *MockWriteClient) ISGOMOCK() struct{} {
+	return struct{}{}
+}
+
+// CloseAndRecv mocks base method.
+func (m *MockWriteClient) CloseAndRecv(arg0 context.Context) (*ingestcli.WriteResponse, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "CloseAndRecv", arg0)
+	ret0, _ := ret[0].(*ingestcli.WriteResponse)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// CloseAndRecv indicates an expected call of CloseAndRecv.
+func (mr *MockWriteClientMockRecorder) CloseAndRecv(arg0 any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CloseAndRecv", reflect.TypeOf((*MockWriteClient)(nil).CloseAndRecv), arg0)
+}
+
+// Write mocks base method.
+func (m *MockWriteClient) Write(arg0 context.Context, arg1 *ingestcli.WriteRequest) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Write", arg0, arg1)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// Write indicates an expected call of Write.
+func (mr *MockWriteClientMockRecorder) Write(arg0, arg1 any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Write", reflect.TypeOf((*MockWriteClient)(nil).Write), arg0, arg1)
+}
diff --git a/pkg/lightning/backend/local/BUILD.bazel b/pkg/lightning/backend/local/BUILD.bazel
index b1001042e7982..4bf0332cdcc35 100644
--- a/pkg/lightning/backend/local/BUILD.bazel
+++ b/pkg/lightning/backend/local/BUILD.bazel
@@ -30,8 +30,10 @@ go_library(
         "//br/pkg/restore/split",
         "//br/pkg/storage",
         "//br/pkg/version",
+        "//pkg/config",
         "//pkg/distsql",
         "//pkg/infoschema",
+        "//pkg/ingestor/ingestcli",
         "//pkg/kv",
         "//pkg/lightning/backend",
         "//pkg/lightning/backend/encode",
@@ -135,6 +137,8 @@ go_test(
         "//br/pkg/storage",
         "//pkg/ddl",
         "//pkg/errno",
+        "//pkg/ingestor/ingestcli",
+        "//pkg/ingestor/ingestcli/mock",
         "//pkg/keyspace",
         "//pkg/kv",
         "//pkg/lightning/backend",
diff --git a/pkg/lightning/backend/local/job_worker.go b/pkg/lightning/backend/local/job_worker.go
index 1a6a3af830fdc..a2fa8030df423 100644
--- a/pkg/lightning/backend/local/job_worker.go
+++ b/pkg/lightning/backend/local/job_worker.go
@@ -16,20 +16,39 @@ package local
 
 import (
 	"context"
+	"io"
+	"strings"
 	"sync"
+	"time"
 
+	"github.com/pingcap/errors"
 	"github.com/pingcap/failpoint"
+	sst "github.com/pingcap/kvproto/pkg/import_sstpb"
 	"github.com/pingcap/kvproto/pkg/metapb"
+	"github.com/pingcap/tidb/br/pkg/logutil"
+	"github.com/pingcap/tidb/pkg/ingestor/ingestcli"
 	"github.com/pingcap/tidb/pkg/lightning/common"
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pingcap/tidb/pkg/lightning/membuf"
 	"github.com/pingcap/tidb/pkg/metrics"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"github.com/tikv/client-go/v2/oracle"
+	pdhttp "github.com/tikv/pd/client/http"
+	"go.uber.org/zap"
 )
 
+type regionJobWorker interface {
+	run(ctx context.Context) error
+}
+
 type regionJobBaseWorker struct {
 	jobInCh  chan *regionJob
 	jobOutCh chan *regionJob
 	jobWg    *sync.WaitGroup
 
-	doRunJobFn func(ctx context.Context, job *regionJob) error
+	writeFn     func(ctx context.Context, job *regionJob) (*tikvWriteResult, error)
+	ingestFn    func(ctx context.Context, job *regionJob) error
+	preRunJobFn func(ctx context.Context, job *regionJob) error
 	// called after the job is executed, success or not.
 	afterRunJobFn func([]*metapb.Peer)
 	// if the region info is stale, we need to generate new jobs based on the old
@@ -63,7 +82,7 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 			}
 			failpoint.InjectCall("beforeExecuteRegionJob", ctx)
 			metrics.GlobalSortIngestWorkerCnt.WithLabelValues("execute job").Inc()
-			err := w.doRunJobFn(ctx, job)
+			err := w.runJob(ctx, job)
 			metrics.GlobalSortIngestWorkerCnt.WithLabelValues("execute job").Dec()
 
 			if w.afterRunJobFn != nil {
@@ -117,6 +136,240 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 	}
 }
 
+// doRunJob handles a regionJob and tries to convert it to ingested stage.
+// If non-retryable error occurs, it will return the error.
+// If retryable error occurs, it will return nil and caller should check the stage
+// of the regionJob to determine what to do with it.
+func (w *regionJobBaseWorker) runJob(ctx context.Context, job *regionJob) error {
+	if err := w.preRunJobFn(ctx, job); err != nil {
+		return err
+	}
+
+	for {
+		// the job might in wrote stage if it comes from retry
+		if job.stage == regionScanned {
+			// writes the data to TiKV and mark this job as wrote stage.
+			// we don't need to do cleanup for the pairs written to TiKV if encounters
+			// an error, TiKV will take the responsibility to do so.
+			// TODO: let client-go provide a high-level write interface.
+			res, err := w.writeFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				// currently only one case will restart write
+				if strings.Contains(err.Error(), "RequestTooNew") {
+					// TiKV hasn't synced the newest region info with PD, it's ok to
+					// rewrite without rescan.
+					job.convertStageTo(regionScanned)
+				} else {
+					job.convertStageTo(needRescan)
+				}
+				job.lastRetryableErr = err
+				log.FromContext(ctx).Warn("meet retryable error when writing to TiKV",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				return nil
+			}
+			if res.emptyJob {
+				job.convertStageTo(ingested)
+			} else {
+				job.writeResult = res
+				job.convertStageTo(wrote)
+			}
+		}
+
+		// if the job is empty, it might go to ingested stage directly.
+		if job.stage == wrote {
+			err := w.ingestFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				log.FromContext(ctx).Warn("meet retryable error when ingesting",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				job.lastRetryableErr = err
+				return nil
+			}
+		}
+		// if the stage is not ingested, it means some error happened, the job should
+		// be sent back to caller to retry later, else we handle remaining data.
+		if job.stage != ingested {
+			return nil
+		}
+
+		if job.writeResult == nil || job.writeResult.remainingStartKey == nil {
+			return nil
+		}
+		// partially write and ingest, update the job key range and continue
+		job.keyRange.Start = job.writeResult.remainingStartKey
+		job.convertStageTo(regionScanned)
+	}
+}
+
+func (*regionJobBaseWorker) isRetryableImportTiKVError(err error) bool {
+	err = errors.Cause(err)
+	// io.EOF is not retryable in normal case
+	// but on TiKV restart, if we're writing to TiKV(through GRPC)
+	// it might return io.EOF(it's GRPC Unavailable in most case),
+	// we need to retry on this error.
+	// see SendMsg in https://pkg.go.dev/google.golang.org/grpc#ClientStream
+	if err == io.EOF {
+		return true
+	}
+	return common.IsRetryableError(err)
+}
+
 type opRegionJobWorker struct {
 	*regionJobBaseWorker
+	checkTiKVSpace bool
+	pdHTTPCli      pdhttp.Client
+}
+
+func (w *opRegionJobWorker) preRunJob(ctx context.Context, job *regionJob) error {
+	failpoint.Inject("WriteToTiKVNotEnoughDiskSpace", func(_ failpoint.Value) {
+		failpoint.Return(
+			errors.New("the remaining storage capacity of TiKV is less than 10%%; please increase the storage capacity of TiKV and try again"))
+	})
+	if w.checkTiKVSpace {
+		for _, peer := range job.region.Region.GetPeers() {
+			store, err := w.pdHTTPCli.GetStore(ctx, peer.StoreId)
+			if err != nil {
+				log.FromContext(ctx).Warn("failed to get StoreInfo from pd http api", zap.Error(err))
+				continue
+			}
+			err = checkDiskAvail(ctx, store)
+			if err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+type cloudRegionJobWorker struct {
+	*regionJobBaseWorker
+	ingestCli      ingestcli.Client
+	writeBatchSize int64
+	bufPool        *membuf.Pool
+}
+
+func (*cloudRegionJobWorker) preRunJob(_ context.Context, _ *regionJob) error {
+	// cloud engine use cloud storage, such as S3, to hold data, so no need to check
+	// disk fullness.
+	return nil
+}
+
+// we don't need to limit write speed as we write to tikv-worker.
+func (w *cloudRegionJobWorker) write(ctx context.Context, job *regionJob) (*tikvWriteResult, error) {
+	firstKey, _, err := job.ingestData.GetFirstAndLastKey(job.keyRange.Start, job.keyRange.End)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	if firstKey == nil {
+		return &tikvWriteResult{emptyJob: true}, nil
+	}
+
+	writeCli, err := w.ingestCli.WriteClient(ctx)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	dataCommitTS := job.ingestData.GetTS()
+	intest.AssertFunc(func() bool {
+		timeOfTS := oracle.GetTimeFromTS(dataCommitTS)
+		now := time.Now()
+		if timeOfTS.Sub(now) > time.Hour {
+			return false
+		}
+		if now.Sub(timeOfTS) > 24*time.Hour {
+			return false
+		}
+		return true
+	}, "TS used in import should in [now-1d, now+1h], but got %d", dataCommitTS)
+	if dataCommitTS == 0 {
+		return nil, errors.New("data commitTS is 0")
+	}
+
+	pairs := make([]*sst.Pair, 0, defaultKVBatchCount)
+	size := int64(0)
+	totalCount := int64(0)
+	totalSize := int64(0)
+
+	iter := job.ingestData.NewIter(ctx, job.keyRange.Start, job.keyRange.End, w.bufPool)
+	//nolint: errcheck
+	defer iter.Close()
+
+	for iter.First(); iter.Valid(); iter.Next() {
+		k, v := iter.Key(), iter.Value()
+		pairs = append(pairs, &sst.Pair{
+			Key:   k,
+			Value: v,
+		})
+		size += int64(len(k) + len(v))
+
+		if size >= w.writeBatchSize {
+			in := &ingestcli.WriteRequest{
+				Pairs: pairs,
+			}
+			if err := writeCli.Write(ctx, in); err != nil {
+				return nil, errors.Trace(err)
+			}
+			totalCount += int64(len(pairs))
+			totalSize += size
+			size = 0
+			pairs = pairs[:0]
+			iter.ReleaseBuf()
+		}
+	}
+
+	if iter.Error() != nil {
+		return nil, errors.Trace(iter.Error())
+	}
+
+	if len(pairs) > 0 {
+		in := &ingestcli.WriteRequest{
+			Pairs: pairs,
+		}
+		if err := writeCli.Write(ctx, in); err != nil {
+			return nil, errors.Trace(err)
+		}
+		totalCount += int64(len(pairs))
+		totalSize += size
+		pairs = pairs[:0]
+		iter.ReleaseBuf()
+	}
+
+	resp, err := writeCli.CloseAndRecv(ctx)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+
+	return &tikvWriteResult{
+		count:      totalCount,
+		totalBytes: totalSize,
+		sstFile:    resp.SSTFile,
+	}, nil
+}
+
+func (w *cloudRegionJobWorker) ingest(ctx context.Context, j *regionJob) error {
+	in := &ingestcli.IngestRequest{
+		Region:  j.region,
+		SSTFile: j.writeResult.sstFile,
+	}
+	err := w.ingestCli.Ingest(ctx, in)
+	if err != nil {
+		// TODO, we should let outer logic handle stage transition, currently, OP
+		//  worker need a lot of change before we can do it, will refactor it later.
+
+		// TODO: choose target stage based on error.
+		j.convertStageTo(needRescan)
+		log.FromContext(ctx).Warn("meet error and handle the job later",
+			zap.Stringer("job stage", j.stage),
+			logutil.ShortError(j.lastRetryableErr),
+			j.region.ToZapFields(),
+			logutil.Key("start", j.keyRange.Start),
+			logutil.Key("end", j.keyRange.End))
+		return err
+	}
+	j.convertStageTo(ingested)
+	return nil
 }
diff --git a/pkg/lightning/backend/local/job_worker_test.go b/pkg/lightning/backend/local/job_worker_test.go
index 62888a4325e0b..241c50a04dc29 100644
--- a/pkg/lightning/backend/local/job_worker_test.go
+++ b/pkg/lightning/backend/local/job_worker_test.go
@@ -16,11 +16,16 @@ package local
 
 import (
 	"context"
+	"io"
 	"sync"
 	"testing"
 
+	"github.com/pingcap/errors"
+	"github.com/pingcap/tidb/pkg/ingestor/ingestcli"
+	ingestclimock "github.com/pingcap/tidb/pkg/ingestor/ingestcli/mock"
 	"github.com/pingcap/tidb/pkg/lightning/common"
 	"github.com/stretchr/testify/require"
+	"go.uber.org/mock/gomock"
 )
 
 func TestRegionJobBaseWorker(t *testing.T) {
@@ -29,7 +34,13 @@ func TestRegionJobBaseWorker(t *testing.T) {
 			jobInCh:  make(chan *regionJob, 10),
 			jobOutCh: make(chan *regionJob, 10),
 			jobWg:    &sync.WaitGroup{},
-			doRunJobFn: func(ctx context.Context, job *regionJob) error {
+			preRunJobFn: func(ctx context.Context, job *regionJob) error {
+				return nil
+			},
+			writeFn: func(ctx context.Context, job *regionJob) (*tikvWriteResult, error) {
+				return &tikvWriteResult{}, nil
+			},
+			ingestFn: func(ctx context.Context, job *regionJob) error {
 				job.convertStageTo(ingested)
 				return nil
 			},
@@ -58,7 +69,7 @@ func TestRegionJobBaseWorker(t *testing.T) {
 
 	t.Run("send job to out channel after run job", func(t *testing.T) {
 		w := newWorker()
-		job := &regionJob{}
+		job := &regionJob{stage: regionScanned, ingestData: mockIngestData{}}
 		w.jobInCh <- job
 		close(w.jobInCh)
 		require.NoError(t, w.run(context.Background()))
@@ -67,14 +78,143 @@ func TestRegionJobBaseWorker(t *testing.T) {
 
 	t.Run("regenerate jobs", func(t *testing.T) {
 		w := newWorker()
-		w.doRunJobFn = func(ctx context.Context, job *regionJob) error {
+		w.ingestFn = func(ctx context.Context, job *regionJob) error {
 			job.convertStageTo(needRescan)
 			return nil
 		}
-		job := &regionJob{}
+		job := &regionJob{stage: regionScanned, ingestData: mockIngestData{}}
 		w.jobInCh <- job
 		close(w.jobInCh)
 		require.NoError(t, w.run(context.Background()))
 		require.Equal(t, 3, len(w.jobOutCh))
 	})
 }
+
+func TestIsRetryableTiKVWriteError(t *testing.T) {
+	w := &regionJobBaseWorker{}
+	require.True(t, w.isRetryableImportTiKVError(io.EOF))
+	require.True(t, w.isRetryableImportTiKVError(errors.Trace(io.EOF)))
+}
+
+func TestCloudRegionJobWorker(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+	mockIngestCli := ingestclimock.NewMockClient(ctrl)
+
+	cloudW := &cloudRegionJobWorker{
+		regionJobBaseWorker: &regionJobBaseWorker{},
+		ingestCli:           mockIngestCli,
+		writeBatchSize:      8,
+		bufPool:             nil,
+	}
+	cloudW.regionJobBaseWorker.writeFn = cloudW.write
+	cloudW.regionJobBaseWorker.ingestFn = cloudW.ingest
+	cloudW.regionJobBaseWorker.preRunJobFn = cloudW.preRunJob
+
+	t.Run("empty job", func(t *testing.T) {
+		job := &regionJob{
+			keyRange:   common.Range{Start: []byte("a"), End: []byte("z")},
+			stage:      regionScanned,
+			ingestData: mockIngestData{},
+		}
+		writeRes, err := cloudW.write(context.Background(), job)
+		require.NoError(t, err)
+		require.True(t, writeRes.emptyJob)
+		require.True(t, ctrl.Satisfied())
+	})
+
+	t.Run("failed to create ingest client", func(t *testing.T) {
+		job := &regionJob{
+			keyRange:   common.Range{Start: []byte("a"), End: []byte("z")},
+			stage:      regionScanned,
+			ingestData: mockIngestData{{[]byte("a"), []byte("a")}},
+		}
+		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(nil, errors.New("mock error"))
+		writeRes, err := cloudW.write(context.Background(), job)
+		require.ErrorContains(t, err, "mock error")
+		require.Nil(t, writeRes)
+		require.True(t, ctrl.Satisfied())
+	})
+
+	t.Run("failed to write data", func(t *testing.T) {
+		job := &regionJob{
+			keyRange:   common.Range{Start: []byte("a"), End: []byte("z")},
+			stage:      regionScanned,
+			ingestData: mockIngestData{{[]byte("a"), []byte("a")}},
+		}
+		writeCli := ingestclimock.NewMockWriteClient(ctrl)
+		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(writeCli, nil)
+		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(errors.New("mock error"))
+		writeRes, err := cloudW.write(context.Background(), job)
+		require.ErrorContains(t, err, "mock error")
+		require.Nil(t, writeRes)
+		require.True(t, ctrl.Satisfied())
+	})
+
+	t.Run("failed to closeAndRecv", func(t *testing.T) {
+		job := &regionJob{
+			keyRange:   common.Range{Start: []byte("a"), End: []byte("z")},
+			stage:      regionScanned,
+			ingestData: mockIngestData{{[]byte("a"), []byte("a")}},
+		}
+		writeCli := ingestclimock.NewMockWriteClient(ctrl)
+		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(writeCli, nil)
+		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(nil)
+		writeCli.EXPECT().CloseAndRecv(gomock.Any()).Return(nil, errors.New("mock error"))
+		resp, err := cloudW.write(context.Background(), job)
+		require.ErrorContains(t, err, "mock error")
+		require.Nil(t, resp)
+		require.True(t, ctrl.Satisfied())
+	})
+
+	t.Run("write data success, and we have trailing pairs after iteration loop", func(t *testing.T) {
+		job := &regionJob{
+			keyRange: common.Range{Start: []byte("a"), End: []byte("z")},
+			stage:    regionScanned,
+			ingestData: mockIngestData{
+				{[]byte("aa"), []byte("aaaa")},
+				{[]byte("ab"), []byte("abab")},
+				{[]byte("ac"), []byte("acac")},
+			},
+		}
+		writeCli := ingestclimock.NewMockWriteClient(ctrl)
+		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(writeCli, nil)
+		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(nil)
+		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(nil)
+		writeCli.EXPECT().CloseAndRecv(gomock.Any()).Return(&ingestcli.WriteResponse{SSTFile: "the-file"}, nil)
+		res, err := cloudW.write(context.Background(), job)
+		require.NoError(t, err)
+		require.EqualValues(t, 3, res.count)
+		require.EqualValues(t, 18, res.totalBytes)
+		require.Equal(t, "the-file", res.sstFile)
+		require.True(t, ctrl.Satisfied())
+	})
+
+	t.Run("ingest failed", func(t *testing.T) {
+		job := &regionJob{
+			keyRange:    common.Range{Start: []byte("a"), End: []byte("z")},
+			stage:       wrote,
+			ingestData:  mockIngestData{},
+			writeResult: &tikvWriteResult{sstFile: "the-file"},
+		}
+		mockIngestCli.EXPECT().Ingest(gomock.Any(), gomock.Any()).Return(errors.New("mock error"))
+		err := cloudW.ingest(context.Background(), job)
+		require.ErrorContains(t, err, "mock error")
+		require.Equal(t, needRescan, job.stage)
+		require.True(t, ctrl.Satisfied())
+	})
+
+	t.Run("ingest success", func(t *testing.T) {
+		job := &regionJob{
+			keyRange:    common.Range{Start: []byte("a"), End: []byte("z")},
+			stage:       wrote,
+			ingestData:  mockIngestData{},
+			writeResult: &tikvWriteResult{sstFile: "the-file"},
+		}
+		mockIngestCli.EXPECT().Ingest(gomock.Any(), gomock.Any()).Return(nil)
+		err := cloudW.ingest(context.Background(), job)
+		require.NoError(t, err)
+		require.Equal(t, ingested, job.stage)
+		require.True(t, ctrl.Satisfied())
+	})
+}
diff --git a/pkg/lightning/backend/local/local.go b/pkg/lightning/backend/local/local.go
index 6e4bcf3ee36cd..7ed5454fede99 100644
--- a/pkg/lightning/backend/local/local.go
+++ b/pkg/lightning/backend/local/local.go
@@ -19,7 +19,6 @@ import (
 	"context"
 	"database/sql"
 	"encoding/hex"
-	"io"
 	"math"
 	"net"
 	"path/filepath"
@@ -38,6 +37,7 @@ import (
 	"github.com/pingcap/tidb/br/pkg/pdutil"
 	"github.com/pingcap/tidb/br/pkg/restore/split"
 	"github.com/pingcap/tidb/br/pkg/version"
+	tidbconfig "github.com/pingcap/tidb/pkg/config"
 	"github.com/pingcap/tidb/pkg/infoschema"
 	"github.com/pingcap/tidb/pkg/lightning/backend"
 	"github.com/pingcap/tidb/pkg/lightning/backend/encode"
@@ -1093,19 +1093,6 @@ func (local *Backend) generateJobForRange(
 	return jobs, nil
 }
 
-func (*Backend) isRetryableImportTiKVError(err error) bool {
-	err = errors.Cause(err)
-	// io.EOF is not retryable in normal case
-	// but on TiKV restart, if we're writing to TiKV(through GRPC)
-	// it might return io.EOF(it's GRPC Unavailable in most case),
-	// we need to retry on this error.
-	// see SendMsg in https://pkg.go.dev/google.golang.org/grpc#ClientStream
-	if err == io.EOF {
-		return true
-	}
-	return common.IsRetryableError(err)
-}
-
 func checkDiskAvail(ctx context.Context, store *pdhttp.StoreInfo) error {
 	logger := log.FromContext(ctx)
 	capacity, err := units.RAMInBytes(store.Status.Capacity)
@@ -1137,92 +1124,6 @@ func checkDiskAvail(ctx context.Context, store *pdhttp.StoreInfo) error {
 	return nil
 }
 
-// executeJob handles a regionJob and tries to convert it to ingested stage.
-// If non-retryable error occurs, it will return the error.
-// If retryable error occurs, it will return nil and caller should check the stage
-// of the regionJob to determine what to do with it.
-func (local *Backend) executeJob(
-	ctx context.Context,
-	job *regionJob,
-) error {
-	failpoint.Inject("WriteToTiKVNotEnoughDiskSpace", func(_ failpoint.Value) {
-		failpoint.Return(
-			errors.New("the remaining storage capacity of TiKV is less than 10%%; please increase the storage capacity of TiKV and try again"))
-	})
-	if local.ShouldCheckTiKV {
-		for _, peer := range job.region.Region.GetPeers() {
-			store, err := local.pdHTTPCli.GetStore(ctx, peer.StoreId)
-			if err != nil {
-				log.FromContext(ctx).Warn("failed to get StoreInfo from pd http api", zap.Error(err))
-				continue
-			}
-			err = checkDiskAvail(ctx, store)
-			if err != nil {
-				return err
-			}
-		}
-	}
-
-	for {
-		// the job might in wrote stage if it comes from retry
-		if job.stage == regionScanned {
-			// writes the data to TiKV and mark this job as wrote stage.
-			// we don't need to do cleanup for the pairs written to TiKV if encounters
-			// an error, TiKV will take the responsibility to do so.
-			// TODO: let client-go provide a high-level write interface.
-			res, err := local.doWrite(ctx, job)
-			if err != nil {
-				if !local.isRetryableImportTiKVError(err) {
-					return err
-				}
-				// currently only one case will restart write
-				if strings.Contains(err.Error(), "RequestTooNew") {
-					// TiKV hasn't synced the newest region info with PD, it's ok to
-					// rewrite without rescan.
-					job.convertStageTo(regionScanned)
-				} else {
-					job.convertStageTo(needRescan)
-				}
-				job.lastRetryableErr = err
-				log.FromContext(ctx).Warn("meet retryable error when writing to TiKV",
-					log.ShortError(err), zap.Stringer("job stage", job.stage))
-				return nil
-			}
-			if res.emptyJob {
-				job.convertStageTo(ingested)
-			} else {
-				job.writeResult = res
-				job.convertStageTo(wrote)
-			}
-		}
-
-		// if the job is empty, it might go to ingested stage directly.
-		if job.stage == wrote {
-			err := local.ingest(ctx, job)
-			if err != nil {
-				if !local.isRetryableImportTiKVError(err) {
-					return err
-				}
-				log.FromContext(ctx).Warn("meet retryable error when ingesting",
-					log.ShortError(err), zap.Stringer("job stage", job.stage))
-				job.lastRetryableErr = err
-				return nil
-			}
-		}
-		// if the stage is not ingested, it means some error happened, the job should
-		// be sent back to caller to retry later, else we handle remaining data.
-		if job.stage != ingested {
-			return nil
-		}
-
-		if job.writeResult == nil || job.writeResult.remainingStartKey == nil {
-			return nil
-		}
-		job.keyRange.Start = job.writeResult.remainingStartKey
-		job.convertStageTo(regionScanned)
-	}
-}
-
 // ImportEngine imports an engine to TiKV.
 func (local *Backend) ImportEngine(
 	ctx context.Context,
@@ -1484,16 +1385,7 @@ func (local *Backend) doImport(
 		afterExecuteJob = balancer.releaseStoreLoad
 	}
 	for i := 0; i < local.WorkerConcurrency; i++ {
-		worker := &opRegionJobWorker{
-			regionJobBaseWorker: &regionJobBaseWorker{
-				jobInCh:          toCh,
-				jobOutCh:         jobFromWorkerCh,
-				jobWg:            &jobWg,
-				doRunJobFn:       local.executeJob,
-				afterRunJobFn:    afterExecuteJob,
-				regenerateJobsFn: local.generateJobForRange,
-			},
-		}
+		worker := local.newRegionJobWorker(toCh, jobFromWorkerCh, &jobWg, afterExecuteJob)
 		workGroup.Go(func() error {
 			return worker.run(workerCtx)
 		})
@@ -1540,6 +1432,43 @@ func (local *Backend) doImport(
 	return err
 }
 
+func (local *Backend) newRegionJobWorker(
+	toCh, jobFromWorkerCh chan *regionJob,
+	jobWg *sync.WaitGroup,
+	afterExecuteJob func([]*metapb.Peer),
+) regionJobWorker {
+	base := &regionJobBaseWorker{
+		jobInCh:          toCh,
+		jobOutCh:         jobFromWorkerCh,
+		jobWg:            jobWg,
+		afterRunJobFn:    afterExecuteJob,
+		regenerateJobsFn: local.generateJobForRange,
+	}
+	if tidbconfig.IsCloudStore() {
+		cloudW := &cloudRegionJobWorker{
+			// TODO fill the cli
+			ingestCli:      nil,
+			writeBatchSize: local.KVWriteBatchSize,
+			bufPool:        local.engineMgr.getBufferPool(),
+		}
+		base.writeFn = cloudW.write
+		base.ingestFn = cloudW.ingest
+		base.preRunJobFn = cloudW.preRunJob
+		cloudW.regionJobBaseWorker = base
+		return cloudW
+	}
+
+	opWorker := &opRegionJobWorker{
+		checkTiKVSpace: local.ShouldCheckTiKV,
+		pdHTTPCli:      local.pdHTTPCli,
+	}
+	base.writeFn = local.doWrite
+	base.ingestFn = local.ingest
+	base.preRunJobFn = opWorker.preRunJob
+	opWorker.regionJobBaseWorker = base
+	return opWorker
+}
+
 // GetImportedKVCount returns the number of imported KV pairs of some engine.
 func (local *Backend) GetImportedKVCount(engineUUID uuid.UUID) int64 {
 	return local.engineMgr.getImportedKVCount(engineUUID)
diff --git a/pkg/lightning/backend/local/local_test.go b/pkg/lightning/backend/local/local_test.go
index a6cad110bc270..037046d29143f 100644
--- a/pkg/lightning/backend/local/local_test.go
+++ b/pkg/lightning/backend/local/local_test.go
@@ -19,7 +19,6 @@ import (
 	"context"
 	"encoding/binary"
 	"fmt"
-	"io"
 	"math"
 	"math/rand"
 	"path"
@@ -1094,28 +1093,18 @@ func TestLocalWriteAndIngestPairsFailFast(t *testing.T) {
 	defer func() {
 		require.NoError(t, failpoint.Disable("github.com/pingcap/tidb/pkg/lightning/backend/local/WriteToTiKVNotEnoughDiskSpace"))
 	}()
-	worker := &opRegionJobWorker{
-		regionJobBaseWorker: &regionJobBaseWorker{
-			jobInCh:          make(chan *regionJob, 1),
-			jobOutCh:         make(chan *regionJob, 1),
-			doRunJobFn:       bak.executeJob,
-			regenerateJobsFn: bak.generateJobForRange,
-		},
-	}
-	worker.jobInCh <- &regionJob{}
+	toCh := make(chan *regionJob, 1)
+	worker := bak.newRegionJobWorker(toCh, make(chan *regionJob, 1), nil, nil)
+
+	toCh <- &regionJob{}
 	err := worker.run(context.Background())
 	require.Error(t, err)
 	require.Regexp(t, "the remaining storage capacity of TiKV.*", err.Error())
-	require.Len(t, worker.jobInCh, 0)
-}
-
-func TestLocalIsRetryableTiKVWriteError(t *testing.T) {
-	l := Backend{}
-	require.True(t, l.isRetryableImportTiKVError(io.EOF))
-	require.True(t, l.isRetryableImportTiKVError(errors.Trace(io.EOF)))
+	require.Len(t, toCh, 0)
 }
 
 // mockIngestData must be ordered on the first element of each [2][]byte.
+// there cannot be duplicated items.
 type mockIngestData [][2][]byte
 
 func (m mockIngestData) GetFirstAndLastKey(lowerBound, upperBound []byte) ([]byte, []byte, error) {
@@ -1150,13 +1139,18 @@ func (m mockIngestData) getFirstAndLastKeyIdx(lowerBound, upperBound []byte) (in
 		if i == 0 {
 			return -1, -1
 		}
-		last = i - 1
+		if i == len(m) || !bytes.Equal(upperBound, m[i][1]) {
+			last = i - 1
+		} else {
+			last = i
+		}
 	}
 	return first, last
 }
 
 type mockIngestIter struct {
-	data                     mockIngestData
+	data mockIngestData
+	// [startIdx, endIdx)
 	startIdx, endIdx, curIdx int
 }
 
@@ -1184,7 +1178,7 @@ func (m *mockIngestIter) ReleaseBuf() {}
 
 func (m mockIngestData) NewIter(_ context.Context, lowerBound, upperBound []byte, _ *membuf.Pool) common.ForwardIter {
 	i, j := m.getFirstAndLastKeyIdx(lowerBound, upperBound)
-	return &mockIngestIter{data: m, startIdx: i, endIdx: j, curIdx: i}
+	return &mockIngestIter{data: m, startIdx: i, endIdx: j + 1, curIdx: i}
 }
 
 func (m mockIngestData) GetTS() uint64 { return oracle.GoTimeToTS(time.Now()) }
@@ -1304,14 +1298,7 @@ func TestCheckPeersBusy(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := &opRegionJobWorker{
-			regionJobBaseWorker: &regionJobBaseWorker{
-				jobInCh:          jobCh,
-				jobOutCh:         jobOutCh,
-				doRunJobFn:       local.executeJob,
-				regenerateJobsFn: local.generateJobForRange,
-			},
-		}
+		worker := local.newRegionJobWorker(jobCh, jobOutCh, nil, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
@@ -1419,15 +1406,7 @@ func TestNotLeaderErrorNeedUpdatePeers(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := &opRegionJobWorker{
-			regionJobBaseWorker: &regionJobBaseWorker{
-				jobInCh:          jobCh,
-				jobOutCh:         jobOutCh,
-				jobWg:            &jobWg,
-				doRunJobFn:       local.executeJob,
-				regenerateJobsFn: local.generateJobForRange,
-			},
-		}
+		worker := local.newRegionJobWorker(jobCh, jobOutCh, &jobWg, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
@@ -1528,15 +1507,7 @@ func TestPartialWriteIngestErrorWontPanic(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := &opRegionJobWorker{
-			regionJobBaseWorker: &regionJobBaseWorker{
-				jobInCh:          jobCh,
-				jobOutCh:         jobOutCh,
-				jobWg:            &jobWg,
-				doRunJobFn:       local.executeJob,
-				regenerateJobsFn: local.generateJobForRange,
-			},
-		}
+		worker := local.newRegionJobWorker(jobCh, jobOutCh, &jobWg, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
@@ -1658,15 +1629,7 @@ func TestPartialWriteIngestBusy(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := &opRegionJobWorker{
-			regionJobBaseWorker: &regionJobBaseWorker{
-				jobInCh:          jobCh,
-				jobOutCh:         jobOutCh,
-				jobWg:            &jobWg,
-				doRunJobFn:       local.executeJob,
-				regenerateJobsFn: local.generateJobForRange,
-			},
-		}
+		worker := local.newRegionJobWorker(jobCh, jobOutCh, &jobWg, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
diff --git a/pkg/lightning/backend/local/region_job.go b/pkg/lightning/backend/local/region_job.go
index 1525dbb43bcd2..2a86784aadcc8 100644
--- a/pkg/lightning/backend/local/region_job.go
+++ b/pkg/lightning/backend/local/region_job.go
@@ -123,13 +123,18 @@ type regionJob struct {
 
 type tikvWriteResult struct {
 	// means there is no data inside this job
-	emptyJob bool
-
-	// this field might be modified in-place to remove SSTs that are ingested successfully.
-	sstMeta           []*sst.SSTMeta
+	emptyJob          bool
 	count             int64
 	totalBytes        int64
 	remainingStartKey []byte
+
+	// below fields are for OP generation store engine.
+	// this field might be modified in-place to remove SSTs that are ingested successfully.
+	sstMeta []*sst.SSTMeta
+
+	// below fields are for cloud generation store engine.
+	// the filename of the written sst file by tikv-worker.
+	sstFile string
 }
 
 type injectedBehaviour struct {


Reviews:
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T10:38:48Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T10:48:11Z
user: login: tangenta
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T06:49:58Z
user: login: lance6716
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T12:41:04Z
user: login: lance6716
body: 6 / 11 files viewed

state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T12:48:34Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:06:14Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:30:31Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:31:57Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:34:59Z
user: login: lance6716
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T03:57:39Z

Review comments:
diff_hunk: @@ -117,6 +136,237 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 	}
 }
 
+// doRunJob handles a regionJob and tries to convert it to ingested stage.
+// If non-retryable error occurs, it will return the error.
+// If retryable error occurs, it will return nil and caller should check the stage
+// of the regionJob to determine what to do with it.
+func (w *regionJobBaseWorker) doRunJob(ctx context.Context, job *regionJob) error {
path: pkg/lightning/backend/local/job_worker.go
user: login: D3Hunter
body: this is the previous `local.executeJob`
created_at: 2025-04-14T10:38:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,6 +136,237 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 	}
 }
 
+// doRunJob handles a regionJob and tries to convert it to ingested stage.
+// If non-retryable error occurs, it will return the error.
+// If retryable error occurs, it will return nil and caller should check the stage
+// of the regionJob to determine what to do with it.
+func (w *regionJobBaseWorker) doRunJob(ctx context.Context, job *regionJob) error {
+	for {
+		// the job might in wrote stage if it comes from retry
+		if job.stage == regionScanned {
+			// writes the data to TiKV and mark this job as wrote stage.
+			// we don't need to do cleanup for the pairs written to TiKV if encounters
+			// an error, TiKV will take the responsibility to do so.
+			// TODO: let client-go provide a high-level write interface.
+			res, err := w.writeFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				// currently only one case will restart write
+				if strings.Contains(err.Error(), "RequestTooNew") {
+					// TiKV hasn't synced the newest region info with PD, it's ok to
+					// rewrite without rescan.
+					job.convertStageTo(regionScanned)
+				} else {
+					job.convertStageTo(needRescan)
+				}
+				job.lastRetryableErr = err
+				log.FromContext(ctx).Warn("meet retryable error when writing to TiKV",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				return nil
+			}
+			if res.emptyJob {
+				job.convertStageTo(ingested)
+			} else {
+				job.writeResult = res
+				job.convertStageTo(wrote)
+			}
+		}
+
+		// if the job is empty, it might go to ingested stage directly.
+		if job.stage == wrote {
+			err := w.ingestFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				log.FromContext(ctx).Warn("meet retryable error when ingesting",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				job.lastRetryableErr = err
+				return nil
+			}
+		}
+		// if the stage is not ingested, it means some error happened, the job should
+		// be sent back to caller to retry later, else we handle remaining data.
+		if job.stage != ingested {
+			return nil
+		}
+
+		if job.writeResult == nil || job.writeResult.remainingStartKey == nil {
+			return nil
+		}
+		// partially write and ingest, update the job key range and continue
+		job.keyRange.Start = job.writeResult.remainingStartKey
+		job.convertStageTo(regionScanned)
+	}
+}
+
+func (*regionJobBaseWorker) isRetryableImportTiKVError(err error) bool {
+	err = errors.Cause(err)
+	// io.EOF is not retryable in normal case
+	// but on TiKV restart, if we're writing to TiKV(through GRPC)
+	// it might return io.EOF(it's GRPC Unavailable in most case),
+	// we need to retry on this error.
+	// see SendMsg in https://pkg.go.dev/google.golang.org/grpc#ClientStream
+	if err == io.EOF {
+		return true
+	}
+	return common.IsRetryableError(err)
+}
+
 type opRegionJobWorker struct {
 	*regionJobBaseWorker
+	checkTiKVSpace bool
+	pdHTTPCli      pdhttp.Client
+}
+
+func (w *opRegionJobWorker) runJob(ctx context.Context, job *regionJob) error {
+	failpoint.Inject("WriteToTiKVNotEnoughDiskSpace", func(_ failpoint.Value) {
+		failpoint.Return(
+			errors.New("the remaining storage capacity of TiKV is less than 10%%; please increase the storage capacity of TiKV and try again"))
+	})
+	if w.checkTiKVSpace {
+		for _, peer := range job.region.Region.GetPeers() {
+			store, err := w.pdHTTPCli.GetStore(ctx, peer.StoreId)
+			if err != nil {
+				log.FromContext(ctx).Warn("failed to get StoreInfo from pd http api", zap.Error(err))
+				continue
+			}
+			err = checkDiskAvail(ctx, store)
+			if err != nil {
+				return err
+			}
+		}
+	}
+	return w.regionJobBaseWorker.doRunJob(ctx, job)
+}
+
+type cloudRegionJobWorker struct {
+	*regionJobBaseWorker
+	ingestCli      ingestcli.Client
+	writeBatchSize int64
+	bufPool        *membuf.Pool
+}
+
+func (w *cloudRegionJobWorker) runJob(ctx context.Context, job *regionJob) error {
+	// cloud engine use cloud storage, such as S3, to hold data, so no need to check
+	// disk fullness.
+	return w.regionJobBaseWorker.doRunJob(ctx, job)
+}
+
+// we don't need to limit write speed as we write to tikv-worker.
+func (w *cloudRegionJobWorker) write(ctx context.Context, job *regionJob) (*tikvWriteResult, error) {
+	firstKey, _, err := job.ingestData.GetFirstAndLastKey(job.keyRange.Start, job.keyRange.End)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	if firstKey == nil {
+		return &tikvWriteResult{emptyJob: true}, nil
+	}
+
+	writeCli, err := w.ingestCli.WriteClient(ctx)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	dataCommitTS := job.ingestData.GetTS()
+	intest.AssertFunc(func() bool {
+		timeOfTS := oracle.GetTimeFromTS(dataCommitTS)
+		now := time.Now()
+		if timeOfTS.Sub(now) > time.Hour {
+			return false
+		}
+		if now.Sub(timeOfTS) > 24*time.Hour {
+			return false
+		}
+		return true
+	}, "TS used in import should in [now-1d, now+1h], but got %d", dataCommitTS)
+	if dataCommitTS == 0 {
+		return nil, errors.New("data commitTS is 0")
+	}
+
+	pairs := make([]*sst.Pair, 0, defaultKVBatchCount)
+	size := int64(0)
+	totalCount := int64(0)
+	totalSize := int64(0)
+
+	iter := job.ingestData.NewIter(ctx, job.keyRange.Start, job.keyRange.End, w.bufPool)
+	//nolint: errcheck
+	defer iter.Close()
+
+	for iter.First(); iter.Valid(); iter.Next() {
+		k, v := iter.Key(), iter.Value()
+		pairs = append(pairs, &sst.Pair{
+			Key:   k,
+			Value: v,
+		})
+		size += int64(len(k) + len(v))
+
+		if size >= w.writeBatchSize {
+			in := &ingestcli.WriteRequest{
+				Pairs: pairs,
+			}
+			if err := writeCli.Write(ctx, in); err != nil {
+				return nil, errors.Trace(err)
+			}
+			totalCount += int64(len(pairs))
+			totalSize += size
+			size = 0
+			pairs = pairs[:0]
+			iter.ReleaseBuf()
+		}
+	}
+
+	if iter.Error() != nil {
+		return nil, errors.Trace(iter.Error())
+	}
+
+	if len(pairs) > 0 {
+		in := &ingestcli.WriteRequest{
+			Pairs: pairs,
+		}
+		if err := writeCli.Write(ctx, in); err != nil {
+			return nil, errors.Trace(err)
+		}
+		totalCount += int64(len(pairs))
+		totalSize += size
+		size = 0
+		pairs = pairs[:0]
+		iter.ReleaseBuf()
+	}
+
+	resp, err := writeCli.CloseAndRecv(ctx)
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+
+	return &tikvWriteResult{
+		count:      totalCount,
+		totalBytes: totalSize,
+		sstFile:    resp.SSTFile,
+	}, nil
+}
+
+func (w *cloudRegionJobWorker) ingest(ctx context.Context, j *regionJob) error {
+	in := &ingestcli.IngestRequest{
+		Region:  j.region,
+		SSTFile: j.writeResult.sstFile,
+	}
+	err := w.ingestCli.Ingest(ctx, in)
+	if err != nil {
+		// TODO, we should let outer logic handle stage transition, currently, OP
+		//  worker need a lot of change before we can do it, will refactor it later.
+
+		// TODO: choose target stage based on error.
+		j.convertStageTo(needRescan)
path: pkg/lightning/backend/local/job_worker.go
user: login: D3Hunter
body: will implement this later after the `ingestcli.Client` is implemented
created_at: 2025-04-14T10:48:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -16,20 +16,39 @@ package local
 
 import (
 	"context"
+	"io"
+	"strings"
 	"sync"
+	"time"
 
+	"github.com/pingcap/errors"
 	"github.com/pingcap/failpoint"
+	sst "github.com/pingcap/kvproto/pkg/import_sstpb"
 	"github.com/pingcap/kvproto/pkg/metapb"
+	"github.com/pingcap/tidb/br/pkg/logutil"
+	"github.com/pingcap/tidb/pkg/ingestor/ingestcli"
 	"github.com/pingcap/tidb/pkg/lightning/common"
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pingcap/tidb/pkg/lightning/membuf"
 	"github.com/pingcap/tidb/pkg/metrics"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"github.com/tikv/client-go/v2/oracle"
+	pdhttp "github.com/tikv/pd/client/http"
+	"go.uber.org/zap"
 )
 
+type regionJobWorker interface {
+	run(ctx context.Context) error
+}
+
 type regionJobBaseWorker struct {
 	jobInCh  chan *regionJob
 	jobOutCh chan *regionJob
 	jobWg    *sync.WaitGroup
 
-	doRunJobFn func(ctx context.Context, job *regionJob) error
+	writeFn  func(ctx context.Context, j *regionJob) (*tikvWriteResult, error)
+	ingestFn func(ctx context.Context, j *regionJob) error
+	runJobFn func(ctx context.Context, job *regionJob) error
path: pkg/lightning/backend/local/job_worker.go
user: login: lance6716
body: My first feeling is write and ingest are sub-steps of runJob, they are not at the same level of concepts. Can you add another precheckFn and delete runJobFn for the special logic of opRegionJobWorker?
created_at: 2025-04-15T12:37:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,6 +136,236 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 	}
 }
 
+// doRunJob handles a regionJob and tries to convert it to ingested stage.
+// If non-retryable error occurs, it will return the error.
+// If retryable error occurs, it will return nil and caller should check the stage
+// of the regionJob to determine what to do with it.
+func (w *regionJobBaseWorker) doRunJob(ctx context.Context, job *regionJob) error {
+	for {
+		// the job might in wrote stage if it comes from retry
+		if job.stage == regionScanned {
+			// writes the data to TiKV and mark this job as wrote stage.
+			// we don't need to do cleanup for the pairs written to TiKV if encounters
+			// an error, TiKV will take the responsibility to do so.
+			// TODO: let client-go provide a high-level write interface.
+			res, err := w.writeFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				// currently only one case will restart write
+				if strings.Contains(err.Error(), "RequestTooNew") {
+					// TiKV hasn't synced the newest region info with PD, it's ok to
+					// rewrite without rescan.
+					job.convertStageTo(regionScanned)
+				} else {
+					job.convertStageTo(needRescan)
+				}
+				job.lastRetryableErr = err
+				log.FromContext(ctx).Warn("meet retryable error when writing to TiKV",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				return nil
+			}
+			if res.emptyJob {
+				job.convertStageTo(ingested)
+			} else {
+				job.writeResult = res
+				job.convertStageTo(wrote)
+			}
+		}
+
+		// if the job is empty, it might go to ingested stage directly.
path: pkg/lightning/backend/local/job_worker.go
user: login: lance6716
body: this should be put above line 170?
created_at: 2025-04-15T12:39:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,6 +136,236 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 	}
 }
 
+// doRunJob handles a regionJob and tries to convert it to ingested stage.
+// If non-retryable error occurs, it will return the error.
+// If retryable error occurs, it will return nil and caller should check the stage
+// of the regionJob to determine what to do with it.
+func (w *regionJobBaseWorker) doRunJob(ctx context.Context, job *regionJob) error {
+	for {
+		// the job might in wrote stage if it comes from retry
+		if job.stage == regionScanned {
+			// writes the data to TiKV and mark this job as wrote stage.
+			// we don't need to do cleanup for the pairs written to TiKV if encounters
+			// an error, TiKV will take the responsibility to do so.
+			// TODO: let client-go provide a high-level write interface.
+			res, err := w.writeFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				// currently only one case will restart write
+				if strings.Contains(err.Error(), "RequestTooNew") {
+					// TiKV hasn't synced the newest region info with PD, it's ok to
+					// rewrite without rescan.
+					job.convertStageTo(regionScanned)
+				} else {
+					job.convertStageTo(needRescan)
+				}
+				job.lastRetryableErr = err
+				log.FromContext(ctx).Warn("meet retryable error when writing to TiKV",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				return nil
+			}
+			if res.emptyJob {
+				job.convertStageTo(ingested)
path: pkg/lightning/backend/local/job_worker.go
user: login: lance6716
body: these `j.convertStageTo` are written outside of `writeFn`. But for `ingestFn` they are written inside `ingestFn`. It's not tidy.
created_at: 2025-04-15T12:43:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -16,20 +16,39 @@ package local
 
 import (
 	"context"
+	"io"
+	"strings"
 	"sync"
+	"time"
 
+	"github.com/pingcap/errors"
 	"github.com/pingcap/failpoint"
+	sst "github.com/pingcap/kvproto/pkg/import_sstpb"
 	"github.com/pingcap/kvproto/pkg/metapb"
+	"github.com/pingcap/tidb/br/pkg/logutil"
+	"github.com/pingcap/tidb/pkg/ingestor/ingestcli"
 	"github.com/pingcap/tidb/pkg/lightning/common"
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pingcap/tidb/pkg/lightning/membuf"
 	"github.com/pingcap/tidb/pkg/metrics"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"github.com/tikv/client-go/v2/oracle"
+	pdhttp "github.com/tikv/pd/client/http"
+	"go.uber.org/zap"
 )
 
+type regionJobWorker interface {
+	run(ctx context.Context) error
+}
+
 type regionJobBaseWorker struct {
 	jobInCh  chan *regionJob
 	jobOutCh chan *regionJob
 	jobWg    *sync.WaitGroup
 
-	doRunJobFn func(ctx context.Context, job *regionJob) error
+	writeFn  func(ctx context.Context, j *regionJob) (*tikvWriteResult, error)
+	ingestFn func(ctx context.Context, j *regionJob) error
+	runJobFn func(ctx context.Context, job *regionJob) error
path: pkg/lightning/backend/local/job_worker.go
user: login: D3Hunter
body: yes, `writeFn/ingestFn` are part of running one job, they are not part of the worker which runs many jobs, initially I tried to abstract a jobRunner, `op/cloud` have different implementation, and they share the same worker, but it doesn't make code more clearer, so i quit.

> add another precheckFn

good idea, i will try make it a `preRunJobFn`
created_at: 2025-04-15T14:06:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,6 +136,236 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 	}
 }
 
+// doRunJob handles a regionJob and tries to convert it to ingested stage.
+// If non-retryable error occurs, it will return the error.
+// If retryable error occurs, it will return nil and caller should check the stage
+// of the regionJob to determine what to do with it.
+func (w *regionJobBaseWorker) doRunJob(ctx context.Context, job *regionJob) error {
+	for {
+		// the job might in wrote stage if it comes from retry
+		if job.stage == regionScanned {
+			// writes the data to TiKV and mark this job as wrote stage.
+			// we don't need to do cleanup for the pairs written to TiKV if encounters
+			// an error, TiKV will take the responsibility to do so.
+			// TODO: let client-go provide a high-level write interface.
+			res, err := w.writeFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				// currently only one case will restart write
+				if strings.Contains(err.Error(), "RequestTooNew") {
+					// TiKV hasn't synced the newest region info with PD, it's ok to
+					// rewrite without rescan.
+					job.convertStageTo(regionScanned)
+				} else {
+					job.convertStageTo(needRescan)
+				}
+				job.lastRetryableErr = err
+				log.FromContext(ctx).Warn("meet retryable error when writing to TiKV",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				return nil
+			}
+			if res.emptyJob {
+				job.convertStageTo(ingested)
+			} else {
+				job.writeResult = res
+				job.convertStageTo(wrote)
+			}
+		}
+
+		// if the job is empty, it might go to ingested stage directly.
path: pkg/lightning/backend/local/job_worker.go
user: login: D3Hunter
body: it's trying to explain why we need to skip calling `ingestFn`

I hope we can refactor this part to following the state machine, now there are too many place changing fields of job
created_at: 2025-04-15T14:30:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,6 +136,236 @@ func (w *regionJobBaseWorker) run(ctx context.Context) error {
 	}
 }
 
+// doRunJob handles a regionJob and tries to convert it to ingested stage.
+// If non-retryable error occurs, it will return the error.
+// If retryable error occurs, it will return nil and caller should check the stage
+// of the regionJob to determine what to do with it.
+func (w *regionJobBaseWorker) doRunJob(ctx context.Context, job *regionJob) error {
+	for {
+		// the job might in wrote stage if it comes from retry
+		if job.stage == regionScanned {
+			// writes the data to TiKV and mark this job as wrote stage.
+			// we don't need to do cleanup for the pairs written to TiKV if encounters
+			// an error, TiKV will take the responsibility to do so.
+			// TODO: let client-go provide a high-level write interface.
+			res, err := w.writeFn(ctx, job)
+			if err != nil {
+				if !w.isRetryableImportTiKVError(err) {
+					return err
+				}
+				// currently only one case will restart write
+				if strings.Contains(err.Error(), "RequestTooNew") {
+					// TiKV hasn't synced the newest region info with PD, it's ok to
+					// rewrite without rescan.
+					job.convertStageTo(regionScanned)
+				} else {
+					job.convertStageTo(needRescan)
+				}
+				job.lastRetryableErr = err
+				log.FromContext(ctx).Warn("meet retryable error when writing to TiKV",
+					log.ShortError(err), zap.Stringer("job stage", job.stage))
+				return nil
+			}
+			if res.emptyJob {
+				job.convertStageTo(ingested)
path: pkg/lightning/backend/local/job_worker.go
user: login: D3Hunter
body: yes, it's hard to refactor this part in a single PR, i write a todo in `cloudRegionJobWorker.ingest`
```go
		// TODO, we should let outer logic handle stage transition, currently, OP
		//  worker need a lot of change before we can do it, will refactor it later.
```
created_at: 2025-04-15T14:31:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -16,20 +16,39 @@ package local
 
 import (
 	"context"
+	"io"
+	"strings"
 	"sync"
+	"time"
 
+	"github.com/pingcap/errors"
 	"github.com/pingcap/failpoint"
+	sst "github.com/pingcap/kvproto/pkg/import_sstpb"
 	"github.com/pingcap/kvproto/pkg/metapb"
+	"github.com/pingcap/tidb/br/pkg/logutil"
+	"github.com/pingcap/tidb/pkg/ingestor/ingestcli"
 	"github.com/pingcap/tidb/pkg/lightning/common"
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pingcap/tidb/pkg/lightning/membuf"
 	"github.com/pingcap/tidb/pkg/metrics"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"github.com/tikv/client-go/v2/oracle"
+	pdhttp "github.com/tikv/pd/client/http"
+	"go.uber.org/zap"
 )
 
+type regionJobWorker interface {
+	run(ctx context.Context) error
+}
+
 type regionJobBaseWorker struct {
 	jobInCh  chan *regionJob
 	jobOutCh chan *regionJob
 	jobWg    *sync.WaitGroup
 
-	doRunJobFn func(ctx context.Context, job *regionJob) error
+	writeFn  func(ctx context.Context, j *regionJob) (*tikvWriteResult, error)
+	ingestFn func(ctx context.Context, j *regionJob) error
+	runJobFn func(ctx context.Context, job *regionJob) error
path: pkg/lightning/backend/local/job_worker.go
user: login: D3Hunter
body: https://github.com/pingcap/tidb/pull/60543/commits/14fcb70432def3eda36184a17cee6ded38d53985
created_at: 2025-04-15T14:34:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: tiprow[bot]
created_at: 2025-04-14T10:34:49Z
body: Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with `/ok-to-test` in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using `/test all`.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=pingcap%2Ftidb).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

author_association: NONE
user: login: codecov[bot]
created_at: 2025-04-14T11:02:10Z
body: ## [Codecov](https://app.codecov.io/gh/pingcap/tidb/pull/60543?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) Report
Attention: Patch coverage is `76.34409%` with `66 lines` in your changes missing coverage. Please review.
> Please [upload](https://docs.codecov.com/docs/codecov-uploader) report for BASE (`feature/ng-tidb@c7d7a96`). [Learn more](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap#section-missing-base-commit) about missing BASE report.

<details><summary>Additional details and impacted files</summary>


```diff
@@                 Coverage Diff                  @@
##             feature/ng-tidb     #60543   +/-   ##
====================================================
  Coverage                   ?   73.8241%           
====================================================
  Files                      ?       1742           
  Lines                      ?     481523           
  Branches                   ?          0           
====================================================
  Hits                       ?     355480           
  Misses                     ?     104077           
  Partials                   ?      21966           
```

| [Flag](https://app.codecov.io/gh/pingcap/tidb/pull/60543/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/pingcap/tidb/pull/60543/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `45.6216% <28.8209%> (?)` | |
| [unit](https://app.codecov.io/gh/pingcap/tidb/pull/60543/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `72.4551% <79.2452%> (?)` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/pingcap/tidb/pull/60543/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | |
|---|---|---|
| [dumpling](https://app.codecov.io/gh/pingcap/tidb/pull/60543/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `52.6910% <0.0000%> (?)` | |
| [parser](https://app.codecov.io/gh/pingcap/tidb/pull/60543/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `∅ <0.0000%> (?)` | |
| [br](https://app.codecov.io/gh/pingcap/tidb/pull/60543/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `47.2367% <0.0000%> (?)` | |

</details>
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-14T11:14:31Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-14T11:14:57Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60543#issuecomment-2801362965):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-14T12:40:54Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-14T12:41:16Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60543#issuecomment-2801583853):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-15T16:39:04Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-15T16:39:27Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60543#issuecomment-2806803403):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-04-16T03:57:45Z
body: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/pingcap/tidb/pull/60543#pullrequestreview-2770584258" title="Approved">lance6716</a>*, *<a href="https://github.com/pingcap/tidb/pull/60543#pullrequestreview-2767033194" title="Approved">tangenta</a>*
**Once this PR has been reviewed and has the lgtm label**, please ask for approval from [d3hunter](https://github.com/d3hunter), **ensuring that each of them provides their approval before proceeding**. For more information see [the Code Review Process](https://book.prow.tidb.net/#/workflows/pr).

The full list of commands accepted by this bot can be found [here](https://prow.tidb.net/command-help?repo=pingcap%2Ftidb).

<details open>
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/pingcap/tidb/blob/feature/ng-tidb/OWNERS)~~ [lance6716,tangenta]
- **[pkg/lightning/OWNERS](https://github.com/pingcap/tidb/blob/feature/ng-tidb/pkg/lightning/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["d3hunter"]} -->
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-04-16T03:57:48Z
body: [LGTM Timeline notifier]
---
Timeline:

- `2025-04-15 06:50:05.998201825 +0000 UTC m=+2757499.682437915`: :ballot_box_with_check: agreed by [tangenta](https://github.com/tangenta).
- `2025-04-16 03:57:47.881745697 +0000 UTC m=+2833561.565981793`: :ballot_box_with_check: agreed by [lance6716](https://github.com/lance6716).
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-17T03:01:04Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-17T03:01:25Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60543#issuecomment-2811604908):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-17T03:35:49Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-17T03:36:11Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60543#issuecomment-2811652283):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-17T04:22:49Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-17T04:23:12Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60543#issuecomment-2811699099):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE