Repository: pingcap/tidb
Pull request #60569: ingestor: abstract http client and impl retry for tikv-worker | tidb-test=ccf9f5de6817ed2e54c18f2c1757056d2c8b4e4c
Author: login: tangenta
Labels: name: size/XXL
description: Denotes a PR that changes 1000+ lines, ignoring generated files.
name: release-note-none
description: Denotes a PR that doesn't merit a release note.
name: approved
name: lgtm
Created: 2025-04-15T06:22:57Z | Merged: 2025-04-27T07:21:05Z

<!--

Thank you for contributing to TiDB!

PR Title Format:
1. pkg [, pkg2, pkg3]: what's changed
2. *: what's changed

-->

### What problem does this PR solve?
<!--

Please create an issue first to describe the problem.

There MUST be one line starting with "Issue Number:  " and
linking the relevant issues via the "close" or "ref".

For more info, check https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/contribute-code.html#referring-to-an-issue.

-->

Issue Number: ref https://github.com/pingcap/tidb/issues/60418

Problem Summary:

### What changed and how does it work?

- Encapsulate the http request and add retry logic.
- Implement `WriteClient` and `Client`.

### Check List

Tests <!-- At least one of them must be included. -->

- [x] Unit test
- [ ] Integration test
- [x] Manual test (add detailed scripts or steps below)
TBD
- [ ] No need to test
  > - [ ] I checked and no code files have been changed.
  > <!-- Or your custom  "No need to test" reasons -->

Side effects

- [ ] Performance regression: Consumes more CPU
- [ ] Performance regression: Consumes more Memory
- [ ] Breaking backward compatibility

Documentation

- [ ] Affects user behaviors
- [ ] Contains syntax changes
- [ ] Contains variable changes
- [ ] Contains experimental features
- [ ] Changes MySQL compatibility

### Release note

<!-- compatibility change, improvement, bugfix, and new feature need a release note -->

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

```release-note
None
```



Diff:
diff --git a/pkg/ingestor/ingestcli/BUILD.bazel b/pkg/ingestor/ingestcli/BUILD.bazel
index 44780f33d382c..b9d285fbe9a02 100644
--- a/pkg/ingestor/ingestcli/BUILD.bazel
+++ b/pkg/ingestor/ingestcli/BUILD.bazel
@@ -1,12 +1,36 @@
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
+load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
 
 go_library(
     name = "ingestcli",
-    srcs = ["interface.go"],
+    srcs = [
+        "client.go",
+        "interface.go",
+    ],
     importpath = "github.com/pingcap/tidb/pkg/ingestor/ingestcli",
     visibility = ["//visibility:public"],
     deps = [
         "//br/pkg/restore/split",
+        "//pkg/util",
+        "@com_github_gogo_protobuf//proto",
+        "@com_github_pingcap_errors//:errors",
+        "@com_github_pingcap_kvproto//pkg/errorpb",
+        "@com_github_pingcap_kvproto//pkg/import_sstpb",
+        "@org_uber_go_atomic//:atomic",
+    ],
+)
+
+go_test(
+    name = "ingestcli_test",
+    timeout = "short",
+    srcs = ["client_test.go"],
+    embed = [":ingestcli"],
+    flaky = True,
+    shard_count = 4,
+    deps = [
+        "//br/pkg/restore/split",
+        "@com_github_pingcap_kvproto//pkg/errorpb",
         "@com_github_pingcap_kvproto//pkg/import_sstpb",
+        "@com_github_pingcap_kvproto//pkg/metapb",
+        "@com_github_stretchr_testify//require",
     ],
 )
diff --git a/pkg/ingestor/ingestcli/client.go b/pkg/ingestor/ingestcli/client.go
new file mode 100644
index 0000000000000..62bf13b61b629
--- /dev/null
+++ b/pkg/ingestor/ingestcli/client.go
@@ -0,0 +1,247 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	ID         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg         *util.ErrorGroupWithRecover
+	sendReqErr atomic.Error
+	writer     *io.PipeWriter
+	reader     *io.PipeReader
+	sstMeta    *nextGenSSTMeta
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr // PipeReader will be closed by the httpClient.Do automatically
+	w.writer = pw
+	return nil
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, err1 := io.ReadAll(resp.Body)
+			if err1 != nil {
+				w.sendReqErr.Store(err1)
+				return fmt.Errorf("failed to readAll response: %s", err1.Error())
+			}
+			err = fmt.Errorf("failed to send chunked request: %s", string(body))
+			w.sendReqErr.Store(err)
+			return err
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		res := &nextGenResp{}
+		if err := json.Unmarshal(data, res); err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		w.sstMeta = &res.SstMeta
+		return nil
+	})
+}
+
+func (w *writeClient) cause(err error) error {
+	if reqErr := w.sendReqErr.Load(); reqErr != nil {
+		return errors.Trace(reqErr)
+	}
+	return errors.Trace(err)
+}
+
+func (w *writeClient) Write(req *WriteRequest) (err error) {
+	var buf bytes.Buffer
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.LittleEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.LittleEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	if _, err := w.writer.Write(buf.Bytes()); err != nil {
+		return w.cause(err)
+	}
+	return nil
+}
+
+func (w *writeClient) Recv() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	return &WriteResponse{nextGenSSTMeta: w.sstMeta}, nil
+}
+
+func (w *writeClient) Close() {
+	//nolint: errcheck
+	_ = w.writer.Close()
+	//nolint: errcheck
+	_ = w.eg.Wait()
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	splitCli      split.SplitClient
+}
+
+// NewClient creates a new Client instance.
+func NewClient(tikvWorkerURL string, clusterID uint64, httpClient *http.Client, splitCli split.SplitClient) Client {
+	return &client{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		httpClient:    httpClient,
+		splitCli:      splitCli,
+	}
+}
+
+func (c *client) WriteClient(ctx context.Context, commitTS uint64) (WriteClient, error) {
+	cli := newWriteClient(c.tikvWorkerURL, c.clusterID, c.httpClient, commitTS)
+	err := cli.init(ctx)
+	return cli, err
+}
+
+func (c *client) Ingest(ctx context.Context, in *IngestRequest) error {
+	ri := in.Region.Region
+	store, err := c.splitCli.GetStore(ctx, in.Region.Leader.GetStoreId())
+	if err != nil {
+		return errors.Trace(err)
+	}
+	url := fmt.Sprintf("http://%s/ingest_s3?cluster_id=%d&region_id=%d&epoch_version=%d",
+		store.GetStatusAddress(), c.clusterID, ri.Id, ri.RegionEpoch.Version)
+
+	data, err := json.Marshal(&in.WriteResp.nextGenSSTMeta)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	bodyRd := bytes.NewReader(data)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, bodyRd)
+	if err != nil {
+		return errors.Trace(err)
+	}
+
+	resp, err := c.httpClient.Do(req)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		body, err1 := io.ReadAll(resp.Body)
+		if err1 != nil {
+			return fmt.Errorf("failed to readAll response: %s", err1.Error())
+		}
+		var pbErr errorpb.Error
+		if err := proto.Unmarshal(body, &pbErr); err != nil {
+			return fmt.Errorf("failed to unmarshal error(%s): %s", string(body), err)
+		}
+		return &PBError{Err: &pbErr}
+	}
+	return nil
+}
+
+// PBError is a implementation of error.
+type PBError struct {
+	Err *errorpb.Error
+}
+
+// Error implements the error.
+func (re *PBError) Error() string {
+	return re.Err.GetMessage()
+}
diff --git a/pkg/ingestor/ingestcli/client_test.go b/pkg/ingestor/ingestcli/client_test.go
new file mode 100644
index 0000000000000..bf5644781c851
--- /dev/null
+++ b/pkg/ingestor/ingestcli/client_test.go
@@ -0,0 +1,146 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/kvproto/pkg/import_sstpb"
+	"github.com/pingcap/kvproto/pkg/metapb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/stretchr/testify/require"
+)
+
+func TestWriteClientWriteChunk(t *testing.T) {
+	sstMeta := nextGenResp{nextGenSSTMeta{ID: 1, Smallest: []int{0}, Biggest: []int{1}, MetaOffset: 1, CommitTs: 1}}
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, err := io.ReadAll(r.Body)
+		require.NoError(t, err)
+		expected := []byte("\x03\x00key\x05\x00\x00\x00value")
+		require.Equal(t, expected, body)
+		w.WriteHeader(http.StatusOK)
+		sstMetaBytes, err := json.Marshal(sstMeta)
+		require.NoError(t, err)
+		_, err = w.Write(sstMetaBytes)
+		require.NoError(t, err)
+	}))
+	defer server.Close()
+
+	client := newWriteClient(server.URL, 12345, server.Client(), 67890)
+	client.commitTS = 67890
+	defer client.Close()
+	err := client.init(context.Background())
+	require.NoError(t, err)
+
+	req := &WriteRequest{
+		Pairs: []*import_sstpb.Pair{
+			{Key: []byte("key"), Value: []byte("value")},
+		},
+	}
+	err = client.Write(req)
+	require.NoError(t, err)
+
+	resp, err := client.Recv()
+	require.NoError(t, err)
+	require.NotNil(t, resp)
+	require.EqualValues(t, &sstMeta.SstMeta, resp.nextGenSSTMeta)
+}
+
+func TestClientWriteServerError(t *testing.T) {
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusInternalServerError)
+		w.Write([]byte("internal server error"))
+	}))
+	defer server.Close()
+
+	client := newWriteClient(server.URL, 12345, server.Client(), 67890)
+	err := client.init(context.Background())
+	require.NoError(t, err)
+
+	req := &WriteRequest{Pairs: []*import_sstpb.Pair{{Key: []byte("key"), Value: []byte("value")}}}
+	err = client.Write(req)
+	require.NoError(t, err) // Error only return when pipeWriter is closed?
+
+	_, err = client.Recv()
+	require.Error(t, err)
+	require.Contains(t, err.Error(), "internal server error")
+}
+
+func TestClientIngest(t *testing.T) {
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		require.Equal(t, "/ingest_s3", r.URL.Path)
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	statusAddr := strings.TrimPrefix(server.URL, "http://")
+	client := NewClient(server.URL, 12345, server.Client(), &storeClient{addr: statusAddr})
+	req := &IngestRequest{
+		WriteResp: &WriteResponse{
+			nextGenSSTMeta: &nextGenSSTMeta{
+				ID: 1,
+			},
+		},
+		Region: &split.RegionInfo{Region: &metapb.Region{Id: 1, RegionEpoch: &metapb.RegionEpoch{Version: 1}}},
+	}
+	err := client.Ingest(context.Background(), req)
+	require.NoError(t, err)
+}
+
+func TestClientIngestError(t *testing.T) {
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusInternalServerError)
+		pbErr := &errorpb.Error{Message: "test error"}
+		data, err := pbErr.Marshal()
+		require.NoError(t, err)
+		w.Write(data)
+	}))
+	defer server.Close()
+
+	// serverURL, err := url.Parse(server.URL)
+	// require.NoError(t, err)
+	statusAddr := strings.TrimPrefix(server.URL, "http://")
+	client := NewClient(server.URL, 12345, server.Client(), &storeClient{addr: statusAddr})
+	req := &IngestRequest{
+		WriteResp: &WriteResponse{
+			nextGenSSTMeta: &nextGenSSTMeta{
+				ID: 1,
+			},
+		},
+		Region: &split.RegionInfo{Region: &metapb.Region{Id: 1, RegionEpoch: &metapb.RegionEpoch{Version: 1}}},
+	}
+	err := client.Ingest(context.Background(), req)
+	require.Error(t, err)
+	require.Contains(t, err.Error(), "test error")
+}
+
+type storeClient struct {
+	split.SplitClient
+	addr string
+}
+
+func (sc *storeClient) GetStore(_ context.Context, _ uint64) (*metapb.Store, error) {
+	return &metapb.Store{
+		Address:       sc.addr,
+		StatusAddress: sc.addr,
+	}, nil
+}
diff --git a/pkg/ingestor/ingestcli/interface.go b/pkg/ingestor/ingestcli/interface.go
index c9e4356fab553..108305b668c27 100644
--- a/pkg/ingestor/ingestcli/interface.go
+++ b/pkg/ingestor/ingestcli/interface.go
@@ -23,33 +23,28 @@ import (
 
 // WriteRequest is the request to write KV to storage layer.
 type WriteRequest struct {
-	ClusterID string
-	Pairs     []*import_sstpb.Pair
+	Pairs []*import_sstpb.Pair
 }
 
 // WriteResponse is the response of Write.
 type WriteResponse struct {
-	SSTFile string
+	nextGenSSTMeta *nextGenSSTMeta
 }
 
 // IngestRequest is the request to ingest SST to storage layer.
 type IngestRequest struct {
-	Region  *split.RegionInfo
-	SSTFile string
+	Region    *split.RegionInfo
+	WriteResp *WriteResponse
 }
 
 // WriteClient is the client for writing KV to storage layer.
 // you can call Write multiple times to write data in a stream way before close.
 // when close, the server will return the info of SSTs generated by the server.
+// It is caller's responsibility to call Close() when it meets an error.
 type WriteClient interface {
-	// Write KV to storage layer.
-	Write(ctx context.Context, in *WriteRequest) error
-	// CloseAndRecv the client, get the response and close the client.
-	// the response should contain the SST info.
-	// we name it this way to mimic GRPC stream, and to avoid confusion with normal
-	// Close method which should always be called after creation, but in here,
-	// it's not a must.
-	CloseAndRecv(ctx context.Context) (*WriteResponse, error)
+	Write(*WriteRequest) error
+	Recv() (*WriteResponse, error)
+	Close()
 }
 
 // Client is the interface to write KV and ingest SST to storage layer.
@@ -65,7 +60,8 @@ type WriteClient interface {
 //	cli.Ingest(xxx)
 type Client interface {
 	// WriteClient returns a WriteClient to write KV to storage layer.
-	WriteClient(ctx context.Context) (WriteClient, error)
+	// WriteClient methods share the same context passed here.
+	WriteClient(ctx context.Context, commitTS uint64) (WriteClient, error)
 	// Ingest ingests the SST to storage layer.
 	Ingest(ctx context.Context, in *IngestRequest) error
 }
diff --git a/pkg/ingestor/ingestcli/mock/client_mock.go b/pkg/ingestor/ingestcli/mock/client_mock.go
index 7ea29946c891c..c4340bd045351 100644
--- a/pkg/ingestor/ingestcli/mock/client_mock.go
+++ b/pkg/ingestor/ingestcli/mock/client_mock.go
@@ -60,18 +60,18 @@ func (mr *MockClientMockRecorder) Ingest(arg0, arg1 any) *gomock.Call {
 }
 
 // WriteClient mocks base method.
-func (m *MockClient) WriteClient(arg0 context.Context) (ingestcli.WriteClient, error) {
+func (m *MockClient) WriteClient(arg0 context.Context, arg1 uint64) (ingestcli.WriteClient, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "WriteClient", arg0)
+	ret := m.ctrl.Call(m, "WriteClient", arg0, arg1)
 	ret0, _ := ret[0].(ingestcli.WriteClient)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
 
 // WriteClient indicates an expected call of WriteClient.
-func (mr *MockClientMockRecorder) WriteClient(arg0 any) *gomock.Call {
+func (mr *MockClientMockRecorder) WriteClient(arg0, arg1 any) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "WriteClient", reflect.TypeOf((*MockClient)(nil).WriteClient), arg0)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "WriteClient", reflect.TypeOf((*MockClient)(nil).WriteClient), arg0, arg1)
 }
 
 // MockWriteClient is a mock of WriteClient interface.
@@ -102,31 +102,43 @@ func (m *MockWriteClient) ISGOMOCK() struct{} {
 	return struct{}{}
 }
 
-// CloseAndRecv mocks base method.
-func (m *MockWriteClient) CloseAndRecv(arg0 context.Context) (*ingestcli.WriteResponse, error) {
+// Close mocks base method.
+func (m *MockWriteClient) Close() {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "CloseAndRecv", arg0)
+	m.ctrl.Call(m, "Close")
+}
+
+// Close indicates an expected call of Close.
+func (mr *MockWriteClientMockRecorder) Close() *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Close", reflect.TypeOf((*MockWriteClient)(nil).Close))
+}
+
+// Recv mocks base method.
+func (m *MockWriteClient) Recv() (*ingestcli.WriteResponse, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Recv")
 	ret0, _ := ret[0].(*ingestcli.WriteResponse)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
 
-// CloseAndRecv indicates an expected call of CloseAndRecv.
-func (mr *MockWriteClientMockRecorder) CloseAndRecv(arg0 any) *gomock.Call {
+// Recv indicates an expected call of Recv.
+func (mr *MockWriteClientMockRecorder) Recv() *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CloseAndRecv", reflect.TypeOf((*MockWriteClient)(nil).CloseAndRecv), arg0)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Recv", reflect.TypeOf((*MockWriteClient)(nil).Recv))
 }
 
 // Write mocks base method.
-func (m *MockWriteClient) Write(arg0 context.Context, arg1 *ingestcli.WriteRequest) error {
+func (m *MockWriteClient) Write(arg0 *ingestcli.WriteRequest) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Write", arg0, arg1)
+	ret := m.ctrl.Call(m, "Write", arg0)
 	ret0, _ := ret[0].(error)
 	return ret0
 }
 
 // Write indicates an expected call of Write.
-func (mr *MockWriteClientMockRecorder) Write(arg0, arg1 any) *gomock.Call {
+func (mr *MockWriteClientMockRecorder) Write(arg0 any) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Write", reflect.TypeOf((*MockWriteClient)(nil).Write), arg0, arg1)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Write", reflect.TypeOf((*MockWriteClient)(nil).Write), arg0)
 }
diff --git a/pkg/lightning/backend/local/job_worker.go b/pkg/lightning/backend/local/job_worker.go
index 26bab0c847cf6..e771b6efd5e2d 100644
--- a/pkg/lightning/backend/local/job_worker.go
+++ b/pkg/lightning/backend/local/job_worker.go
@@ -282,10 +282,11 @@ func (w *objStoreRegionJobWorker) write(ctx context.Context, job *regionJob) (*t
 		return &tikvWriteResult{emptyJob: true}, nil
 	}
 
-	writeCli, err := w.ingestCli.WriteClient(ctx)
+	writeCli, err := w.ingestCli.WriteClient(ctx, job.ingestData.GetTS())
 	if err != nil {
 		return nil, errors.Trace(err)
 	}
+	defer writeCli.Close()
 	dataCommitTS := job.ingestData.GetTS()
 	intest.AssertFunc(func() bool {
 		timeOfTS := oracle.GetTimeFromTS(dataCommitTS)
@@ -323,7 +324,7 @@ func (w *objStoreRegionJobWorker) write(ctx context.Context, job *regionJob) (*t
 			in := &ingestcli.WriteRequest{
 				Pairs: pairs,
 			}
-			if err := writeCli.Write(ctx, in); err != nil {
+			if err := writeCli.Write(in); err != nil {
 				return nil, errors.Trace(err)
 			}
 			totalCount += int64(len(pairs))
@@ -342,7 +343,7 @@ func (w *objStoreRegionJobWorker) write(ctx context.Context, job *regionJob) (*t
 		in := &ingestcli.WriteRequest{
 			Pairs: pairs,
 		}
-		if err := writeCli.Write(ctx, in); err != nil {
+		if err := writeCli.Write(in); err != nil {
 			return nil, errors.Trace(err)
 		}
 		totalCount += int64(len(pairs))
@@ -351,22 +352,22 @@ func (w *objStoreRegionJobWorker) write(ctx context.Context, job *regionJob) (*t
 		iter.ReleaseBuf()
 	}
 
-	resp, err := writeCli.CloseAndRecv(ctx)
+	resp, err := writeCli.Recv()
 	if err != nil {
 		return nil, errors.Trace(err)
 	}
 
 	return &tikvWriteResult{
-		count:      totalCount,
-		totalBytes: totalSize,
-		sstFile:    resp.SSTFile,
+		count:            totalCount,
+		totalBytes:       totalSize,
+		nextGenWriteResp: resp,
 	}, nil
 }
 
 func (w *objStoreRegionJobWorker) ingest(ctx context.Context, j *regionJob) error {
 	in := &ingestcli.IngestRequest{
-		Region:  j.region,
-		SSTFile: j.writeResult.sstFile,
+		Region:    j.region,
+		WriteResp: j.writeResult.nextGenWriteResp,
 	}
 	err := w.ingestCli.Ingest(ctx, in)
 	if err != nil {
diff --git a/pkg/lightning/backend/local/job_worker_test.go b/pkg/lightning/backend/local/job_worker_test.go
index 6e2becd8160ab..f2ab8cbf206f0 100644
--- a/pkg/lightning/backend/local/job_worker_test.go
+++ b/pkg/lightning/backend/local/job_worker_test.go
@@ -192,7 +192,7 @@ func TestCloudRegionJobWorker(t *testing.T) {
 			stage:      regionScanned,
 			ingestData: mockIngestData{{[]byte("a"), []byte("a")}},
 		}
-		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(nil, errors.New("mock error"))
+		mockIngestCli.EXPECT().WriteClient(gomock.Any(), gomock.Any()).Return(nil, errors.New("mock error"))
 		writeRes, err := cloudW.write(context.Background(), job)
 		require.ErrorContains(t, err, "mock error")
 		require.Nil(t, writeRes)
@@ -206,8 +206,9 @@ func TestCloudRegionJobWorker(t *testing.T) {
 			ingestData: mockIngestData{{[]byte("a"), []byte("a")}},
 		}
 		writeCli := ingestclimock.NewMockWriteClient(ctrl)
-		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(writeCli, nil)
-		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(errors.New("mock error"))
+		mockIngestCli.EXPECT().WriteClient(gomock.Any(), gomock.Any()).Return(writeCli, nil)
+		writeCli.EXPECT().Write(gomock.Any()).Return(errors.New("mock error"))
+		writeCli.EXPECT().Close()
 		writeRes, err := cloudW.write(context.Background(), job)
 		require.ErrorContains(t, err, "mock error")
 		require.Nil(t, writeRes)
@@ -221,9 +222,10 @@ func TestCloudRegionJobWorker(t *testing.T) {
 			ingestData: mockIngestData{{[]byte("a"), []byte("a")}},
 		}
 		writeCli := ingestclimock.NewMockWriteClient(ctrl)
-		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(writeCli, nil)
-		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(nil)
-		writeCli.EXPECT().CloseAndRecv(gomock.Any()).Return(nil, errors.New("mock error"))
+		mockIngestCli.EXPECT().WriteClient(gomock.Any(), gomock.Any()).Return(writeCli, nil)
+		writeCli.EXPECT().Write(gomock.Any()).Return(nil)
+		writeCli.EXPECT().Recv().Return(nil, errors.New("mock error"))
+		writeCli.EXPECT().Close()
 		resp, err := cloudW.write(context.Background(), job)
 		require.ErrorContains(t, err, "mock error")
 		require.Nil(t, resp)
@@ -241,15 +243,15 @@ func TestCloudRegionJobWorker(t *testing.T) {
 			},
 		}
 		writeCli := ingestclimock.NewMockWriteClient(ctrl)
-		mockIngestCli.EXPECT().WriteClient(gomock.Any()).Return(writeCli, nil)
-		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(nil)
-		writeCli.EXPECT().Write(gomock.Any(), gomock.Any()).Return(nil)
-		writeCli.EXPECT().CloseAndRecv(gomock.Any()).Return(&ingestcli.WriteResponse{SSTFile: "the-file"}, nil)
+		mockIngestCli.EXPECT().WriteClient(gomock.Any(), gomock.Any()).Return(writeCli, nil)
+		writeCli.EXPECT().Write(gomock.Any()).Return(nil)
+		writeCli.EXPECT().Write(gomock.Any()).Return(nil)
+		writeCli.EXPECT().Recv().Return(&ingestcli.WriteResponse{}, nil)
+		writeCli.EXPECT().Close()
 		res, err := cloudW.write(context.Background(), job)
 		require.NoError(t, err)
 		require.EqualValues(t, 3, res.count)
 		require.EqualValues(t, 18, res.totalBytes)
-		require.Equal(t, "the-file", res.sstFile)
 		require.True(t, ctrl.Satisfied())
 	})
 
@@ -258,7 +260,7 @@ func TestCloudRegionJobWorker(t *testing.T) {
 			keyRange:    common.Range{Start: []byte("a"), End: []byte("z")},
 			stage:       wrote,
 			ingestData:  mockIngestData{},
-			writeResult: &tikvWriteResult{sstFile: "the-file"},
+			writeResult: &tikvWriteResult{},
 		}
 		mockIngestCli.EXPECT().Ingest(gomock.Any(), gomock.Any()).Return(errors.New("mock error"))
 		err := cloudW.ingest(context.Background(), job)
@@ -271,7 +273,7 @@ func TestCloudRegionJobWorker(t *testing.T) {
 			keyRange:    common.Range{Start: []byte("a"), End: []byte("z")},
 			stage:       wrote,
 			ingestData:  mockIngestData{},
-			writeResult: &tikvWriteResult{sstFile: "the-file"},
+			writeResult: &tikvWriteResult{},
 		}
 		mockIngestCli.EXPECT().Ingest(gomock.Any(), gomock.Any()).Return(nil)
 		err := cloudW.ingest(context.Background(), job)
diff --git a/pkg/lightning/backend/local/local.go b/pkg/lightning/backend/local/local.go
index 1650fda320419..9fc65db5b65ed 100644
--- a/pkg/lightning/backend/local/local.go
+++ b/pkg/lightning/backend/local/local.go
@@ -21,6 +21,7 @@ import (
 	"encoding/hex"
 	"math"
 	"net"
+	"net/http"
 	"path/filepath"
 	"strings"
 	"sync"
@@ -39,6 +40,7 @@ import (
 	"github.com/pingcap/tidb/br/pkg/version"
 	"github.com/pingcap/tidb/pkg/config/kerneltype"
 	"github.com/pingcap/tidb/pkg/infoschema"
+	"github.com/pingcap/tidb/pkg/ingestor/ingestcli"
 	"github.com/pingcap/tidb/pkg/lightning/backend"
 	"github.com/pingcap/tidb/pkg/lightning/backend/encode"
 	"github.com/pingcap/tidb/pkg/lightning/backend/external"
@@ -1376,7 +1378,12 @@ func (local *Backend) doImport(
 	var (
 		toCh            = jobToWorkerCh
 		afterExecuteJob func([]*metapb.Peer)
+		clusterID       uint64
 	)
+	if local.pdCli != nil {
+		clusterID = local.pdCli.GetClusterID(ctx)
+	}
+
 	failpoint.Inject("skipStartWorker", func() {
 		failpoint.Goto("afterStartWorker")
 	})
@@ -1386,7 +1393,7 @@ func (local *Backend) doImport(
 		afterExecuteJob = balancer.releaseStoreLoad
 	}
 	for i := 0; i < local.WorkerConcurrency; i++ {
-		worker := local.newRegionJobWorker(toCh, jobFromWorkerCh, &jobWg, afterExecuteJob)
+		worker := local.newRegionJobWorker(clusterID, toCh, jobFromWorkerCh, &jobWg, afterExecuteJob)
 		workGroup.Go(func() error {
 			return worker.run(workerCtx)
 		})
@@ -1434,6 +1441,7 @@ func (local *Backend) doImport(
 }
 
 func (local *Backend) newRegionJobWorker(
+	clusterID uint64,
 	toCh, jobFromWorkerCh chan *regionJob,
 	jobWg *sync.WaitGroup,
 	afterExecuteJob func([]*metapb.Peer),
@@ -1446,9 +1454,10 @@ func (local *Backend) newRegionJobWorker(
 		regenerateJobsFn: local.generateJobForRange,
 	}
 	if kerneltype.IsNextGen() {
+		// TODO: add support for TLS.
+		httpClient := &http.Client{}
 		cloudW := &objStoreRegionJobWorker{
-			// TODO fill the cli
-			ingestCli:      nil,
+			ingestCli:      ingestcli.NewClient(local.TiKVWorkerURL, clusterID, httpClient, local.splitCli),
 			writeBatchSize: local.KVWriteBatchSize,
 			bufPool:        local.engineMgr.getBufferPool(),
 		}
diff --git a/pkg/lightning/backend/local/local_test.go b/pkg/lightning/backend/local/local_test.go
index 87318d291f66c..a522e28da427c 100644
--- a/pkg/lightning/backend/local/local_test.go
+++ b/pkg/lightning/backend/local/local_test.go
@@ -1094,7 +1094,7 @@ func TestLocalWriteAndIngestPairsFailFast(t *testing.T) {
 		require.NoError(t, failpoint.Disable("github.com/pingcap/tidb/pkg/lightning/backend/local/WriteToTiKVNotEnoughDiskSpace"))
 	}()
 	toCh := make(chan *regionJob, 1)
-	worker := bak.newRegionJobWorker(toCh, make(chan *regionJob, 1), nil, nil)
+	worker := bak.newRegionJobWorker(1, toCh, make(chan *regionJob, 1), nil, nil)
 
 	toCh <- &regionJob{}
 	err := worker.run(context.Background())
@@ -1298,7 +1298,7 @@ func TestCheckPeersBusy(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := local.newRegionJobWorker(jobCh, jobOutCh, nil, nil)
+		worker := local.newRegionJobWorker(1, jobCh, jobOutCh, nil, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
@@ -1406,7 +1406,7 @@ func TestNotLeaderErrorNeedUpdatePeers(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := local.newRegionJobWorker(jobCh, jobOutCh, &jobWg, nil)
+		worker := local.newRegionJobWorker(1, jobCh, jobOutCh, &jobWg, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
@@ -1507,7 +1507,7 @@ func TestPartialWriteIngestErrorWontPanic(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := local.newRegionJobWorker(jobCh, jobOutCh, &jobWg, nil)
+		worker := local.newRegionJobWorker(1, jobCh, jobOutCh, &jobWg, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
@@ -1629,7 +1629,7 @@ func TestPartialWriteIngestBusy(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		worker := local.newRegionJobWorker(jobCh, jobOutCh, &jobWg, nil)
+		worker := local.newRegionJobWorker(1, jobCh, jobOutCh, &jobWg, nil)
 		err := worker.run(ctx)
 		require.NoError(t, err)
 	}()
diff --git a/pkg/lightning/backend/local/region_job.go b/pkg/lightning/backend/local/region_job.go
index a5f86596906ca..6b20a1028f553 100644
--- a/pkg/lightning/backend/local/region_job.go
+++ b/pkg/lightning/backend/local/region_job.go
@@ -36,6 +36,7 @@ import (
 	berrors "github.com/pingcap/tidb/br/pkg/errors"
 	"github.com/pingcap/tidb/br/pkg/logutil"
 	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/ingestor/ingestcli"
 	"github.com/pingcap/tidb/pkg/kv"
 	"github.com/pingcap/tidb/pkg/lightning/common"
 	"github.com/pingcap/tidb/pkg/lightning/config"
@@ -135,7 +136,7 @@ type tikvWriteResult struct {
 
 	// below fields are for cloud generation store engine.
 	// the filename of the written sst file by tikv-worker.
-	sstFile string
+	nextGenWriteResp *ingestcli.WriteResponse
 }
 
 type injectedBehaviour struct {


Reviews:
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T02:37:20Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T04:30:58Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T10:22:37Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T10:25:03Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T10:27:20Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T11:09:52Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T11:21:29Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T06:29:00Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T07:03:02Z
user: login: tangenta
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:55:15Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T09:01:12Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T09:02:57Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T04:43:50Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T11:15:58Z
user: login: D3Hunter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T02:40:44Z
user: login: D3Hunter
body: rest lgtm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T07:24:13Z
user: login: D3Hunter
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T07:31:40Z
user: login: GMHDBJD
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-27T06:01:58Z
user: login: tangenta
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-27T06:05:30Z
user: login: GMHDBJD
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-27T06:20:04Z

Review comments:
diff_hunk: @@ -24,6 +24,7 @@ import (
 // WriteRequest is the request to write KV to storage layer.
 type WriteRequest struct {
 	ClusterID string
+	ChunkID   int64
path: pkg/ingestor/ingestcli/interface.go
user: login: D3Hunter
body: WriteClient should maintain this `ChunkID`, after each success call of it, it increase it internally. As `WriteClient` do retry internally, the outer logic will retry the whole task if any write chunk fails, and a new WriteClient with new task-id will be created in this case
created_at: 2025-04-18T02:37:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"io"
+	"net/http"
+	"time"
+
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pkg/errors"
+	"github.com/prometheus/client_golang/prometheus"
+	"go.uber.org/zap"
+)
+
+var (
+	sleepDuration = 3 * time.Second
+	retryCount    = 6000 // 3s * 6000 = 5h
path: pkg/ingestor/ingestcli/util.go
user: login: D3Hunter
body: region job worker part will retry the job internally, we can retry for transient error here, maybe 5 times, with interval starting from 100ms should be fine, 
created_at: 2025-04-18T02:50:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"io"
+	"net/http"
+	"time"
+
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pkg/errors"
+	"github.com/prometheus/client_golang/prometheus"
+	"go.uber.org/zap"
+)
+
+var (
+	sleepDuration = 3 * time.Second
+	retryCount    = 6000 // 3s * 6000 = 5h
+)
+
+func retryableHTTPStatusCode(statusCode int) bool {
+	return statusCode == http.StatusServiceUnavailable || // 503
+		statusCode == http.StatusInternalServerError || // 500
+		statusCode == http.StatusRequestTimeout // 408
+}
+
+func sendRequestWithRetry(ctx context.Context, httpClient *http.Client,
+	method, url string, data []byte, retryCounter prometheus.Counter) (*http.Response, error) {
+	var (
+		req  *http.Request
+		resp *http.Response
+		err  error
+	)
+	for retry := 0; retry < retryCount; retry++ {
path: pkg/ingestor/ingestcli/util.go
user: login: D3Hunter
body: ```suggestion
	for range retryCount {
```
created_at: 2025-04-18T02:55:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"io"
+	"net/http"
+	"time"
+
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pkg/errors"
+	"github.com/prometheus/client_golang/prometheus"
+	"go.uber.org/zap"
+)
+
+var (
+	sleepDuration = 3 * time.Second
+	retryCount    = 6000 // 3s * 6000 = 5h
+)
+
+func retryableHTTPStatusCode(statusCode int) bool {
+	return statusCode == http.StatusServiceUnavailable || // 503
+		statusCode == http.StatusInternalServerError || // 500
+		statusCode == http.StatusRequestTimeout // 408
+}
+
+func sendRequestWithRetry(ctx context.Context, httpClient *http.Client,
+	method, url string, data []byte, retryCounter prometheus.Counter) (*http.Response, error) {
+	var (
+		req  *http.Request
+		resp *http.Response
+		err  error
+	)
+	for retry := 0; retry < retryCount; retry++ {
+		body := bytes.NewReader(data)
+		req, err = http.NewRequestWithContext(ctx, method, url, body)
+		if err != nil {
+			return nil, err
path: pkg/ingestor/ingestcli/util.go
user: login: D3Hunter
body: add errors.Trace
created_at: 2025-04-18T02:56:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,88 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"io"
+	"net/http"
+	"time"
+
+	"github.com/pingcap/tidb/pkg/lightning/log"
+	"github.com/pkg/errors"
+	"github.com/prometheus/client_golang/prometheus"
+	"go.uber.org/zap"
+)
+
+var (
+	sleepDuration = 3 * time.Second
+	retryCount    = 6000 // 3s * 6000 = 5h
+)
+
+func retryableHTTPStatusCode(statusCode int) bool {
+	return statusCode == http.StatusServiceUnavailable || // 503
+		statusCode == http.StatusInternalServerError || // 500
+		statusCode == http.StatusRequestTimeout // 408
+}
+
+func sendRequestWithRetry(ctx context.Context, httpClient *http.Client,
+	method, url string, data []byte, retryCounter prometheus.Counter) (*http.Response, error) {
+	var (
+		req  *http.Request
+		resp *http.Response
+		err  error
+	)
+	for retry := 0; retry < retryCount; retry++ {
+		body := bytes.NewReader(data)
+		req, err = http.NewRequestWithContext(ctx, method, url, body)
+		if err != nil {
+			return nil, err
+		}
+		resp, err = httpClient.Do(req)
+		if err != nil || retryableHTTPStatusCode(resp.StatusCode) {
+			statusCode := 0
+			if resp != nil {
+				statusCode = resp.StatusCode
+			}
+			log.FromContext(ctx).Warn("failed to send http request", zap.Error(err), zap.Int("status code", statusCode))
+			select {
+			case <-ctx.Done():
+				return nil, ctx.Err()
+			case <-time.After(sleepDuration):
+				if retryCounter != nil {
+					retryCounter.Inc()
+				}
+				continue
+			}
+		}
+		break
+	}
+	return resp, err
path: pkg/ingestor/ingestcli/util.go
user: login: D3Hunter
body: doitto
created_at: 2025-04-18T02:57:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,106 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"net/http"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        string
+	httpClient    *http.Client
+}
+
+func newWriteClient(tikvWorkerURL string, clusterID uint64, taskID string, httpClient *http.Client) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		taskID:        taskID,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) Write(ctx context.Context, in *WriteRequest) error {
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&task_id=%s&chunk_id=%d",
+		w.tikvWorkerURL, w.clusterID, w.taskID, in.ChunkID)
+
+	// TODO: pass retry counter metrics.
+	var buf bytes.Buffer
+	for _, pair := range in.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.BigEndian, keyLen); err != nil {
+			return err
+		}
+		buf.Write(pair.Key)
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.BigEndian, valLen); err != nil {
+			return err
+		}
+		buf.Write(pair.Value)
+	}
+
+	data := buf.Bytes()
+	_, err := sendRequest(ctx, w.httpClient, "PUT", url, data, nil)
+	return err
+}
+
+func (w *writeClient) CloseAndRecv(ctx context.Context) (*WriteResponse, error) {
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&task_id=%s&build=true&compression=zstd",
+		w.tikvWorkerURL, w.clusterID, w.taskID)
+
+	// TODO: pass retry counter metrics.
+	data, err := sendRequest(ctx, w.httpClient, "POST", url, nil, nil)
+	if err != nil {
+		return nil, err
+	}
+
+	result := new(fileNameResult)
+	err = json.Unmarshal(data, result)
+	if err != nil {
+		return nil, err
+	}
+
+	return &WriteResponse{SSTFile: result.FileName}, nil
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        string
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: in my understanding of the API, taskID represent one SST file, chunkID means one chunk of data for one SST, so each `WriteClient` need a taskID
created_at: 2025-04-18T02:59:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,106 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"net/http"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        string
+	httpClient    *http.Client
+}
+
+func newWriteClient(tikvWorkerURL string, clusterID uint64, taskID string, httpClient *http.Client) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		taskID:        taskID,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) Write(ctx context.Context, in *WriteRequest) error {
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&task_id=%s&chunk_id=%d",
+		w.tikvWorkerURL, w.clusterID, w.taskID, in.ChunkID)
+
+	// TODO: pass retry counter metrics.
+	var buf bytes.Buffer
+	for _, pair := range in.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.BigEndian, keyLen); err != nil {
+			return err
+		}
+		buf.Write(pair.Key)
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.BigEndian, valLen); err != nil {
+			return err
+		}
+		buf.Write(pair.Value)
+	}
+
+	data := buf.Bytes()
+	_, err := sendRequest(ctx, w.httpClient, "PUT", url, data, nil)
+	return err
+}
+
+func (w *writeClient) CloseAndRecv(ctx context.Context) (*WriteResponse, error) {
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&task_id=%s&build=true&compression=zstd",
+		w.tikvWorkerURL, w.clusterID, w.taskID)
+
+	// TODO: pass retry counter metrics.
+	data, err := sendRequest(ctx, w.httpClient, "POST", url, nil, nil)
+	if err != nil {
+		return nil, err
+	}
+
+	result := new(fileNameResult)
+	err = json.Unmarshal(data, result)
+	if err != nil {
+		return nil, err
+	}
+
+	return &WriteResponse{SSTFile: result.FileName}, nil
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        string
+	httpClient    *http.Client
+}
+
+func (c *client) WriteClient(ctx context.Context) (WriteClient, error) {
+	return newWriteClient(c.tikvWorkerURL, c.clusterID, c.taskID, c.httpClient), nil
+}
+
+type fileNameResult struct {
+	FileName string `json:"file_name"`
+}
+
+func (c *client) Ingest(ctx context.Context, in *IngestRequest) error {
+	return nil
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: the API not defined already ? please leave a todo here
created_at: 2025-04-18T03:00:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,14 +41,8 @@ type IngestRequest struct {
 // you can call Write multiple times to write data in a stream way before close.
 // when close, the server will return the info of SSTs generated by the server.
 type WriteClient interface {
-	// Write KV to storage layer.
-	Write(ctx context.Context, in *WriteRequest) error
-	// CloseAndRecv the client, get the response and close the client.
-	// the response should contain the SST info.
-	// we name it this way to mimic GRPC stream, and to avoid confusion with normal
-	// Close method which should always be called after creation, but in here,
-	// it's not a must.
-	CloseAndRecv(ctx context.Context) (*WriteResponse, error)
+	WriteChunk(*WriteRequest) error
+	Close() (*WriteResponse, error)
path: pkg/ingestor/ingestcli/interface.go
user: login: D3Hunter
body: I purposely named as CloseAndRecv as the comment says, why change it back?
created_at: 2025-04-21T10:22:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,14 +41,8 @@ type IngestRequest struct {
 // you can call Write multiple times to write data in a stream way before close.
 // when close, the server will return the info of SSTs generated by the server.
 type WriteClient interface {
-	// Write KV to storage layer.
-	Write(ctx context.Context, in *WriteRequest) error
-	// CloseAndRecv the client, get the response and close the client.
-	// the response should contain the SST info.
-	// we name it this way to mimic GRPC stream, and to avoid confusion with normal
-	// Close method which should always be called after creation, but in here,
-	// it's not a must.
-	CloseAndRecv(ctx context.Context) (*WriteResponse, error)
+	WriteChunk(*WriteRequest) error
path: pkg/ingestor/ingestcli/interface.go
user: login: D3Hunter
body: there is plan to unify first-gen tidb api into the same interface, in that case, there's no concept of `chunk`
created_at: 2025-04-21T10:25:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,187 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/pingcap/errors"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: how to set it?
created_at: 2025-04-21T10:27:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,187 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/pingcap/errors"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(tikvWorkerURL string, clusterID uint64, taskID int64, httpClient *http.Client) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		taskID:        taskID,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: there is no need to pass host:port part inside uri, the `httpClient` already have it
created_at: 2025-04-21T11:07:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,187 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/pingcap/errors"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(tikvWorkerURL string, clusterID uint64, taskID int64, httpClient *http.Client) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		taskID:        taskID,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) WriteChunk(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(w.writer, binary.BigEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := w.writer.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(w.writer, binary.BigEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := w.writer.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	return nil
+}
+
+func (w *writeClient) Close() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	if err := w.reader.Close(); err != nil {
+		return nil, err
+	}
+	return &WriteResponse{SSTFile: w.sstFilePath}, nil
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+
+	currentTaskID int64
+}
+
+// NewClient creates a new Client instance.
+func NewClient(tikvWorkerURL string, clusterID uint64, httpClient *http.Client) Client {
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: one of `httpClient` and `tikvWorkerURL` is enough
created_at: 2025-04-21T11:07:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,187 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/pingcap/errors"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(tikvWorkerURL string, clusterID uint64, taskID int64, httpClient *http.Client) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		taskID:        taskID,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) WriteChunk(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(w.writer, binary.BigEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := w.writer.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(w.writer, binary.BigEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := w.writer.Write(pair.Value); err != nil {
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: if we write this way, will each pair became a http chunk?
created_at: 2025-04-21T11:08:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -338,7 +338,7 @@ func (w *cloudRegionJobWorker) write(ctx context.Context, job *regionJob) (*tikv
 		iter.ReleaseBuf()
 	}
 
-	resp, err := writeCli.CloseAndRecv(ctx)
+	resp, err := writeCli.Close()
path: pkg/lightning/backend/local/job_worker.go
user: login: D3Hunter
body: we must close on error of Write now
created_at: 2025-04-21T11:20:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: please move `It is caller's responsibility to call Close() when it meets an error.` to the interface
created_at: 2025-04-22T06:29:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: why add it, when will it be used
created_at: 2025-04-22T06:30:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: use NewErrorGroupWithRecover
created_at: 2025-04-22T06:32:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) Write(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
+	var buf bytes.Buffer
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.BigEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.BigEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	if _, err := w.writer.Write(buf.Bytes()); err != nil {
+		return errors.Trace(err)
+	}
+	return nil
+}
+
+func (w *writeClient) Recv() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	if err := w.reader.Close(); err != nil {
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: ```
// If the provided body is also an [io.Closer], the returned
// [Request.Body] is set to body and will be closed (possibly
// asynchronously) by the Client methods Do, Post, and PostForm,
// and [Transport.RoundTrip].
```
created_at: 2025-04-22T06:40:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) Write(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
+	var buf bytes.Buffer
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.BigEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.BigEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	if _, err := w.writer.Write(buf.Bytes()); err != nil {
+		return errors.Trace(err)
+	}
+	return nil
+}
+
+func (w *writeClient) Recv() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	if err := w.reader.Close(); err != nil {
+		return nil, err
+	}
+	return &WriteResponse{SSTFile: w.sstFilePath}, nil
+}
+
+func (w *writeClient) Close() {
+	w.writer.Close()
+	w.eg.Wait()
+	w.reader.Close()
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+}
+
+// NewClient creates a new Client instance.
+func NewClient(tikvWorkerURL string, clusterID uint64, httpClient *http.Client, commitTS uint64) Client {
+	return &client{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		httpClient:    httpClient,
+		commitTS:      commitTS,
+	}
+}
+
+func (c *client) WriteClient(ctx context.Context) (WriteClient, error) {
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: add to the interface that the returned WriteClient will share this context
created_at: 2025-04-22T06:42:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) Write(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: if `Do` close the reader, there is no need to check, the Write will fail anyway
created_at: 2025-04-22T06:51:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) Write(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
+	var buf bytes.Buffer
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.BigEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.BigEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	if _, err := w.writer.Write(buf.Bytes()); err != nil {
+		return errors.Trace(err)
+	}
+	return nil
+}
+
+func (w *writeClient) Recv() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	if err := w.reader.Close(); err != nil {
+		return nil, err
+	}
+	return &WriteResponse{SSTFile: w.sstFilePath}, nil
+}
+
+func (w *writeClient) Close() {
+	w.writer.Close()
+	w.eg.Wait()
+	w.reader.Close()
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+}
+
+// NewClient creates a new Client instance.
+func NewClient(tikvWorkerURL string, clusterID uint64, httpClient *http.Client, commitTS uint64) Client {
+	return &client{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		httpClient:    httpClient,
+		commitTS:      commitTS,
+	}
+}
+
+func (c *client) WriteClient(ctx context.Context) (WriteClient, error) {
+	cli := newWriteClient(c.tikvWorkerURL, c.clusterID, c.httpClient, c.commitTS)
+	err := cli.init(ctx)
+	return cli, err
+}
+
+func (c *client) Ingest(ctx context.Context, in *IngestRequest) error {
+	ri := in.Region.Region
+	url := fmt.Sprintf("%s/ingest_s3?cluster_id=%d&file_name=%s&region_id=%d&epoch_version=%d",
+		c.tikvWorkerURL, c.clusterID, in.SSTFile, ri.Id, ri.RegionEpoch.Version)
+
+	req, err := http.NewRequestWithContext(ctx, "POST", url, nil)
+	if err != nil {
+		return errors.Trace(err)
+	}
+
+	resp, err := c.httpClient.Do(req)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		body, _ := io.ReadAll(resp.Body)
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: we need to handle the error to avoid returning the wrong `failed to unmarshal` err
created_at: 2025-04-22T06:56:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) Write(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
+	var buf bytes.Buffer
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.BigEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.BigEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	if _, err := w.writer.Write(buf.Bytes()); err != nil {
+		return errors.Trace(err)
+	}
+	return nil
+}
+
+func (w *writeClient) Recv() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	if err := w.reader.Close(); err != nil {
+		return nil, err
+	}
+	return &WriteResponse{SSTFile: w.sstFilePath}, nil
+}
+
+func (w *writeClient) Close() {
+	w.writer.Close()
+	w.eg.Wait()
+	w.reader.Close()
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+}
+
+// NewClient creates a new Client instance.
+func NewClient(tikvWorkerURL string, clusterID uint64, httpClient *http.Client, commitTS uint64) Client {
+	return &client{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		httpClient:    httpClient,
+		commitTS:      commitTS,
+	}
+}
+
+func (c *client) WriteClient(ctx context.Context) (WriteClient, error) {
+	cli := newWriteClient(c.tikvWorkerURL, c.clusterID, c.httpClient, c.commitTS)
+	err := cli.init(ctx)
+	return cli, err
+}
+
+func (c *client) Ingest(ctx context.Context, in *IngestRequest) error {
+	ri := in.Region.Region
+	url := fmt.Sprintf("%s/ingest_s3?cluster_id=%d&file_name=%s&region_id=%d&epoch_version=%d",
+		c.tikvWorkerURL, c.clusterID, in.SSTFile, ri.Id, ri.RegionEpoch.Version)
+
+	req, err := http.NewRequestWithContext(ctx, "POST", url, nil)
+	if err != nil {
+		return errors.Trace(err)
+	}
+
+	resp, err := c.httpClient.Do(req)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		body, _ := io.ReadAll(resp.Body)
+		var pbErr errorpb.Error
+		if err := proto.Unmarshal(body, &pbErr); err != nil {
+			return fmt.Errorf("failed to unmarshal error(%s): %s", string(body), err)
+		}
+		return &PBError{Err: &pbErr}
+	}
+	return nil
+}
+
+// PBError is a implementation of error.
+type PBError struct {
+	Err *errorpb.Error
+}
+
+// Error implements the error.
+func (re *PBError) Error() string {
+	return re.Err.String()
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: ```suggestion
	return re.Err.GetMessage()
```
created_at: 2025-04-22T06:57:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,61 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"context"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/pingcap/kvproto/pkg/import_sstpb"
+	"github.com/stretchr/testify/require"
+)
+
+func TestWriteClientWriteChunk(t *testing.T) {
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, err := io.ReadAll(r.Body)
+		require.NoError(t, err)
+		expected := []byte{
+			0x00, 0x03, 'k', 'e', 'y',
+			0x00, 0x00, 0x00, 0x05, 'v', 'a', 'l', 'u', 'e',
+		}
path: pkg/ingestor/ingestcli/client_test.go
user: login: D3Hunter
body: ```suggestion
		expected := []byte("\x00\x03key\x00\x00\x00\x05value")
```
created_at: 2025-04-22T06:58:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,61 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"context"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/pingcap/kvproto/pkg/import_sstpb"
+	"github.com/stretchr/testify/require"
+)
+
+func TestWriteClientWriteChunk(t *testing.T) {
path: pkg/ingestor/ingestcli/client_test.go
user: login: D3Hunter
body: please add cases for ingest and negative cases
created_at: 2025-04-22T07:00:49Z
author_association: CONTRIBUTOR
path: pkg/ingestor/ingestcli/util.go
user: login: D3Hunter
body: seems useless
created_at: 2025-04-22T07:02:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) Write(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
path: pkg/ingestor/ingestcli/client.go
user: login: tangenta
body: Will the origin error be passed correctly?
created_at: 2025-04-22T08:55:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,236 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util/intest"
+	"golang.org/x/sync/errgroup"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          errgroup.Group
+	inited      bool
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient. It is caller's responsibility to call Close() when it meets an error.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	intest.Assert(!w.inited)
+	defer func() {
+		w.inited = true
+	}()
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.errCh = make(chan error, 1)
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		defer close(w.errCh)
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		result := new(sstFileResult)
+		err = json.Unmarshal(data, result)
+		if err != nil {
+			w.errCh <- err
+			return errors.Trace(err)
+		}
+		w.sstFilePath = result.SSTFile
+		return nil
+	})
+}
+
+func (w *writeClient) checkErrBeforeWrite() error {
+	select {
+	case err := <-w.errCh:
+		return err
+	default:
+		return nil
+	}
+}
+
+func (w *writeClient) Write(req *WriteRequest) error {
+	err := w.checkErrBeforeWrite()
+	if err != nil {
+		return err
+	}
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: i guess the error would be `closed pipe`

even with current `checkErrBeforeWrite`, it still might hide the real error, maybe check it when `Write` return error
created_at: 2025-04-22T09:01:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,227 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          *util.ErrorGroupWithRecover
+	errCh       chan error
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: no need use channel, use `atomic.Error`
created_at: 2025-04-22T09:02:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,232 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/pkg/util"
+)
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg          *util.ErrorGroupWithRecover
+	errCh       chan error
+	writer      *io.PipeWriter
+	reader      *io.PipeReader
+	sstFilePath string
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, pr)
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: ```suggestion
	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
```
created_at: 2025-04-23T04:43:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1446,9 +1449,9 @@ func (local *Backend) newRegionJobWorker(
 		regenerateJobsFn: local.generateJobForRange,
 	}
 	if kerneltype.IsNextGen() {
+		httpClient := &http.Client{}
path: pkg/lightning/backend/local/local.go
user: login: D3Hunter
body: please add todo that we need to support TLS later, ok for now
created_at: 2025-04-23T11:15:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,249 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	Id         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: useless now
created_at: 2025-04-25T02:29:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,249 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	Id         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg         *util.ErrorGroupWithRecover
+	sendReqErr atomic.Error
+	writer     *io.PipeWriter
+	reader     *io.PipeReader
+	sstMeta    *nextGenSSTMeta
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
+	w.writer = pw
+	return nil
+}
+
+type sstFileResult struct {
+	SSTFile string `json:"sst_file"`
+}
+
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: ```suggestion
```
created_at: 2025-04-25T02:30:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,249 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	Id         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	taskID        int64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg         *util.ErrorGroupWithRecover
+	sendReqErr atomic.Error
+	writer     *io.PipeWriter
+	reader     *io.PipeReader
+	sstMeta    *nextGenSSTMeta
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr
path: pkg/ingestor/ingestcli/client.go
user: login: D3Hunter
body: add some comment that `w.reader` will be closed by the `httpClient.Do` automatically, it's easy to forget the reason why we don't do it and think it's a bug
created_at: 2025-04-25T02:33:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1446,9 +1449,10 @@ func (local *Backend) newRegionJobWorker(
 		regenerateJobsFn: local.generateJobForRange,
 	}
 	if kerneltype.IsNextGen() {
+		// TODO: add support for TLS.
+		httpClient := &http.Client{}
 		cloudW := &objStoreRegionJobWorker{
-			// TODO fill the cli
-			ingestCli:      nil,
+			ingestCli:      ingestcli.NewClient(local.TiKVWorkerURL, local.pdCli.GetClusterID(ctx), httpClient, local.splitCli),
path: pkg/lightning/backend/local/local.go
user: login: D3Hunter
body: we can do `local.pdCli.GetClusterID(ctx)` inside `doImport`, and pass to this method
created_at: 2025-04-25T02:40:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1434,6 +1437,8 @@ func (local *Backend) doImport(
 }
 
 func (local *Backend) newRegionJobWorker(
+	ctx context.Context,
path: pkg/lightning/backend/local/local.go
user: login: D3Hunter
body: seems useless
created_at: 2025-04-25T07:24:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,246 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	ID         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg         *util.ErrorGroupWithRecover
+	sendReqErr atomic.Error
+	writer     *io.PipeWriter
+	reader     *io.PipeReader
+	sstMeta    *nextGenSSTMeta
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr // PipeReader will be closed by the httpClient.Do automatically
+	w.writer = pw
+	return nil
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, err1 := io.ReadAll(resp.Body)
+			if err1 != nil {
+				w.sendReqErr.Store(err)
+				return fmt.Errorf("failed to readAll response: %s", err1.Error())
+			}
+			w.sendReqErr.Store(err)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		res := &nextGenResp{}
+		if err := json.Unmarshal(data, res); err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		w.sstMeta = &res.SstMeta
+		return nil
+	})
+}
+
+func (w *writeClient) cause(err error) error {
+	if reqErr := w.sendReqErr.Load(); reqErr != nil {
+		return errors.Trace(reqErr)
+	}
+	return errors.Trace(err)
+}
+
+func (w *writeClient) Write(req *WriteRequest) (err error) {
+	var buf bytes.Buffer
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.LittleEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.LittleEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	if _, err := w.writer.Write(buf.Bytes()); err != nil {
+		return w.cause(err)
+	}
+	return nil
+}
+
+func (w *writeClient) Recv() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	return &WriteResponse{nextGenSSTMeta: w.sstMeta}, nil
+}
+
+func (w *writeClient) Close() {
+	//nolint: errcheck
+	_ = w.writer.Close()
+	//nolint: errcheck
+	_ = w.eg.Wait()
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	splitCli      split.SplitClient
+}
+
+// NewClient creates a new Client instance.
+func NewClient(tikvWorkerURL string, clusterID uint64, httpClient *http.Client, splitCli split.SplitClient) Client {
+	return &client{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		httpClient:    httpClient,
+		splitCli:      splitCli,
+	}
+}
+
+func (c *client) WriteClient(ctx context.Context, commitTS uint64) (WriteClient, error) {
+	cli := newWriteClient(c.tikvWorkerURL, c.clusterID, c.httpClient, commitTS)
+	err := cli.init(ctx)
+	return cli, err
+}
+
+func (c *client) Ingest(ctx context.Context, in *IngestRequest) error {
+	ri := in.Region.Region
+	store, err := c.splitCli.GetStore(ctx, in.Region.Leader.GetStoreId())
+	if err != nil {
+		return errors.Trace(err)
+	}
+	url := fmt.Sprintf("http://%s/ingest_s3?cluster_id=%d&region_id=%d&epoch_version=%d",
+		store.GetStatusAddress(), c.clusterID, ri.Id, ri.RegionEpoch.Version)
+
+	data, err := json.Marshal(&in.WriteResp.nextGenSSTMeta)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	bodyRd := bytes.NewReader(data)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, bodyRd)
+	if err != nil {
+		return errors.Trace(err)
+	}
+
+	resp, err := c.httpClient.Do(req)
path: pkg/ingestor/ingestcli/client.go
user: login: GMHDBJD
body: Do we need set Content-Type to "application/json" here?
created_at: 2025-04-27T05:57:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,246 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	ID         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg         *util.ErrorGroupWithRecover
+	sendReqErr atomic.Error
+	writer     *io.PipeWriter
+	reader     *io.PipeReader
+	sstMeta    *nextGenSSTMeta
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr // PipeReader will be closed by the httpClient.Do automatically
+	w.writer = pw
+	return nil
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, err1 := io.ReadAll(resp.Body)
+			if err1 != nil {
+				w.sendReqErr.Store(err)
path: pkg/ingestor/ingestcli/client.go
user: login: GMHDBJD
body: 
```suggestion
				w.sendReqErr.Store(err1)
```
created_at: 2025-04-27T05:59:08Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,246 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	ID         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg         *util.ErrorGroupWithRecover
+	sendReqErr atomic.Error
+	writer     *io.PipeWriter
+	reader     *io.PipeReader
+	sstMeta    *nextGenSSTMeta
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr // PipeReader will be closed by the httpClient.Do automatically
+	w.writer = pw
+	return nil
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, err1 := io.ReadAll(resp.Body)
+			if err1 != nil {
+				w.sendReqErr.Store(err)
+				return fmt.Errorf("failed to readAll response: %s", err1.Error())
+			}
+			w.sendReqErr.Store(err)
path: pkg/ingestor/ingestcli/client.go
user: login: GMHDBJD
body: Why we store a nil error here?
created_at: 2025-04-27T06:00:05Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,246 @@
+// Copyright 2025 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ingestcli
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+
+	"github.com/gogo/protobuf/proto"
+	"github.com/pingcap/errors"
+	"github.com/pingcap/kvproto/pkg/errorpb"
+	"github.com/pingcap/tidb/br/pkg/restore/split"
+	"github.com/pingcap/tidb/pkg/util"
+	"go.uber.org/atomic"
+)
+
+type nextGenResp struct {
+	SstMeta nextGenSSTMeta `json:"sst_meta"`
+}
+
+type nextGenSSTMeta struct {
+	ID         int64 `json:"id"`
+	Smallest   []int `json:"smallest"`
+	Biggest    []int `json:"biggest"`
+	MetaOffset int   `json:"meta-offset"`
+	CommitTs   int   `json:"commit-ts"`
+}
+
+var _ WriteClient = &writeClient{}
+
+type writeClient struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	commitTS      uint64
+
+	eg         *util.ErrorGroupWithRecover
+	sendReqErr atomic.Error
+	writer     *io.PipeWriter
+	reader     *io.PipeReader
+	sstMeta    *nextGenSSTMeta
+}
+
+// newWriteClient creates a writeClient.
+func newWriteClient(
+	tikvWorkerURL string,
+	clusterID uint64,
+	httpClient *http.Client,
+	commitTS uint64,
+) *writeClient {
+	return &writeClient{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		commitTS:      commitTS,
+		httpClient:    httpClient,
+		eg:            util.NewErrorGroupWithRecover(),
+	}
+}
+
+func (w *writeClient) init(ctx context.Context) error {
+	pr, pw := io.Pipe()
+	url := fmt.Sprintf("%s/write_sst?cluster_id=%d&commit_ts=%d",
+		w.tikvWorkerURL, w.clusterID, w.commitTS)
+	req, err := http.NewRequestWithContext(ctx, "PUT", url, pr)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	req.Header.Set("Content-Type", "application/octet-stream")
+	w.startChunkedHTTPRequest(req)
+	w.reader = pr // PipeReader will be closed by the httpClient.Do automatically
+	w.writer = pw
+	return nil
+}
+
+func (w *writeClient) startChunkedHTTPRequest(req *http.Request) {
+	w.eg.Go(func() error {
+		resp, err := w.httpClient.Do(req)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, err1 := io.ReadAll(resp.Body)
+			if err1 != nil {
+				w.sendReqErr.Store(err)
+				return fmt.Errorf("failed to readAll response: %s", err1.Error())
+			}
+			w.sendReqErr.Store(err)
+			return fmt.Errorf("failed to send chunked request: %s", string(body))
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		res := &nextGenResp{}
+		if err := json.Unmarshal(data, res); err != nil {
+			w.sendReqErr.Store(err)
+			return errors.Trace(err)
+		}
+		w.sstMeta = &res.SstMeta
+		return nil
+	})
+}
+
+func (w *writeClient) cause(err error) error {
+	if reqErr := w.sendReqErr.Load(); reqErr != nil {
+		return errors.Trace(reqErr)
+	}
+	return errors.Trace(err)
+}
+
+func (w *writeClient) Write(req *WriteRequest) (err error) {
+	var buf bytes.Buffer
+	for _, pair := range req.Pairs {
+		keyLen := uint16(len(pair.Key))
+		if err := binary.Write(&buf, binary.LittleEndian, keyLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Key); err != nil {
+			return errors.Trace(err)
+		}
+		valLen := uint32(len(pair.Value))
+		if err := binary.Write(&buf, binary.LittleEndian, valLen); err != nil {
+			return errors.Trace(err)
+		}
+		if _, err := buf.Write(pair.Value); err != nil {
+			return errors.Trace(err)
+		}
+	}
+	if _, err := w.writer.Write(buf.Bytes()); err != nil {
+		return w.cause(err)
+	}
+	return nil
+}
+
+func (w *writeClient) Recv() (*WriteResponse, error) {
+	if err := w.writer.Close(); err != nil {
+		return nil, err
+	}
+	err := w.eg.Wait()
+	if err != nil {
+		return nil, err
+	}
+	return &WriteResponse{nextGenSSTMeta: w.sstMeta}, nil
+}
+
+func (w *writeClient) Close() {
+	//nolint: errcheck
+	_ = w.writer.Close()
+	//nolint: errcheck
+	_ = w.eg.Wait()
+}
+
+var _ Client = &client{}
+
+type client struct {
+	tikvWorkerURL string
+	clusterID     uint64
+	httpClient    *http.Client
+	splitCli      split.SplitClient
+}
+
+// NewClient creates a new Client instance.
+func NewClient(tikvWorkerURL string, clusterID uint64, httpClient *http.Client, splitCli split.SplitClient) Client {
+	return &client{
+		tikvWorkerURL: tikvWorkerURL,
+		clusterID:     clusterID,
+		httpClient:    httpClient,
+		splitCli:      splitCli,
+	}
+}
+
+func (c *client) WriteClient(ctx context.Context, commitTS uint64) (WriteClient, error) {
+	cli := newWriteClient(c.tikvWorkerURL, c.clusterID, c.httpClient, commitTS)
+	err := cli.init(ctx)
+	return cli, err
+}
+
+func (c *client) Ingest(ctx context.Context, in *IngestRequest) error {
+	ri := in.Region.Region
+	store, err := c.splitCli.GetStore(ctx, in.Region.Leader.GetStoreId())
+	if err != nil {
+		return errors.Trace(err)
+	}
+	url := fmt.Sprintf("http://%s/ingest_s3?cluster_id=%d&region_id=%d&epoch_version=%d",
+		store.GetStatusAddress(), c.clusterID, ri.Id, ri.RegionEpoch.Version)
+
+	data, err := json.Marshal(&in.WriteResp.nextGenSSTMeta)
+	if err != nil {
+		return errors.Trace(err)
+	}
+	bodyRd := bytes.NewReader(data)
+	req, err := http.NewRequestWithContext(ctx, "POST", url, bodyRd)
+	if err != nil {
+		return errors.Trace(err)
+	}
+
+	resp, err := c.httpClient.Do(req)
path: pkg/ingestor/ingestcli/client.go
user: login: tangenta
body: No, I think tikv-workers don't support other content-type.
created_at: 2025-04-27T06:05:29Z
author_association: CONTRIBUTOR

PR comments:
user: login: tiprow[bot]
created_at: 2025-04-15T06:23:13Z
body: Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with `/ok-to-test` in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using `/test all`.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=pingcap%2Ftidb).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

author_association: NONE
user: login: codecov[bot]
created_at: 2025-04-15T07:06:13Z
body: ## [Codecov](https://app.codecov.io/gh/pingcap/tidb/pull/60569?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) Report
Attention: Patch coverage is `61.14286%` with `68 lines` in your changes missing coverage. Please review.
> Please [upload](https://docs.codecov.com/docs/codecov-uploader) report for BASE (`feature/next-gen-tidb@8757bd9`). [Learn more](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap#section-missing-base-commit) about missing BASE report.

<details><summary>Additional details and impacted files</summary>


```diff
@@                    Coverage Diff                     @@
##             feature/next-gen-tidb     #60569   +/-   ##
==========================================================
  Coverage                         ?   73.7817%           
==========================================================
  Files                            ?       1744           
  Lines                            ?     481253           
  Branches                         ?          0           
==========================================================
  Hits                             ?     355077           
  Misses                           ?     104182           
  Partials                         ?      21994           
```

| [Flag](https://app.codecov.io/gh/pingcap/tidb/pull/60569/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/pingcap/tidb/pull/60569/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `45.5538% <3.1847%> (?)` | |
| [unit](https://app.codecov.io/gh/pingcap/tidb/pull/60569/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `72.3683% <61.1428%> (?)` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/pingcap/tidb/pull/60569/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | |
|---|---|---|
| [dumpling](https://app.codecov.io/gh/pingcap/tidb/pull/60569/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `52.6910% <0.0000%> (?)` | |
| [parser](https://app.codecov.io/gh/pingcap/tidb/pull/60569/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `∅ <0.0000%> (?)` | |
| [br](https://app.codecov.io/gh/pingcap/tidb/pull/60569/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `47.2451% <0.0000%> (?)` | |

</details>
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: wuhuizuo
created_at: 2025-04-17T03:21:31Z
body: /override idc-jenkins-ci-tidb/check_dev
author_association: CONTRIBUTOR
user: login: ti-chi-bot[bot]
created_at: 2025-04-17T03:21:40Z
body: @wuhuizuo: wuhuizuo unauthorized: /override is restricted to Repo administrators, and the following github teams:pingcap/tidb: tidb-maintainers Benjamin2037.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60569#issuecomment-2811630113):

>/override idc-jenkins-ci-tidb/check_dev


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: wuhuizuo
created_at: 2025-04-17T03:22:59Z
body: > /override idc-jenkins-ci-tidb/check_dev

just for testing for new function deployed on ti-chi-bot app.
author_association: CONTRIBUTOR
user: login: D3Hunter
created_at: 2025-04-17T03:43:50Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-17T03:44:12Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60569#issuecomment-2811659295):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-27T03:38:41Z
body: have panic
author_association: CONTRIBUTOR
user: login: D3Hunter
created_at: 2025-04-27T04:58:56Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-27T04:59:21Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60569#issuecomment-2833134807):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-04-27T06:20:10Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/pingcap/tidb/pull/60569#pullrequestreview-2793239614" title="Approved">D3Hunter</a>*, *<a href="https://github.com/pingcap/tidb/pull/60569#pullrequestreview-2797371472" title="Approved">GMHDBJD</a>*

The full list of commands accepted by this bot can be found [here](https://prow.tidb.net/command-help?repo=pingcap%2Ftidb).

The pull request process is described [here](https://book.prow.tidb.net/#/workflows/pr)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/pingcap/tidb/blob/feature/next-gen-tidb/OWNERS)~~ [D3Hunter,GMHDBJD]
- ~~[pkg/lightning/OWNERS](https://github.com/pingcap/tidb/blob/feature/next-gen-tidb/pkg/lightning/OWNERS)~~ [D3Hunter,GMHDBJD]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE
user: login: ti-chi-bot[bot]
created_at: 2025-04-27T06:20:12Z
body: [LGTM Timeline notifier]
---
Timeline:

- `2025-04-25 07:31:48.043608915 +0000 UTC m=+600051.855399295`: :ballot_box_with_check: agreed by [D3Hunter](https://github.com/D3Hunter).
- `2025-04-27 06:20:12.164781573 +0000 UTC m=+768555.976571954`: :ballot_box_with_check: agreed by [GMHDBJD](https://github.com/GMHDBJD).
author_association: NONE
user: login: D3Hunter
created_at: 2025-04-27T06:53:41Z
body: /retest
author_association: CONTRIBUTOR
user: login: tiprow[bot]
created_at: 2025-04-27T06:54:05Z
body: @D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/pingcap/tidb/pull/60569#issuecomment-2833229250):

>/retest


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: NONE