Repository: tikv/pd
Pull request #4218 (2021-10-19 05:57:11)

core: allow cancel load region (#4175)
This is an automated cherry-pick of #4175

### What problem does this PR solve?

Fix https://github.com/tikv/pd/issues/3936

### What is changed and how it works?

- add cancel support for core.LoadRegion functions
- replace `chan bool` with `contextes` in region syncer client
- add test case

### Check List

<!-- Remove the items that are not applicable. -->

Tests

<!-- At least one of them must be included. -->

- Unit test

### Release note

```release-note
Fix the issue that PD may not elect leader as soon as leader step down
```


ti-chi-bot: @ti-chi-bot: This cherry pick PR is for a release branch and has not yet been approved by release team.
Adding the `do-not-merge/cherry-pick-not-approved` label.

To merge this cherry pick, it must first be approved by the collaborators.

**AFTER** it has been approved by collaborators, please ping the release team in a comment to request a cherry pick review.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
nolouch: the statics check failed in test. 
ti-chi-bot: 
[FORMAT CHECKER NOTIFICATION]

Please provide the associated issue number in the commit messages, for example: `close #12345`, or `ref #12345`.


<!--Checker Notification Identifier-->

ti-chi-bot: 
[FORMAT CHECKER NOTIFICATION]

Please provide the associated issue number in the commit messages, for example: `close #12345`, or `ref #12345`.


<!--Checker Notification Identifier-->

disksing: /merge
ti-chi-bot: @disksing: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests


<details>

Instructions for interacting with me using PR comments are available [here](https://prow.tidb.io/command-help).  If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
</details>
ti-chi-bot: This pull request has been accepted and is ready to merge. <details>Commit hash: b1ff18b280fc398616c13b8add7210a27f3ba6e3</details>
disksing: /run-all-tests


ti-chi-bot: 
[FORMAT CHECKER NOTIFICATION]

Please provide the associated issue number in the commit messages, for example: `close #12345`, or `ref #12345`.


<!--Checker Notification Identifier-->

disksing: /run-all-tests
disksing: /run-all-tests
disksing: /run-all-tests
codecov[bot]: # [Codecov](https://codecov.io/gh/tikv/pd/pull/4218?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv) Report
> Merging [#4218](https://codecov.io/gh/tikv/pd/pull/4218?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv) (2538a29) into [release-5.0](https://codecov.io/gh/tikv/pd/commit/e482eed0af40c3500f116e5a8f3858f979d9f2c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv) (e482eed) will **increase** coverage by `0.20%`.
> The diff coverage is `89.65%`.

[![Impacted file tree graph](https://codecov.io/gh/tikv/pd/pull/4218/graphs/tree.svg?width=650&height=150&src=pr&token=J1yRL471OI&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv)](https://codecov.io/gh/tikv/pd/pull/4218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv)

```diff
@@               Coverage Diff               @@
##           release-5.0    #4218      +/-   ##
===============================================
+ Coverage        74.78%   74.98%   +0.20%     
===============================================
  Files              245      245              
  Lines            24370    24362       -8     
===============================================
+ Hits             18225    18268      +43     
+ Misses            4520     4489      -31     
+ Partials          1625     1605      -20     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `74.98% <89.65%> (+0.20%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/tikv/pd/pull/4218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv) | Coverage Δ | |
|---|---|---|
| [server/cluster/cluster.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL2NsdXN0ZXIvY2x1c3Rlci5nbw==) | `82.92% <0.00%> (+0.90%)` | :arrow_up: |
| [server/region\_syncer/server.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL3JlZ2lvbl9zeW5jZXIvc2VydmVyLmdv) | `88.88% <ø> (+5.80%)` | :arrow_up: |
| [server/core/storage.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL2NvcmUvc3RvcmFnZS5nbw==) | `69.20% <75.00%> (+1.52%)` | :arrow_up: |
| [server/region\_syncer/client.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL3JlZ2lvbl9zeW5jZXIvY2xpZW50Lmdv) | `79.68% <94.11%> (-2.01%)` | :arrow_down: |
| [server/core/region\_storage.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL2NvcmUvcmVnaW9uX3N0b3JhZ2UuZ28=) | `77.65% <100.00%> (+1.79%)` | :arrow_up: |
| [server/schedulers/shuffle\_hot\_region.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL3NjaGVkdWxlcnMvc2h1ZmZsZV9ob3RfcmVnaW9uLmdv) | `55.10% <0.00%> (-10.21%)` | :arrow_down: |
| [server/election/leadership.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL2VsZWN0aW9uL2xlYWRlcnNoaXAuZ28=) | `83.52% <0.00%> (-3.53%)` | :arrow_down: |
| [server/schedulers/random\_merge.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL3NjaGVkdWxlcnMvcmFuZG9tX21lcmdlLmdv) | `60.00% <0.00%> (-3.34%)` | :arrow_down: |
| [server/grpc\_service.go](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv#diff-c2VydmVyL2dycGNfc2VydmljZS5nbw==) | `46.94% <0.00%> (-0.23%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/tikv/pd/pull/4218/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/tikv/pd/pull/4218?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/tikv/pd/pull/4218?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv). Last update [e482eed...2538a29](https://codecov.io/gh/tikv/pd/pull/4218?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tikv).


Patch:
diff --git a/server/cluster/cluster.go b/server/cluster/cluster.go
index 52840d1b9..ebdcab696 100644
--- a/server/cluster/cluster.go
+++ b/server/cluster/cluster.go
@@ -290,7 +290,7 @@ func (c *RaftCluster) LoadClusterInfo() (*RaftCluster, error) {
 	start = time.Now()
 
 	// used to load region from kv storage to cache storage.
-	if err := c.storage.LoadRegionsOnce(c.core.CheckAndPutRegion); err != nil {
+	if err := c.storage.LoadRegionsOnce(c.ctx, c.core.CheckAndPutRegion); err != nil {
 		return nil, err
 	}
 	log.Info("load regions",
diff --git a/server/core/region_storage.go b/server/core/region_storage.go
index d5ee5d546..8566e92d2 100644
--- a/server/core/region_storage.go
+++ b/server/core/region_storage.go
@@ -19,6 +19,7 @@ import (
 	"sync"
 	"time"
 
+	"github.com/pingcap/failpoint"
 	"github.com/pingcap/kvproto/pkg/metapb"
 	"github.com/pingcap/log"
 	"github.com/tikv/pd/pkg/encryption"
@@ -131,6 +132,7 @@ func deleteRegion(kv kv.Base, region *metapb.Region) error {
 }
 
 func loadRegions(
+	ctx context.Context,
 	kv kv.Base,
 	encryptionKeyManager *encryptionkm.KeyManager,
 	f func(region *RegionInfo) []*RegionInfo,
@@ -143,6 +145,10 @@ func loadRegions(
 	// a variable rangeLimit to work around.
 	rangeLimit := maxKVRangeLimit
 	for {
+		failpoint.Inject("slowLoadRegion", func() {
+			rangeLimit = 1
+			time.Sleep(time.Second)
+		})
 		startKey := regionPath(nextID)
 		_, res, err := kv.LoadRange(startKey, endKey, rangeLimit)
 		if err != nil {
@@ -151,6 +157,11 @@ func loadRegions(
 			}
 			return err
 		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		default:
+		}
 
 		for _, s := range res {
 			region := &metapb.Region{}
diff --git a/server/core/storage.go b/server/core/storage.go
index 1fc5f6fc2..d65c6610a 100644
--- a/server/core/storage.go
+++ b/server/core/storage.go
@@ -14,6 +14,7 @@
 package core
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
 	"math"
@@ -193,22 +194,22 @@ func (s *Storage) LoadRegion(regionID uint64, region *metapb.Region) (ok bool, e
 }
 
 // LoadRegions loads all regions from storage to RegionsInfo.
-func (s *Storage) LoadRegions(f func(region *RegionInfo) []*RegionInfo) error {
+func (s *Storage) LoadRegions(ctx context.Context, f func(region *RegionInfo) []*RegionInfo) error {
 	if atomic.LoadInt32(&s.useRegionStorage) > 0 {
-		return loadRegions(s.regionStorage, s.encryptionKeyManager, f)
+		return loadRegions(ctx, s.regionStorage, s.encryptionKeyManager, f)
 	}
-	return loadRegions(s.Base, s.encryptionKeyManager, f)
+	return loadRegions(ctx, s.Base, s.encryptionKeyManager, f)
 }
 
 // LoadRegionsOnce loads all regions from storage to RegionsInfo.Only load one time from regionStorage.
-func (s *Storage) LoadRegionsOnce(f func(region *RegionInfo) []*RegionInfo) error {
+func (s *Storage) LoadRegionsOnce(ctx context.Context, f func(region *RegionInfo) []*RegionInfo) error {
 	if atomic.LoadInt32(&s.useRegionStorage) == 0 {
-		return loadRegions(s.Base, s.encryptionKeyManager, f)
+		return loadRegions(ctx, s.Base, s.encryptionKeyManager, f)
 	}
 	s.mu.Lock()
 	defer s.mu.Unlock()
 	if s.regionLoaded == 0 {
-		if err := loadRegions(s.regionStorage, s.encryptionKeyManager, f); err != nil {
+		if err := loadRegions(ctx, s.regionStorage, s.encryptionKeyManager, f); err != nil {
 			return err
 		}
 		s.regionLoaded = 1
diff --git a/server/region_syncer/client.go b/server/region_syncer/client.go
index 2f425cf8f..58f516fd6 100644
--- a/server/region_syncer/client.go
+++ b/server/region_syncer/client.go
@@ -40,10 +40,6 @@ const (
 // StopSyncWithLeader stop to sync the region with leader.
 func (s *RegionSyncer) StopSyncWithLeader() {
 	s.reset()
-	s.mu.Lock()
-	close(s.mu.closed)
-	s.mu.closed = make(chan struct{})
-	s.mu.Unlock()
 	s.wg.Wait()
 }
 
@@ -51,19 +47,15 @@ func (s *RegionSyncer) reset() {
 	s.mu.Lock()
 	defer s.mu.Unlock()
 
-	if s.mu.regionSyncerCancel == nil {
-		return
+	if s.mu.clientCancel != nil {
+		s.mu.clientCancel()
 	}
-	s.mu.regionSyncerCancel()
-	s.mu.regionSyncerCancel, s.mu.regionSyncerCtx = nil, nil
+	s.mu.clientCancel, s.mu.clientCtx = nil, nil
 }
 
-func (s *RegionSyncer) establish(addr string) (*grpc.ClientConn, error) {
-	s.reset()
-	ctx, cancel := context.WithCancel(s.server.LoopContext())
+func (s *RegionSyncer) establish(ctx context.Context, addr string) (*grpc.ClientConn, error) {
 	tlsCfg, err := s.tlsConfig.ToTLSConfig()
 	if err != nil {
-		cancel()
 		return nil, err
 	}
 	cc, err := grpcutil.GetClientConn(
@@ -88,28 +80,16 @@ func (s *RegionSyncer) establish(addr string) (*grpc.ClientConn, error) {
 		grpc.WithBlock(),
 	)
 	if err != nil {
-		cancel()
 		return nil, errors.WithStack(err)
 	}
-
-	s.mu.Lock()
-	s.mu.regionSyncerCtx, s.mu.regionSyncerCancel = ctx, cancel
-	s.mu.Unlock()
 	return cc, nil
 }
 
-func (s *RegionSyncer) syncRegion(conn *grpc.ClientConn) (ClientStream, error) {
+func (s *RegionSyncer) syncRegion(ctx context.Context, conn *grpc.ClientConn) (ClientStream, error) {
 	cli := pdpb.NewPDClient(conn)
-	var ctx context.Context
-	s.mu.RLock()
-	ctx = s.mu.regionSyncerCtx
-	s.mu.RUnlock()
-	if ctx == nil {
-		return nil, errors.New("syncRegion failed due to regionSyncerCtx is nil")
-	}
 	syncStream, err := cli.SyncRegions(ctx)
 	if err != nil {
-		return syncStream, errs.ErrGRPCCreateStream.Wrap(err).FastGenWithCause()
+		return nil, errs.ErrGRPCCreateStream.Wrap(err).FastGenWithCause()
 	}
 	err = syncStream.Send(&pdpb.SyncRegionRequest{
 		Header:     &pdpb.RequestHeader{ClusterId: s.server.ClusterID()},
@@ -117,7 +97,7 @@ func (s *RegionSyncer) syncRegion(conn *grpc.ClientConn) (ClientStream, error) {
 		StartIndex: s.history.GetNextIndex(),
 	})
 	if err != nil {
-		return syncStream, errs.ErrGRPCSend.Wrap(err).FastGenWithCause()
+		return nil, errs.ErrGRPCSend.Wrap(err).FastGenWithCause()
 	}
 
 	return syncStream, nil
@@ -128,15 +108,21 @@ var regionGuide = core.GenerateRegionGuideFunc(false)
 // StartSyncWithLeader starts to sync with leader.
 func (s *RegionSyncer) StartSyncWithLeader(addr string) {
 	s.wg.Add(1)
-	s.mu.RLock()
-	closed := s.mu.closed
-	s.mu.RUnlock()
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.mu.clientCtx, s.mu.clientCancel = context.WithCancel(s.server.LoopContext())
+	ctx := s.mu.clientCtx
+
 	go func() {
 		defer s.wg.Done()
 		// used to load region from kv storage to cache storage.
 		bc := s.server.GetBasicCluster()
 		storage := s.server.GetStorage()
-		err := storage.LoadRegionsOnce(bc.CheckAndPutRegion)
+		log.Info("region syncer start load region")
+		start := time.Now()
+		err := storage.LoadRegionsOnce(ctx, bc.CheckAndPutRegion)
+		log.Info("region syncer finished load region", zap.Duration("time-cost", time.Since(start)))
 		if err != nil {
 			log.Warn("failed to load regions.", errs.ZapError(err))
 		}
@@ -144,11 +130,11 @@ func (s *RegionSyncer) StartSyncWithLeader(addr string) {
 		var conn *grpc.ClientConn
 		for {
 			select {
-			case <-closed:
+			case <-ctx.Done():
 				return
 			default:
 			}
-			conn, err = s.establish(addr)
+			conn, err = s.establish(ctx, addr)
 			if err != nil {
 				log.Error("cannot establish connection with leader", zap.String("server", s.server.Name()), zap.String("leader", s.server.GetLeader().GetName()), errs.ZapError(err))
 				continue
@@ -160,12 +146,12 @@ func (s *RegionSyncer) StartSyncWithLeader(addr string) {
 		// Start syncing data.
 		for {
 			select {
-			case <-closed:
+			case <-ctx.Done():
 				return
 			default:
 			}
 
-			stream, err := s.syncRegion(conn)
+			stream, err := s.syncRegion(ctx, conn)
 			if err != nil {
 				if ev, ok := status.FromError(err); ok {
 					if ev.Code() == codes.Canceled {
diff --git a/server/region_syncer/server.go b/server/region_syncer/server.go
index d0ab8f7f2..6569b8863 100644
--- a/server/region_syncer/server.go
+++ b/server/region_syncer/server.go
@@ -69,10 +69,9 @@ type Server interface {
 type RegionSyncer struct {
 	mu struct {
 		sync.RWMutex
-		streams            map[string]ServerStream
-		regionSyncerCtx    context.Context
-		regionSyncerCancel context.CancelFunc
-		closed             chan struct{}
+		streams      map[string]ServerStream
+		clientCtx    context.Context
+		clientCancel context.CancelFunc
 	}
 	server    Server
 	wg        sync.WaitGroup
@@ -94,7 +93,6 @@ func NewRegionSyncer(s Server) *RegionSyncer {
 		tlsConfig: s.GetTLSConfig(),
 	}
 	syncer.mu.streams = make(map[string]ServerStream)
-	syncer.mu.closed = make(chan struct{})
 	return syncer
 }
 


Test patch:
diff --git a/server/core/storage_test.go b/server/core/storage_test.go
index d513da0d0..bc6855c42 100644
--- a/server/core/storage_test.go
+++ b/server/core/storage_test.go
@@ -14,6 +14,7 @@
 package core
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
 	"math"
@@ -143,7 +144,7 @@ func (s *testKVSuite) TestLoadRegions(c *C) {
 
 	n := 10
 	regions := mustSaveRegions(c, storage, n)
-	c.Assert(storage.LoadRegions(cache.SetRegion), IsNil)
+	c.Assert(storage.LoadRegions(context.Background(), cache.SetRegion), IsNil)
 
 	c.Assert(cache.GetRegionCount(), Equals, n)
 	for _, region := range cache.GetMetaRegions() {
@@ -157,7 +158,7 @@ func (s *testKVSuite) TestLoadRegionsToCache(c *C) {
 
 	n := 10
 	regions := mustSaveRegions(c, storage, n)
-	c.Assert(storage.LoadRegionsOnce(cache.SetRegion), IsNil)
+	c.Assert(storage.LoadRegionsOnce(context.Background(), cache.SetRegion), IsNil)
 
 	c.Assert(cache.GetRegionCount(), Equals, n)
 	for _, region := range cache.GetMetaRegions() {
@@ -166,7 +167,7 @@ func (s *testKVSuite) TestLoadRegionsToCache(c *C) {
 
 	n = 20
 	mustSaveRegions(c, storage, n)
-	c.Assert(storage.LoadRegionsOnce(cache.SetRegion), IsNil)
+	c.Assert(storage.LoadRegionsOnce(context.Background(), cache.SetRegion), IsNil)
 	c.Assert(cache.GetRegionCount(), Equals, n)
 }
 
@@ -176,7 +177,7 @@ func (s *testKVSuite) TestLoadRegionsExceedRangeLimit(c *C) {
 
 	n := 1000
 	regions := mustSaveRegions(c, storage, n)
-	c.Assert(storage.LoadRegions(cache.SetRegion), IsNil)
+	c.Assert(storage.LoadRegions(context.Background(), cache.SetRegion), IsNil)
 	c.Assert(cache.GetRegionCount(), Equals, n)
 	for _, region := range cache.GetMetaRegions() {
 		c.Assert(region, DeepEquals, regions[region.GetId()])
diff --git a/server/region_syncer/client_test.go b/server/region_syncer/client_test.go
new file mode 100644
index 000000000..1084aa45b
--- /dev/null
+++ b/server/region_syncer/client_test.go
@@ -0,0 +1,105 @@
+// Copyright 2021 TiKV Project Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package syncer
+
+import (
+	"context"
+	"io/ioutil"
+	"os"
+	"time"
+
+	. "github.com/pingcap/check"
+	"github.com/pingcap/failpoint"
+	"github.com/pingcap/kvproto/pkg/metapb"
+	"github.com/pingcap/kvproto/pkg/pdpb"
+	"github.com/tikv/pd/pkg/grpcutil"
+	"github.com/tikv/pd/server/core"
+	"github.com/tikv/pd/server/kv"
+)
+
+var _ = Suite(&testClientSuite{})
+
+type testClientSuite struct{}
+
+// For issue https://github.com/tikv/pd/issues/3936
+func (t *testClientSuite) TestLoadRegion(c *C) {
+	tempDir, err := ioutil.TempDir(os.TempDir(), "region_syncer_load_region")
+	c.Assert(err, IsNil)
+	defer os.RemoveAll(tempDir)
+	rs, err := core.NewRegionStorage(context.Background(), tempDir, nil)
+	c.Assert(err, IsNil)
+
+	server := &mockServer{
+		ctx:     context.Background(),
+		storage: core.NewStorage(kv.NewMemoryKV(), core.WithRegionStorage(rs)),
+		bc:      core.NewBasicCluster(),
+	}
+	for i := 0; i < 30; i++ {
+		rs.SaveRegion(&metapb.Region{Id: uint64(i) + 1})
+	}
+	c.Assert(failpoint.Enable("github.com/tikv/pd/server/core/slowLoadRegion", "return(true)"), IsNil)
+	defer func() { c.Assert(failpoint.Disable("github.com/tikv/pd/server/core/slowLoadRegion"), IsNil) }()
+
+	rc := NewRegionSyncer(server)
+	start := time.Now()
+	rc.StartSyncWithLeader("")
+	time.Sleep(time.Second)
+	rc.StopSyncWithLeader()
+	c.Assert(time.Since(start), Greater, time.Second) // make sure failpoint is injected
+	c.Assert(time.Since(start), Less, time.Second*2)
+}
+
+type mockServer struct {
+	ctx            context.Context
+	member, leader *pdpb.Member
+	storage        *core.Storage
+	bc             *core.BasicCluster
+}
+
+func (s *mockServer) LoopContext() context.Context {
+	return s.ctx
+}
+
+func (s *mockServer) ClusterID() uint64 {
+	return 1
+}
+
+func (s *mockServer) GetMemberInfo() *pdpb.Member {
+	return s.member
+}
+
+func (s *mockServer) GetLeader() *pdpb.Member {
+	return s.leader
+}
+
+func (s *mockServer) GetStorage() *core.Storage {
+	return s.storage
+}
+
+func (s *mockServer) Name() string {
+	return "mock-server"
+}
+
+func (s *mockServer) GetRegions() []*core.RegionInfo {
+	return s.bc.GetRegions()
+}
+
+func (s *mockServer) GetTLSConfig() *grpcutil.TLSConfig {
+	return &grpcutil.TLSConfig{}
+}
+
+func (s *mockServer) GetBasicCluster() *core.BasicCluster {
+	return s.bc
+}
diff --git a/tests/server/cluster/cluster_test.go b/tests/server/cluster/cluster_test.go
index c3c15c280..ce0ddcfdc 100644
--- a/tests/server/cluster/cluster_test.go
+++ b/tests/server/cluster/cluster_test.go
@@ -718,7 +718,7 @@ func (s *clusterTestSuite) TestLoadClusterInfo(c *C) {
 	for _, region := range regions {
 		c.Assert(storage.SaveRegion(region), IsNil)
 	}
-	raftCluster.GetStorage().LoadRegionsOnce(raftCluster.GetCacheCluster().PutRegion)
+	raftCluster.GetStorage().LoadRegionsOnce(s.ctx, raftCluster.GetCacheCluster().PutRegion)
 	c.Assert(raftCluster.GetRegionCount(), Equals, n)
 }