Repository: tikv/client-go (go)
Issue: replica-read request with mixed strategy and with label, should be able to retry all remain replicas
# Issue

### 1. Minimal reproduce step

In the following case, suppose the region leader is in `store1`.

```go
	ca = replicaSelectorAccessPathCase{
		reqType:   tikvrpc.CmdGet,
		readType:  kv.ReplicaReadMixed,
		staleRead: false,
		timeout:   time.Second,
		label:     &metapb.StoreLabel{Key: "id", Value: "2"},
		accessErr: []RegionErrorType{DeadLineExceededErr, ServerIsBusyErr},
```

### 2. What did you expect to see?

```go
		expect: &accessPathResult{
			accessPath: []string{
				"{addr: store2, replica-read: true, stale-read: false}",
				"{addr: store1, replica-read: true, stale-read: false}",
				"{addr: store3, replica-read: true, stale-read: false}"},
			respErr:         "",
			respRegionError: nil,
```


### 3. What did you see instead?

```go
		expect: &accessPathResult{
			accessPath: []string{
				"{addr: store2, replica-read: true, stale-read: false}",
				"{addr: store1, replica-read: true, stale-read: false}"}, // retry leader
			respErr:         "",
			respRegionError: fakeEpochNotMatch,  // return fake EpochNotMatchErr since no available replica.
```

Currently, the replica selector logic is:

1. try store2 by replica-read, since store2 has match label, but got `DeadLineExceededErr`.
2. retry leader in store1, but got `ServerIsBusyErr`.
3. return fake EpochNotMatchErr since no available replica.

The problem is, store3 is available, and we can use replica-read to retry store3, since store3 is never been tried.

Pull request description: replica-read request with mixed strategy and with label, should be able to retry all remain replicas

close #1176

Patch:
diff --git a/internal/locate/region_request.go b/internal/locate/region_request.go
index 87d310b..277899d 100644
--- a/internal/locate/region_request.go
+++ b/internal/locate/region_request.go
@@ -775,7 +775,8 @@ func (state *accessFollower) next(bo *retry.Backoffer, selector *replicaSelector
 			// If the leader is also unavailable, we can fallback to the follower and use replica-read flag again,
 			// The remote follower not tried yet, and the local follower can retry without stale-read flag.
 			// If leader tried and received deadline exceeded error, try follower.
-			if state.isStaleRead || leader.deadlineErrUsingConfTimeout {
+			// If labels are used, some followers would be filtered by the labels and can't be candidates, they still need to be retried.
+			if state.isStaleRead || leader.deadlineErrUsingConfTimeout || len(state.option.labels) > 0 {
 				selector.state = &tryFollower{
 					leaderIdx: state.leaderIdx,
 					lastIdx:   state.leaderIdx,


Test patch:
diff --git a/internal/locate/replica_selector_test.go b/internal/locate/replica_selector_test.go
index 46bc370..0df8a09 100644
--- a/internal/locate/replica_selector_test.go
+++ b/internal/locate/replica_selector_test.go
@@ -94,7 +94,7 @@ type accessPathResult struct {
 	regionIsValid   bool
 }
 
-func TestReplicaReadStaleReadAccessPathByCase(t *testing.T) {
+func TestReplicaReadAccessPathByCase(t *testing.T) {
 	s := new(testReplicaSelectorSuite)
 	s.SetupTest(t)
 	defer s.TearDownTest()
@@ -348,6 +348,27 @@ func TestReplicaReadStaleReadAccessPathByCase(t *testing.T) {
 	}
 	s.True(s.runCaseAndCompare(ca))
 	s.changeRegionLeader(1)
+
+	ca = replicaSelectorAccessPathCase{
+		reqType:   tikvrpc.CmdGet,
+		readType:  kv.ReplicaReadMixed,
+		staleRead: false,
+		timeout:   time.Second,
+		label:     &metapb.StoreLabel{Key: "id", Value: "2"},
+		accessErr: []RegionErrorType{DeadLineExceededErr, ServerIsBusyErr},
+		expect: &accessPathResult{
+			accessPath: []string{
+				"{addr: store2, replica-read: true, stale-read: false}",
+				"{addr: store1, replica-read: true, stale-read: false}",
+				"{addr: store3, replica-read: true, stale-read: false}"},
+			respErr:         "",
+			respRegionError: nil,
+			backoffCnt:      1,
+			backoffDetail:   []string{"tikvServerBusy+1"},
+			regionIsValid:   true,
+		},
+	}
+	s.True(s.runCaseAndCompare(ca))
 }
 
 func (s *testReplicaSelectorSuite) changeRegionLeader(storeId uint64) {