Repository: pingcap/tispark (language: python)

Issue:
Retry when store is down in high layer
### What problem does this PR solve? <!--add issue link with summary if exists-->

https://github.com/pingcap/tispark/issues/2705

### What is changed and how it works?

1. when the store is unreachable or null, invalid cache and retry. 

This enhancement will fix this situation: voter down and TiSpark fail 

2. catch the exception thrown by splitrange and retry

This enhancement will fix this situation: pd leader down and the region get from pd will be null then an exception will be thrown. According to the implementation of client-java, the switch leader needs to get members from pd, when it requests the down pd, it takes  `spark.tispark.grpc.timeout_in_sec(180s in default)*2` to retry, which may block the leader switch for `spark.tispark.grpc.timeout_in_sec(180s in default)*2` s

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

 - Unit test
 - Integration test
 - Manual test (add detailed scripts or steps below)
 - No code

Code changes

 - Has exported function/method change
 - Has exported variable/fields change
 - Has interface methods change
 - Has persistent data change

Side effects

 - Possible performance regression
 - Increased code complexity
 - Breaking backward compatibility

Related changes

 - Need to cherry-pick to the release branch
 - Need to update the documentation
 - Need to update the `tidb-ansible` repository
 - Need to be included in the release note



Discussion / hints:

Fix patch:
diff --git a/tikv-client/src/main/java/com/pingcap/tikv/operation/iterator/DAGIterator.java b/tikv-client/src/main/java/com/pingcap/tikv/operation/iterator/DAGIterator.java
index 844cf34f9..adad6aebe 100644
--- a/tikv-client/src/main/java/com/pingcap/tikv/operation/iterator/DAGIterator.java
+++ b/tikv-client/src/main/java/com/pingcap/tikv/operation/iterator/DAGIterator.java
@@ -43,6 +43,7 @@
 import org.tikv.common.region.TiRegion;
 import org.tikv.common.region.TiStore;
 import org.tikv.common.region.TiStoreType;
+import org.tikv.common.util.BackOffFunction;
 import org.tikv.common.util.BackOffer;
 import org.tikv.common.util.ConcreteBackOffer;
 import org.tikv.common.util.RangeSplitter;
@@ -206,8 +207,9 @@ private SelectResponse process(RegionTask regionTask) {
     Queue<SelectResponse> responseQueue = new ArrayDeque<>();
     remainTasks.add(regionTask);
     BackOffer backOffer = ConcreteBackOffer.newCopNextMaxBackOff();
-
     HashSet<Long> resolvedLocks = new HashSet<>();
+    BackOffer storeUnreachableBackOffer = ConcreteBackOffer.newCustomBackOff(60 * 1000);
+
     // In case of one region task spilt into several others, we ues a queue to properly handle all
     // the remaining tasks.
     while (!remainTasks.isEmpty()) {
@@ -220,6 +222,31 @@ private SelectResponse process(RegionTask regionTask) {
       TiStore store = task.getStore();
 
       try {
+        if (store == null || !store.isReachable()) {
+          storeUnreachableBackOffer.doBackOffWithMaxSleep(
+              BackOffFunction.BackOffFuncType.BoServerBusy,
+              2000,
+              new TiClientInternalException("retry timeout: store is null or unreachable"));
+          if (store == null) {
+            logger.warn("TiKV store is null, invalid cache and retry");
+          } else {
+            logger.warn(
+                "TiKV store " + store.getAddress() + " is unreachable, invalid cache and retry");
+          }
+          clientSession.getTiKVSession().getRegionManager().invalidateRegion(region);
+          try {
+            remainTasks.addAll(
+                RangeSplitter.newSplitter(clientSession.getTiKVSession().getRegionManager())
+                    .splitRangeByRegion(ranges, storeType));
+          } catch (Exception e) {
+            // If the pd is switching leader, the region invalid exception will be thrown. In this
+            // case, we can retry with the original task.
+            logger.warn("split range by region error, retry with the original task", e);
+            remainTasks.add(task);
+          }
+          continue;
+        }
+
         RegionStoreClient client =
             clientSession
                 .getTiKVSession()


Test patch:
diff --git a/.github/workflows/alter-primary-key-false-test.yml b/.github/workflows/alter-primary-key-false-test.yml
index 286aad274..8eda62314 100644
--- a/.github/workflows/alter-primary-key-false-test.yml
+++ b/.github/workflows/alter-primary-key-false-test.yml
@@ -1,12 +1,12 @@
 name: alter-primary-key-false-test
 
 on:
-  push:
-    branches:
-      - master
   pull_request:
     branches:
-      - master
+      - release-3.1
+      - release-3.0
+      - release-2.5
+      - release-2.4
 
 jobs:
   test: