Repository: tikv/tikv (language: rust)

Issue:
[Proposal] Ingest SST without pausing writes
Update: Since the compaction-filter performs RocksDB writes in background threads, which may run concurrently with SST ingestion during apply snapshot, so we introduce a range latch to ensure mutual exclusion between the compaction-filter and apply snapshot ingestion. For more details, refer to this PR:  https://github.com/tikv/tikv/pull/18096.

## Summary
This proposal introduces a special option to RocksDB that allows writes to continue during SST file ingestion. TiKV can safely enable this option because there are no concurrent threads writing data that overlap with the data being ingested.

## Motivation
Currently, RocksDB pauses writes during SST file ingestion, which increases TiKV’s apply wait duration. This behavior should be optimized to reduce foreground duration.

## Design Details
We propose adding an allow_write option to RocksDB’s IngestExternalFileOptions. When set to true, RocksDB will no longer pause writes to the database during SST ingestion.

TiKV can safely enable the allow_write option when ingesting SST files during apply-snapshot or destroy-region(delete by ingest) because:
- During apply-snapshot or destroy-region, no foreground writes overlap with the region being ingested.
- The single-threaded region-worker ensures sequential execution of apply-snapshot and destroy-region tasks. Even if a region is migrated and quickly returns, apply-snapshot and destroy-region cannot occur concurrently.

## Drawbacks

## Alternatives
An alternative approach is to modify RocksDB to support ingestion without pausing writes by restructuring the sequence number assignment process and temporarily pausing the memtable flushing instead of writes:
1. Assign and publish sequence numbers to the SST before ingestion.
2. Flush memtables if there are overlapping keys.
3. Pause further memtable flushes.
4. Perform the SST ingestion.
5. Resume memtable flushes.

While this approach ensures safe sequence number assignment, it introduces two major drawbacks:
- **Immutable snapshot is broken**: After assigning and publish sequence numbers to ingested SST files, a new snapshot is created. Initially, this snapshot might not see the ingested SST files because the ingestion has not started. Once the ingestion completes, the snapshot can see the files, breaking snapshot immutability.
- **Atomic writes are broken**: RocksDB SST file ingestion is not atomic, especially when involving multiple column families. If sequence numbers are published before the ingestion completes, snapshots may observe a partial state where some SST data is visible while others are not, violating atomicity guarantees.

These issues may not impact TiKV in practice due to its usage patterns. However, the complexity and potential risks outweigh the benefits.


Discussion / hints:
/assign hhwyt

Fix patch:
diff --git a/Cargo.lock b/Cargo.lock
index 061a2e2559f..06fc67351c7 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2143,6 +2143,7 @@ dependencies = [
  "engine_traits",
  "kvproto",
  "raft",
+ "tikv_util",
  "tracker",
  "txn_types",
 ]
@@ -2230,6 +2231,7 @@ dependencies = [
  "log_wrappers",
  "protobuf",
  "raft",
+ "rand 0.8.5",
  "serde",
  "slog",
  "slog-global",
@@ -3814,7 +3816,7 @@ dependencies = [
 [[package]]
 name = "librocksdb_sys"
 version = "0.1.0"
-source = "git+https://github.com/tikv/rust-rocksdb.git#19eeae6dc7734af12475fbcf9d368168a0314085"
+source = "git+https://github.com/tikv/rust-rocksdb.git#c55bb14ec41439c2c1849358a1c57bb6c0fbdbf7"
 dependencies = [
  "bindgen 0.65.1",
  "bzip2-sys",
@@ -3833,7 +3835,7 @@ dependencies = [
 [[package]]
 name = "libtitan_sys"
 version = "0.0.1"
-source = "git+https://github.com/tikv/rust-rocksdb.git#19eeae6dc7734af12475fbcf9d368168a0314085"
+source = "git+https://github.com/tikv/rust-rocksdb.git#c55bb14ec41439c2c1849358a1c57bb6c0fbdbf7"
 dependencies = [
  "bzip2-sys",
  "cc",
@@ -5755,9 +5757,9 @@ checksum = "18eb52b6664d331053136fcac7e4883bdc6f5fc04a6aab3b0f75eafb80ab88b3"
 
 [[package]]
 name = "rgb"
-version = "0.8.32"
+version = "0.8.50"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e74fdc210d8f24a7dbfedc13b04ba5764f5232754ccebfdf5fff1bad791ccbc6"
+checksum = "57397d16646700483b67d2dd6511d79318f9d057fdbd21a4066aeac8b41d310a"
 dependencies = [
  "bytemuck",
 ]
@@ -5791,7 +5793,7 @@ dependencies = [
 [[package]]
 name = "rocksdb"
 version = "0.3.0"
-source = "git+https://github.com/tikv/rust-rocksdb.git#19eeae6dc7734af12475fbcf9d368168a0314085"
+source = "git+https://github.com/tikv/rust-rocksdb.git#c55bb14ec41439c2c1849358a1c57bb6c0fbdbf7"
 dependencies = [
  "libc 0.2.151",
  "librocksdb_sys",
diff --git a/components/engine_panic/Cargo.toml b/components/engine_panic/Cargo.toml
index c125ef7ccf2..46b42392357 100644
--- a/components/engine_panic/Cargo.toml
+++ b/components/engine_panic/Cargo.toml
@@ -14,5 +14,6 @@ encryption = { workspace = true }
 engine_traits = { workspace = true }
 kvproto = { workspace = true }
 raft = { workspace = true }
+tikv_util = { workspace = true }
 tracker = { workspace = true }
 txn_types = { workspace = true }
diff --git a/components/engine_panic/src/import.rs b/components/engine_panic/src/import.rs
index a30eeb3e005..a34ce7b2e03 100644
--- a/components/engine_panic/src/import.rs
+++ b/components/engine_panic/src/import.rs
@@ -1,15 +1,23 @@
 // Copyright 2019 TiKV Project Authors. Licensed under Apache-2.0.
 
-use std::path::Path;
-
-use engine_traits::{ImportExt, IngestExternalFileOptions, Result};
+use engine_traits::{ImportExt, IngestExternalFileOptions, Range, Result};
+use tikv_util::range_latch::RangeLatchGuard;
 
 use crate::engine::PanicEngine;
 
 impl ImportExt for PanicEngine {
     type IngestExternalFileOptions = PanicIngestExternalFileOptions;
 
-    fn ingest_external_file_cf(&self, cf: &str, files: &[&str]) -> Result<()> {
+    fn ingest_external_file_cf(
+        &self,
+        cf: &str,
+        files: &[&str],
+        range: Option<Range<'_>>,
+    ) -> Result<()> {
+        panic!()
+    }
+
+    fn acquire_ingest_latch(&self, range: Range<'_>) -> RangeLatchGuard<'_> {
         panic!()
     }
 }
@@ -24,4 +32,8 @@ impl IngestExternalFileOptions for PanicIngestExternalFileOptions {
     fn move_files(&mut self, f: bool) {
         panic!()
     }
+
+    fn allow_write(&mut self, f: bool) {
+        panic!()
+    }
 }
diff --git a/components/engine_rocks/src/engine.rs b/components/engine_rocks/src/engine.rs
index e393f58721c..4cd51a7fcad 100644
--- a/components/engine_rocks/src/engine.rs
+++ b/components/engine_rocks/src/engine.rs
@@ -4,6 +4,7 @@ use std::{any::Any, sync::Arc};
 
 use engine_traits::{IterOptions, Iterable, KvEngine, Peekable, ReadOptions, Result, SyncMutable};
 use rocksdb::{DBIterator, Writable, DB};
+use tikv_util::range_latch::RangeLatch;
 
 use crate::{
     db_vector::RocksDbVector, options::RocksReadOptions, r2e, util::get_cf_handle,
@@ -148,6 +149,9 @@ pub struct RocksEngine {
     support_multi_batch_write: bool,
     #[cfg(feature = "trace-lifetime")]
     _id: trace::TabletTraceId,
+    // Used to ensure mutual exclusivity between compaction filter writes and the SST ingestion
+    // operation.
+    pub ingest_latch: Arc<RangeLatch>,
 }
 
 impl RocksEngine {
@@ -158,6 +162,7 @@ impl RocksEngine {
             #[cfg(feature = "trace-lifetime")]
             _id: trace::TabletTraceId::new(db.path(), &db),
             db,
+            ingest_latch: Arc::new(RangeLatch::new()),
         }
     }
 
diff --git a/components/engine_rocks/src/import.rs b/components/engine_rocks/src/import.rs
index 930772d8c09..14947149c1b 100644
--- a/components/engine_rocks/src/import.rs
+++ b/components/engine_rocks/src/import.rs
@@ -1,20 +1,52 @@
 // Copyright 2019 TiKV Project Authors. Licensed under Apache-2.0.
 
-use engine_traits::{ImportExt, IngestExternalFileOptions, Result};
+use engine_traits::{ImportExt, IngestExternalFileOptions, Range, Result};
+use fail::fail_point;
 use rocksdb::IngestExternalFileOptions as RawIngestExternalFileOptions;
-use tikv_util::time::Instant;
+use tikv_util::{range_latch::RangeLatchGuard, time::Instant};
 
 use crate::{
-    engine::RocksEngine, perf_context_metrics::INGEST_EXTERNAL_FILE_TIME_HISTOGRAM, r2e, util,
+    engine::RocksEngine,
+    perf_context_metrics::{
+        INGEST_EXTERNAL_FILE_ALLOW_WRITE_COUNTER, INGEST_EXTERNAL_FILE_TIME_HISTOGRAM,
+    },
+    r2e, util,
 };
 
 impl ImportExt for RocksEngine {
     type IngestExternalFileOptions = RocksIngestExternalFileOptions;
 
-    fn ingest_external_file_cf(&self, cf_name: &str, files: &[&str]) -> Result<()> {
+    fn ingest_external_file_cf(
+        &self,
+        cf_name: &str,
+        files: &[&str],
+        range: Option<Range<'_>>,
+    ) -> Result<()> {
+        // Acquire latch to prevent concurrency with compaction-filter operations
+        // when using RocksDB IngestExternalFileOptions.allow_write = true.
+        let _region_inject_latch_guard = range.as_ref().map(|r| {
+            self.ingest_latch
+                .acquire(r.start_key.to_vec(), r.end_key.to_vec())
+        });
+        fail_point!("after_apply_snapshot_ingest_latch_acquired");
+
         let cf = util::get_cf_handle(self.as_inner(), cf_name)?;
         let mut opts = RocksIngestExternalFileOptions::new();
         opts.move_files(true);
+        let allow_write = range.is_some();
+        opts.allow_write(allow_write);
+        if allow_write {
+            INGEST_EXTERNAL_FILE_ALLOW_WRITE_COUNTER
+                .with_label_values(&["
+            allow_write"])
+                .inc();
+        } else {
+            INGEST_EXTERNAL_FILE_ALLOW_WRITE_COUNTER
+                .with_label_values(&["
+            not_allow_write"])
+                .inc();
+        }
+
         // Note: no need reset the global seqno to 0 for compatibility as #16992
         // enable the TiKV to handle the case on applying abnormal snapshot.
         let now = Instant::now_coarse();
@@ -41,6 +73,11 @@ impl ImportExt for RocksEngine {
         }
         Ok(())
     }
+
+    fn acquire_ingest_latch(&self, range: Range<'_>) -> RangeLatchGuard<'_> {
+        self.ingest_latch
+            .acquire(range.start_key.to_vec(), range.end_key.to_vec())
+    }
 }
 
 pub struct RocksIngestExternalFileOptions(RawIngestExternalFileOptions);
@@ -53,6 +90,10 @@ impl IngestExternalFileOptions for RocksIngestExternalFileOptions {
     fn move_files(&mut self, f: bool) {
         self.0.move_files(f);
     }
+
+    fn allow_write(&mut self, f: bool) {
+        self.0.set_allow_write(f);
+    }
 }
 
 #[cfg(test)]
@@ -126,7 +167,11 @@ mod tests {
             sst2.put(v.as_bytes(), v.as_bytes()).unwrap();
         }
         sst2.finish().unwrap();
-        db.ingest_external_file_cf(CF_DEFAULT, &[p1.to_str().unwrap(), p2.to_str().unwrap()])
-            .unwrap();
+        db.ingest_external_file_cf(
+            CF_DEFAULT,
+            &[p1.to_str().unwrap(), p2.to_str().unwrap()],
+            None,
+        )
+        .unwrap();
     }
 }
diff --git a/components/engine_rocks/src/misc.rs b/components/engine_rocks/src/misc.rs
index b50002b61a5..9cd1f023577 100644
--- a/components/engine_rocks/src/misc.rs
+++ b/components/engine_rocks/src/misc.rs
@@ -5,6 +5,7 @@ use engine_traits::{
     Range, RangeStats, Result, SstWriter, SstWriterBuilder, WriteBatch, WriteBatchExt,
     WriteOptions,
 };
+use fail::fail_point;
 use rocksdb::{FlushOptions, Range as RocksRange};
 use tikv_util::{box_try, keybuilder::KeyBuilder};
 
@@ -32,6 +33,7 @@ impl RocksEngine {
         cf: &str,
         sst_path: String,
         ranges: &[Range<'_>],
+        allow_write_during_ingestion: bool,
     ) -> Result<bool> {
         let mut written = false;
         let mut ranges = ranges.to_owned();
@@ -40,13 +42,13 @@ impl RocksEngine {
         let mut writer_wrapper: Option<RocksSstWriter> = None;
         let mut data: Vec<Vec<u8>> = vec![];
         let mut last_end_key: Option<Vec<u8>> = None;
-        for r in ranges {
+        for r in &ranges {
             // There may be a range overlap with next range
             if last_end_key
                 .as_ref()
                 .map_or(false, |key| key.as_slice() > r.start_key)
             {
-                written |= self.delete_all_in_range_cf_by_key(wopts, cf, &r)?;
+                written |= self.delete_all_in_range_cf_by_key(wopts, cf, r)?;
                 continue;
             }
             last_end_key = Some(r.end_key.to_owned());
@@ -72,7 +74,11 @@ impl RocksEngine {
                 } else {
                     data.push(it.key().to_vec());
                 }
-                if data.len() > MAX_DELETE_COUNT_BY_KEY {
+                let max_delete_count_by_key = (|| {
+                    fail_point!("manually_set_max_delete_count_by_key", |_| { 0 });
+                    MAX_DELETE_COUNT_BY_KEY
+                })();
+                if data.len() > max_delete_count_by_key {
                     let builder = RocksSstWriterBuilder::new().set_db(self).set_cf(cf);
                     let mut writer = builder.build(sst_path.as_str())?;
                     for key in data.iter() {
@@ -87,7 +93,24 @@ impl RocksEngine {
 
         if let Some(writer) = writer_wrapper {
             writer.finish()?;
-            self.ingest_external_file_cf(cf, &[sst_path.as_str()])?;
+            let (min_start_key, max_end_key) = ranges.iter().fold(
+                (ranges[0].start_key, ranges[0].end_key),
+                |(min_start, max_end), range| {
+                    (
+                        std::cmp::min(min_start, range.start_key),
+                        std::cmp::max(max_end, range.end_key),
+                    )
+                },
+            );
+            let range_to_lock = if allow_write_during_ingestion {
+                Some(Range {
+                    start_key: min_start_key,
+                    end_key: max_end_key,
+                })
+            } else {
+                None
+            };
+            self.ingest_external_file_cf(cf, &[sst_path.as_str()], range_to_lock)?;
         } else {
             let mut wb = self.write_batch();
             for key in data.iter() {
@@ -266,8 +289,17 @@ impl MiscExt for RocksEngine {
                     written |= self.delete_all_in_range_cf_by_key(wopts, cf, r)?;
                 }
             }
-            DeleteStrategy::DeleteByWriter { sst_path } => {
-                written |= self.delete_all_in_range_cf_by_ingest(wopts, cf, sst_path, ranges)?;
+            DeleteStrategy::DeleteByWriter {
+                sst_path,
+                allow_write_during_ingestion,
+            } => {
+                written |= self.delete_all_in_range_cf_by_ingest(
+                    wopts,
+                    cf,
+                    sst_path,
+                    ranges,
+                    allow_write_during_ingestion,
+                )?;
             }
         }
         Ok(written)
@@ -615,7 +647,10 @@ mod tests {
             data.push(i.to_string().as_bytes().to_vec());
         }
         test_delete_ranges(
-            DeleteStrategy::DeleteByWriter { sst_path },
+            DeleteStrategy::DeleteByWriter {
+                sst_path,
+                allow_write_during_ingestion: false,
+            },
             &data,
             &[
                 Range::new(&data[2], &data[499]),
diff --git a/components/engine_rocks/src/perf_context_metrics.rs b/components/engine_rocks/src/perf_context_metrics.rs
index cd55c8e8e04..54febd34fcd 100644
--- a/components/engine_rocks/src/perf_context_metrics.rs
+++ b/components/engine_rocks/src/perf_context_metrics.rs
@@ -92,4 +92,11 @@ lazy_static! {
             exponential_buckets(0.005, 2.0, 20).unwrap()
         )
         .unwrap();
+    pub static ref INGEST_EXTERNAL_FILE_ALLOW_WRITE_COUNTER: IntCounterVec =
+        register_int_counter_vec!(
+            "tikv_storage_ingest_external_file_allow_write_counter",
+            "Total number of RocksDB ingest SST operations for both allow_write enabled and disabled",
+            &["type"]
+        )
+        .unwrap();
 }
diff --git a/components/engine_traits/Cargo.toml b/components/engine_traits/Cargo.toml
index 85b8337ef0d..9b4b9404d56 100644
--- a/components/engine_traits/Cargo.toml
+++ b/components/engine_traits/Cargo.toml
@@ -29,4 +29,5 @@ tracker = { workspace = true }
 txn_types = { workspace = true }
 
 [dev-dependencies]
+rand = "0.8.5"
 toml = "0.5"
diff --git a/components/engine_traits/src/import.rs b/components/engine_traits/src/import.rs
index 06a913932b0..1a6ef4f9334 100644
--- a/components/engine_traits/src/import.rs
+++ b/components/engine_traits/src/import.rs
@@ -1,15 +1,31 @@
 // Copyright 2019 TiKV Project Authors. Licensed under Apache-2.0.
 
-use crate::errors::Result;
+use tikv_util::range_latch::RangeLatchGuard;
+
+use crate::{errors::Result, Range};
 
 pub trait ImportExt {
     type IngestExternalFileOptions: IngestExternalFileOptions;
 
-    fn ingest_external_file_cf(&self, cf: &str, files: &[&str]) -> Result<()>;
+    /// Ingests external files into the specified column family.
+    ///
+    /// If the range is specified, it enables `RocksDB
+    /// IngestExternalFileOptions.allow_write` and locks the
+    /// specified range.  
+    fn ingest_external_file_cf(
+        &self,
+        cf: &str,
+        files: &[&str],
+        range: Option<Range<'_>>,
+    ) -> Result<()>;
+
+    fn acquire_ingest_latch(&self, range: Range<'_>) -> RangeLatchGuard<'_>;
 }
 
 pub trait IngestExternalFileOptions {
     fn new() -> Self;
 
     fn move_files(&mut self, f: bool);
+
+    fn allow_write(&mut self, f: bool);
 }
diff --git a/components/engine_traits/src/misc.rs b/components/engine_traits/src/misc.rs
index 5e1b1347751..f8318948fc1 100644
--- a/components/engine_traits/src/misc.rs
+++ b/components/engine_traits/src/misc.rs
@@ -35,7 +35,13 @@ pub enum DeleteStrategy {
     DeleteByRange,
     /// Delete by ingesting a SST file with deletions. Useful when the number of
     /// ranges is too many.
-    DeleteByWriter { sst_path: String },
+    /// Set `allow_write_during_ingestion` to true to minimize the impact on
+    /// foreground performance, but you must ensure that no concurrent
+    /// writes overlap with the data being ingested.
+    DeleteByWriter {
+        sst_path: String,
+        allow_write_during_ingestion: bool,
+    },
 }
 
 /// `StatisticsReporter` can be used to report engine's private statistics to
diff --git a/components/raftstore/src/store/snap.rs b/components/raftstore/src/store/snap.rs
index 19959540bff..f53de9a37ed 100644
--- a/components/raftstore/src/store/snap.rs
+++ b/components/raftstore/src/store/snap.rs
@@ -1197,7 +1197,13 @@ impl Snapshot {
                     )?;
                 } else {
                     // Apply the snapshot by ingest.
-                    snap_io::apply_sst_cf_files_by_ingest(clone_files.as_slice(), &options.db, cf)?;
+                    snap_io::apply_sst_cf_files_by_ingest(
+                        clone_files.as_slice(),
+                        &options.db,
+                        cf,
+                        enc_start_key(&region),
+                        enc_end_key(&region),
+                    )?;
                     coprocessor_host.post_apply_sst_from_snapshot(&region, cf, path);
                 }
             }
@@ -2053,6 +2059,7 @@ impl SnapManagerCore {
     }
 
     pub fn can_apply_cf_without_ingest(&self, cf_size: u64, cf_kvs: u64) -> bool {
+        fail_point!("apply_cf_without_ingest_false", |_| { false });
         if self.min_ingest_cf_size == 0 {
             return false;
         }
diff --git a/components/raftstore/src/store/snap/io.rs b/components/raftstore/src/store/snap/io.rs
index bf1218fa09b..aec9ce1d0d6 100644
--- a/components/raftstore/src/store/snap/io.rs
+++ b/components/raftstore/src/store/snap/io.rs
@@ -10,7 +10,8 @@ use std::{
 use encryption::{DataKeyManager, DecrypterReader, EncrypterWriter, Iv};
 use engine_traits::{
     CfName, Error as EngineError, ExternalSstFileInfo, IterOptions, Iterable, Iterator, KvEngine,
-    Mutable, RefIterable, SstCompressionType, SstReader, SstWriter, SstWriterBuilder, WriteBatch,
+    Mutable, Range, RefIterable, SstCompressionType, SstReader, SstWriter, SstWriterBuilder,
+    WriteBatch,
 };
 use fail::fail_point;
 use file_system::{File, IoBytesTracker, IoType, OpenOptions, WithIoType};
@@ -352,7 +353,13 @@ where
     }
 }
 
-pub fn apply_sst_cf_files_by_ingest<E>(files: &[&str], db: &E, cf: &str) -> Result<(), Error>
+pub fn apply_sst_cf_files_by_ingest<E>(
+    files: &[&str],
+    db: &E,
+    cf: &str,
+    start_key: Vec<u8>,
+    end_key: Vec<u8>,
+) -> Result<(), Error>
 where
     E: KvEngine,
 {
@@ -362,7 +369,31 @@ where
             cf, files
         );
     }
-    box_try!(db.ingest_external_file_cf(cf, files));
+    // We set start_key and end_key to enable RocksDB
+    // IngestExternalFileOptions.allow_write = true, minimizing the impact on
+    // foreground performance.
+    //
+    // We can safely enable `allow_write` because no concurrent writes overlap with
+    // the data to be ingested, due to:
+    //   1. The region's snapshot is unapplied, ensuring there are no foreground
+    //      write operations.
+    //   2. If a peer is migrated out and then migrated back, and we are in the
+    //      apply snapshot phase, the delete_all_in_range in DestroyTask cannot
+    //      concurrently delete the overlapping key range. This is because the
+    //      single-threaded, queue-based region worker ensures that DestroyTask is
+    //      always completed before ApplyTask is executed.
+    //   3. The compaction filter may write to the default column family
+    //      concurrently, but we use ingest latch to avoid such situations.
+    //
+    // Refer to https://github.com/tikv/tikv/issues/18081.
+    box_try!(db.ingest_external_file_cf(
+        cf,
+        files,
+        Some(Range {
+            start_key: start_key.as_slice(),
+            end_key: end_key.as_slice()
+        })
+    ));
     Ok(())
 }
 
@@ -651,7 +682,8 @@ mod tests {
                         .iter()
                         .map(|s| s.as_str())
                         .collect::<Vec<&str>>();
-                    apply_sst_cf_files_by_ingest(&tmp_file_paths, &db1, CF_DEFAULT).unwrap();
+                    apply_sst_cf_files_by_ingest(&tmp_file_paths, &db1, CF_DEFAULT, vec![], vec![])
+                        .unwrap();
                     assert_eq_db(&db, &db1);
                 }
             }
diff --git a/components/raftstore/src/store/worker/region.rs b/components/raftstore/src/store/worker/region.rs
index 6fbfbc385ae..9d1f67238bf 100644
--- a/components/raftstore/src/store/worker/region.rs
+++ b/components/raftstore/src/store/worker/region.rs
@@ -325,7 +325,7 @@ where
         check_abort(&abort)?;
         self.clean_overlap_ranges(start_key, end_key)?;
         check_abort(&abort)?;
-        fail_point!("apply_snap_cleanup_range");
+        fail_point!("apply_snap_cleanup_range", |_| { Ok(()) });
 
         // apply snapshot
         let apply_state = self.apply_state(region_id)?;
@@ -370,6 +370,7 @@ where
             "region_id" => region_id,
             "time_takes" => ?timer.saturating_elapsed(),
         );
+        fail_point!("apply_snapshot_finished");
         Ok(())
     }
 
@@ -486,8 +487,18 @@ where
     /// Cleans up data in the given range and all pending ranges overlapping
     /// with it.
     fn clean_overlap_ranges(&mut self, start_key: Vec<u8>, end_key: Vec<u8>) -> Result<()> {
+        fail_point!("before_clean_overlap_ranges", |_| { Ok(()) });
         let (start_key, end_key) = self.clean_overlap_ranges_roughly(start_key, end_key);
-        self.delete_all_in_range(&[Range::new(&start_key, &end_key)], false)
+        // We enable the `allow_write_during_ingestion` to enable RocksDB
+        // IngestExternalFileOptions.allow_write = true, minimizing the impact on
+        // foreground performance.
+        // This is safe because no overlapping writes occur during ingestion; for the
+        // reason, refer to the comments in `apply_sst_cf_files_by_ingest`.
+        self.delete_all_in_range(
+            &[Range::new(&start_key, &end_key)],
+            false,
+            true, // allow_write_during_ingestion
+        )
     }
 
     /// Inserts a new pending range, and it will be cleaned up with some delay.
@@ -510,6 +521,7 @@ where
 
     /// Cleans up stale ranges.
     fn clean_stale_ranges(&mut self) {
+        fail_point!("before_clean_stale_ranges", |_| {});
         if self.mgr.is_offlined() {
             info!("no need to clear stale range as store is marked offlined");
             return;
@@ -554,7 +566,7 @@ where
             })
             .unwrap();
         // Remove all overlapped ranges directly without ingesting.
-        if let Err(e) = self.delete_all_in_range(&ranges, true) {
+        if let Err(e) = self.delete_all_in_range(&ranges, true, false) {
             error!("failed to cleanup stale range"; "err" => %e);
             return;
         }
@@ -595,7 +607,12 @@ where
         false
     }
 
-    fn delete_all_in_range(&self, ranges: &[Range<'_>], forcely_by_key: bool) -> Result<()> {
+    fn delete_all_in_range(
+        &self,
+        ranges: &[Range<'_>],
+        forcely_by_key: bool,
+        allow_write_during_ingestion: bool,
+    ) -> Result<()> {
         let wopts = WriteOptions::default();
         for cf in self.engine.cf_names() {
             // CF_LOCK usually contains fewer keys than other CFs, so we delete them by key.
@@ -624,6 +641,7 @@ where
                 (
                     DeleteStrategy::DeleteByWriter {
                         sst_path: self.mgr.get_temp_path_for_ingest(),
+                        allow_write_during_ingestion,
                     },
                     &CLEAR_OVERLAP_REGION_DURATION.by_ingest_files,
                 )
diff --git a/components/sst_importer/src/import_file.rs b/components/sst_importer/src/import_file.rs
index a8fdea6a564..a7770ddbad5 100644
--- a/components/sst_importer/src/import_file.rs
+++ b/components/sst_importer/src/import_file.rs
@@ -413,7 +413,8 @@ impl<E: KvEngine> ImportDir<E> {
 
         for (cf, cf_paths) in paths {
             let files: Vec<&str> = cf_paths.iter().map(|p| p.clone.to_str().unwrap()).collect();
-            engine.ingest_external_file_cf(cf, &files)?;
+            // TODO(hwy): Use RocksDB IngestExternalFileOptions.allow_write = true.
+            engine.ingest_external_file_cf(cf, &files, None)?;
         }
         INPORTER_INGEST_COUNT.observe(metas.len() as _);
         IMPORTER_INGEST_BYTES.observe(ingest_bytes as _);
@@ -607,7 +608,7 @@ mod test {
 
     #[cfg(feature = "test-engines-rocksdb")]
     fn test_path_with_range_and_km(km: Option<DataKeyManager>) {
-        use engine_rocks::{RocksEngine, RocksSstWriterBuilder};
+        use engine_rocks::RocksSstWriterBuilder;
         use engine_test::ctor::{CfOptions, DbOptions};
         use engine_traits::{SstWriter, SstWriterBuilder};
         use tempfile::TempDir;
diff --git a/components/sst_importer/src/util.rs b/components/sst_importer/src/util.rs
index 55ae771c8ae..c20ef14ea28 100644
--- a/components/sst_importer/src/util.rs
+++ b/components/sst_importer/src/util.rs
@@ -210,7 +210,7 @@ mod tests {
         prepare_sst_for_ingestion(&sst_path, &sst_clone, key_manager).unwrap();
         check_hard_link(&sst_path, 2);
         check_hard_link(&sst_clone, 2);
-        db.ingest_external_file_cf(CF_DEFAULT, &[sst_clone.to_str().unwrap()])
+        db.ingest_external_file_cf(CF_DEFAULT, &[sst_clone.to_str().unwrap()], None)
             .unwrap();
         check_db_with_kvs(&db, CF_DEFAULT, &kvs);
         assert!(!sst_clone.exists());
@@ -227,7 +227,7 @@ mod tests {
         prepare_sst_for_ingestion(&sst_path, &sst_clone, key_manager).unwrap();
         check_hard_link(&sst_path, 2);
         check_hard_link(&sst_clone, 1);
-        db.ingest_external_file_cf(CF_DEFAULT, &[sst_clone.to_str().unwrap()])
+        db.ingest_external_file_cf(CF_DEFAULT, &[sst_clone.to_str().unwrap()], None)
             .unwrap();
         check_db_with_kvs(&db, CF_DEFAULT, &kvs);
         assert!(!sst_clone.exists());
@@ -305,7 +305,7 @@ mod tests {
         check_hard_link(&sst_path, 1);
         check_hard_link(&sst_clone, 1);
 
-        db.ingest_external_file_cf(CF_DEFAULT, &[sst_clone.to_str().unwrap()])
+        db.ingest_external_file_cf(CF_DEFAULT, &[sst_clone.to_str().unwrap()], None)
             .unwrap();
         check_db_with_kvs(&db, CF_DEFAULT, &kvs);
         assert!(!sst_clone.exists());
diff --git a/components/tikv_util/src/lib.rs b/components/tikv_util/src/lib.rs
index 7d30fe11874..918d5a9169c 100644
--- a/components/tikv_util/src/lib.rs
+++ b/components/tikv_util/src/lib.rs
@@ -54,6 +54,7 @@ pub mod memory;
 pub mod metrics;
 pub mod mpsc;
 pub mod quota_limiter;
+pub mod range_latch;
 pub mod resizable_threadpool;
 pub mod resource_control;
 pub mod smoother;
diff --git a/components/tikv_util/src/range_latch.rs b/components/tikv_util/src/range_latch.rs
new file mode 100644
index 00000000000..c9552018b13
--- /dev/null
+++ b/components/tikv_util/src/range_latch.rs
@@ -0,0 +1,281 @@
+use std::{
+    collections::{
+        BTreeMap,
+        Bound::{Excluded, Unbounded},
+    },
+    sync::{Arc, Mutex},
+};
+
+/// A structure used to manage range-based latch with mutual exclusion.
+///
+/// Currently used to ensure mutual exclusion between compaction filter and
+/// ingest SST operations. When using RocksDB
+/// IngestExternalFileOptions.allow_write = true for ingest SST, concurrent
+/// overlapping writes must be avoided during ingestion to ensure
+/// sequence number consistency across LSM-tree levels in RocksDB. However, the
+/// compaction filter might concurrently write overlapping keys to the default
+/// column family. For example, if a region is migrated out and then migrated
+/// back, it might cause concurrent compaction filter overlaps with the
+/// apply-snapshot-ingest process. Therefore, it must be made mutually exclusive
+/// with the ingest latch.
+//
+/// This implementation is designed for scenarios with low concurrency.
+/// In the current scenario, the number of compaction filter threads are limited
+/// by `RocksDB.max_background_jobs`, and the region worker, which handles
+/// apply-snapshot-ingest or destroy-peer-ingest, operates as a single thread.
+/// Due to this low level of concurrency, the overhead of range latch is
+/// minimal, and potential conflicts are rare.
+///
+/// NOTE: Both the compaction filter and ingest SST operations that
+/// use this `RangeLatch` enforce self-mutual exclusion. While this might
+/// appear somewhat overkill, its impact on performance is minimal due to
+/// the system's low concurrency and the brief duration of latch holding.
+///
+/// NOTE: Why do we use a range-latch instead of a region-id latch? This is
+/// because region-id is mutable and there is currently no mechanism to notify
+/// the compaction filter in real time. For example, if a region is migrated
+/// out, split, and then migrated back, the compaction filter might hold the old
+/// region-id while the apply-snapshot-ingest process holds the new region-id.
+#[derive(Debug, Default)]
+pub struct RangeLatch {
+    /// A BTreeMap storing active range latches.
+    /// Key: The start key of the range (sorted order).
+    /// Value: A tuple of:
+    ///   - `Arc<Mutex<()>>`: The latch object for this range.
+    ///   - `(Vec<u8>, Vec<u8>)`: The actual range definition (start_key,
+    ///     end_key).
+    range_latches: Mutex<BTreeMap<Vec<u8>, (Arc<Mutex<()>>, (Vec<u8>, Vec<u8>))>>,
+}
+
+impl RangeLatch {
+    pub fn new() -> Self {
+        Self {
+            range_latches: Mutex::new(BTreeMap::new()),
+        }
+    }
+
+    /// Acquires a range latch for the specified `[start_key, end_key)`.
+    /// This function ensures that a range latch is only granted when there are
+    /// no overlapping ranges already latched. If overlaps exist, the
+    /// function waits until those latches are released before proceeding.
+    ///
+    /// Live-locks can occur in this implementation. For example, a thread
+    /// attempting to acquire a latch for the range `[a, z)` might encounter
+    /// that `[a, b)` and `[b, c)` are already latched by other threads. The
+    /// thread will first wait for [a, b) to be released and successfully
+    /// acquire it. However, while waiting for [b, c) to be released,
+    /// another thread might acquire [a, b) again, forcing the original thread
+    /// to wait for [a, b) once more. This process could repeat
+    /// indefinitely if threads continuously "jump the queue.". Fortunately,
+    /// such live-locks are unlikely to persist for long due to the user's low
+    /// concurrency, as conflicting latches are eventually released, allowing
+    /// threads to make progress naturally.
+    ///
+    /// Deadlocks cannot occur in the current scenario, as each caller thread
+    /// holds at most one lock at a time.
+    pub fn acquire<'a>(
+        self: &'a Arc<Self>,
+        start_key: Vec<u8>,
+        end_key: Vec<u8>,
+    ) -> RangeLatchGuard<'a> {
+        loop {
+            let mut range_latches = self.range_latches.lock().unwrap();
+
+            // Collect all overlapping ranges
+            let overlapping_ranges = range_latches
+                .range((Unbounded::<Vec<u8>>, Excluded(end_key.clone())))
+                .filter(|(_, (_, (existing_start, existing_end)))| {
+                    !(end_key <= *existing_start || start_key >= *existing_end)
+                })
+                .map(|(key, (mutex, range))| (key.clone(), (mutex.clone(), range.clone())))
+                .collect::<Vec<_>>();
+
+            // If no conflicts, insert the new range and return the guard
+            if overlapping_ranges.is_empty() {
+                let mutex = Arc::new(Mutex::new(()));
+                let previous_value = range_latches.insert(
+                    start_key.clone(),
+                    (mutex.clone(), (start_key.clone(), end_key.clone())),
+                );
+                debug_assert!(previous_value.is_none());
+
+                // Now acquire the latch after releasing the write guard
+                let mutex_guard = mutex.lock().unwrap();
+                // Safety: `_mutex_guard` is declared before `handle` in `KeyHandleGuard`.
+                // So the mutex guard will be released earlier than the `Arc<KeyHandle>`.
+                // Then we can make sure the mutex guard doesn't point to released memory.
+                let mutex_guard = unsafe { std::mem::transmute(mutex_guard) };
+
+                return RangeLatchGuard {
+                    start_key,
+                    _mutex_guard: mutex_guard,
+                    handle: self,
+                };
+            }
+            drop(range_latches);
+
+            // Wait for all overlapping ranges to be released
+            for (_, (mutex, (..))) in overlapping_ranges {
+                let _guard = mutex.lock().unwrap();
+            }
+        }
+    }
+}
+
+/// A guard that holds the range latch.
+#[derive(Debug)]
+pub struct RangeLatchGuard<'a> {
+    start_key: Vec<u8>,
+    /// Hold the mutex guard to prevent concurrent access to the same range.
+    ///
+    /// This field must be declared before `handle` so it will be dropped before
+    /// `handle`.
+    _mutex_guard: std::sync::MutexGuard<'a, ()>,
+    /// Holds a reference to RangeLatch to release the latch when the guard is
+    /// dropped.
+    handle: &'a RangeLatch,
+}
+
+impl<'a> Drop for RangeLatchGuard<'a> {
+    fn drop(&mut self) {
+        let mut range_latches = self.handle.range_latches.lock().unwrap();
+        range_latches.remove(&self.start_key);
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::{
+        collections::HashSet,
+        sync::{
+            atomic::{AtomicBool, Ordering},
+            Arc,
+        },
+        thread,
+        time::Duration,
+    };
+
+    use rand::Rng;
+
+    use super::*;
+
+    #[test]
+    fn test_single_thread_non_overlapping_ranges() {
+        let latch = Arc::new(RangeLatch::new());
+
+        // Acquire non-overlapping ranges
+        let guard1 = latch.acquire(vec![0], vec![10]);
+        let guard2 = latch.acquire(vec![20], vec![30]);
+        let guard3 = latch.acquire(vec![30], vec![50]);
+
+        // Drop guards to release latches
+        drop(guard1);
+        drop(guard2);
+        drop(guard3);
+    }
+
+    #[test]
+    fn test_two_threads_overlapping_ranges() {
+        fn do_test(range1: (Vec<u8>, Vec<u8>), range2: (Vec<u8>, Vec<u8>)) {
+            let latch = Arc::new(RangeLatch::new());
+            let latch_clone = Arc::clone(&latch);
+            let is_thread_done = Arc::new(AtomicBool::new(false));
+            let is_thread_done_clone = Arc::clone(&is_thread_done);
+
+            // Main thread acquires the first latch
+            let guard = latch.acquire(range1.0, range1.1);
+
+            let handle = thread::spawn(move || {
+                let _guard = latch_clone.acquire(range2.0, range2.1);
+                is_thread_done_clone.store(true, Ordering::SeqCst);
+            });
+
+            thread::sleep(Duration::from_millis(500));
+            // Verify that the second thread is still waiting
+            assert!(!is_thread_done.load(Ordering::SeqCst));
+
+            // Release the first latch
+            drop(guard);
+
+            // Wait a bit to let the second thread complete
+            thread::sleep(Duration::from_millis(100));
+            assert!(is_thread_done.load(Ordering::SeqCst));
+
+            // Ensure the second thread completes successfully
+            handle.join().unwrap();
+        }
+
+        // Test different overlapping cases
+        do_test((vec![0], vec![10]), (vec![5], vec![15])); // Overlap at the end
+        do_test((vec![5], vec![15]), (vec![0], vec![10])); // Overlap at the begin
+        do_test((vec![0], vec![10]), (vec![4], vec![5])); // Contained within
+        do_test((vec![5], vec![15]), (vec![0], vec![20])); // Fully contained
+    }
+
+    fn concurrent_random_ranges_test(num_threads: usize, num_latches: usize) {
+        // Shared latch and active ranges tracker
+        let latch = Arc::new(RangeLatch::new());
+        let active_ranges = Arc::new(Mutex::new(HashSet::new()));
+
+        let mut handles = Vec::new();
+
+        for _ in 0..num_threads {
+            let latch_clone = Arc::clone(&latch);
+            let active_ranges_clone = Arc::clone(&active_ranges);
+
+            handles.push(thread::spawn(move || {
+                let mut rng = rand::thread_rng();
+
+                for _ in 0..num_latches {
+                    let start: u8 = rng.gen_range(0..200);
+                    let end = rng.gen_range(start + 1..201);
+
+                    // Acquire the range latch
+                    let _guard = latch_clone.acquire(vec![start], vec![end]);
+
+                    // Check for overlap
+                    {
+                        let mut ranges = active_ranges_clone.lock().unwrap();
+                        for (existing_start, existing_end) in ranges.iter() {
+                            assert!(
+                                end <= *existing_start || start >= *existing_end,
+                                "Overlapping range detected: [{}, {}) overlaps with [{}, {})",
+                                start,
+                                end,
+                                existing_start,
+                                existing_end
+                            );
+                        }
+
+                        // Add the current range to the active set
+                        ranges.insert((start, end));
+                    }
+
+                    // Hold the latch for a short period
+                    thread::sleep(Duration::from_millis(10));
+
+                    // Remove the range from the active set
+                    {
+                        let mut ranges = active_ranges_clone.lock().unwrap();
+                        ranges.remove(&(start, end));
+                    }
+                }
+            }));
+        }
+
+        // Wait for all threads to complete
+        for handle in handles {
+            handle.join().unwrap();
+        }
+    }
+
+    #[test]
+    fn test_concurrent_random_ranges_many_latches() {
+        concurrent_random_ranges_test(5, 100);
+    }
+
+    #[test]
+    fn test_concurrent_random_ranges_many_threads() {
+        concurrent_random_ranges_test(100, 5);
+    }
+}
diff --git a/metrics/grafana/tikv_details.dashboard.py b/metrics/grafana/tikv_details.dashboard.py
index 315f12f1a8b..093c65f3024 100644
--- a/metrics/grafana/tikv_details.dashboard.py
+++ b/metrics/grafana/tikv_details.dashboard.py
@@ -6637,6 +6637,24 @@ def RocksDB() -> RowPanel:
             ),
         ]
     )
+    layout.row(
+        [
+            graph_panel(
+                title="Ingest SST allow_write",
+                description=None,
+                yaxes=yaxes(left_format=UNITS.SHORT),
+                targets=[
+                    target(
+                        expr=expr_sum_rate(
+                            "tikv_storage_ingest_external_file_allow_write_counter",
+                            by_labels=["type"],
+                        ),
+                        additional_groupby=True,
+                    ),
+                ],
+            ),
+        ]
+    )
     layout.row(
         [
             graph_panel(
diff --git a/metrics/grafana/tikv_details.json b/metrics/grafana/tikv_details.json
index db5325a7cf3..17b12d422af 100644
--- a/metrics/grafana/tikv_details.json
+++ b/metrics/grafana/tikv_details.json
@@ -34326,7 +34326,7 @@
           },
           "gridPos": {
             "h": 7,
-            "w": 12,
+            "w": 24,
             "x": 0,
             "y": 119
           },
@@ -34372,6 +34372,139 @@
           "span": null,
           "stack": false,
           "steppedLine": false,
+          "targets": [
+            {
+              "datasource": "${DS_TEST-CLUSTER}",
+              "expr": "sum(rate(\n    tikv_storage_ingest_external_file_allow_write_counter\n    {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\"}\n    [$__rate_interval]\n)) by (type, $additional_groupby) ",
+              "format": "time_series",
+              "hide": false,
+              "instant": false,
+              "interval": "",
+              "intervalFactor": 1,
+              "legendFormat": "{{type}} {{$additional_groupby}}",
+              "metric": "",
+              "query": "sum(rate(\n    tikv_storage_ingest_external_file_allow_write_counter\n    {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\"}\n    [$__rate_interval]\n)) by (type, $additional_groupby) ",
+              "refId": "",
+              "step": 10,
+              "target": ""
+            }
+          ],
+          "thresholds": [],
+          "timeFrom": null,
+          "timeShift": null,
+          "title": "Ingest SST allow_write",
+          "tooltip": {
+            "msResolution": true,
+            "shared": true,
+            "sort": 0,
+            "value_type": "individual"
+          },
+          "transformations": [],
+          "transparent": false,
+          "type": "graph",
+          "xaxis": {
+            "mode": "time",
+            "name": null,
+            "show": true,
+            "values": []
+          },
+          "yaxes": [
+            {
+              "decimals": null,
+              "format": "short",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            },
+            {
+              "decimals": null,
+              "format": "short",
+              "label": null,
+              "logBase": 1,
+              "max": null,
+              "min": null,
+              "show": true
+            }
+          ],
+          "yaxis": {
+            "align": false,
+            "alignLevel": 0
+          }
+        },
+        {
+          "aliasColors": {},
+          "bars": false,
+          "cacheTimeout": null,
+          "datasource": "${DS_TEST-CLUSTER}",
+          "description": null,
+          "editable": true,
+          "error": false,
+          "fieldConfig": {
+            "defaults": {
+              "thresholds": {
+                "mode": "absolute",
+                "steps": []
+              }
+            }
+          },
+          "fill": 1,
+          "fillGradient": 1,
+          "grid": {
+            "threshold1": null,
+            "threshold1Color": "rgba(216, 200, 27, 0.27)",
+            "threshold2": null,
+            "threshold2Color": "rgba(234, 112, 112, 0.22)"
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 12,
+            "x": 0,
+            "y": 126
+          },
+          "height": null,
+          "hideTimeOverride": false,
+          "id": 244,
+          "interval": null,
+          "isNew": true,
+          "legend": {
+            "alignAsTable": true,
+            "avg": false,
+            "current": true,
+            "hideEmpty": true,
+            "hideZero": true,
+            "max": true,
+            "min": false,
+            "rightSide": true,
+            "show": true,
+            "sideWidth": null,
+            "sort": "max",
+            "sortDesc": true,
+            "total": false,
+            "values": true
+          },
+          "lines": true,
+          "linewidth": 1,
+          "links": [],
+          "maxDataPoints": null,
+          "maxPerRow": null,
+          "minSpan": null,
+          "nullPointMode": "null as zero",
+          "options": {
+            "alertThreshold": true,
+            "dataLinks": []
+          },
+          "percentage": false,
+          "pointradius": 5,
+          "points": false,
+          "renderer": "flot",
+          "repeat": null,
+          "repeatDirection": null,
+          "seriesOverrides": [],
+          "span": null,
+          "stack": false,
+          "steppedLine": false,
           "targets": [
             {
               "datasource": "${DS_TEST-CLUSTER}",
@@ -34461,11 +34594,11 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 119
+            "y": 126
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 244,
+          "id": 245,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -34639,11 +34772,11 @@
             "h": 7,
             "w": 12,
             "x": 0,
-            "y": 126
+            "y": 133
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 245,
+          "id": 246,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -34772,11 +34905,11 @@
             "h": 7,
             "w": 12,
             "x": 12,
-            "y": 126
+            "y": 133
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 246,
+          "id": 247,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -34912,7 +35045,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 247,
+      "id": 248,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -34951,7 +35084,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 248,
+          "id": 249,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -35099,7 +35232,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 249,
+          "id": 250,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -35247,7 +35380,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 250,
+          "id": 251,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -35380,7 +35513,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 251,
+          "id": 252,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -35513,7 +35646,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 252,
+          "id": 253,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -35691,7 +35824,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 253,
+          "id": 254,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -35869,7 +36002,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 254,
+          "id": 255,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -36047,7 +36180,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 255,
+          "id": 256,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -36180,7 +36313,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 256,
+          "id": 257,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -36358,7 +36491,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 257,
+          "id": 258,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -36491,7 +36624,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 258,
+          "id": 259,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -36654,7 +36787,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 259,
+          "id": 260,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -36832,7 +36965,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 260,
+          "id": 261,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -37010,7 +37143,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 261,
+          "id": 262,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -37188,7 +37321,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 262,
+          "id": 263,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -37321,7 +37454,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 263,
+          "id": 264,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -37499,7 +37632,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 264,
+          "id": 265,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -37632,7 +37765,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 265,
+          "id": 266,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -37810,7 +37943,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 266,
+          "id": 267,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -37943,7 +38076,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 267,
+          "id": 268,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -38076,7 +38209,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 268,
+          "id": 269,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -38254,7 +38387,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 269,
+          "id": 270,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -38432,7 +38565,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 270,
+          "id": 271,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -38565,7 +38698,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 271,
+          "id": 272,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -38743,7 +38876,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 272,
+          "id": 273,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -38876,7 +39009,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 273,
+          "id": 274,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -39054,7 +39187,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 274,
+          "id": 275,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -39190,7 +39323,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 275,
+      "id": 276,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -39229,7 +39362,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 276,
+          "id": 277,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -39362,7 +39495,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 277,
+          "id": 278,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -39510,7 +39643,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 278,
+          "id": 279,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -39711,7 +39844,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 279,
+          "id": 280,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -39844,7 +39977,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 280,
+          "id": 281,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -39977,7 +40110,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 281,
+          "id": 282,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -40110,7 +40243,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 282,
+          "id": 283,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -40243,7 +40376,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 283,
+          "id": 284,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -40376,7 +40509,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 284,
+          "id": 285,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -40516,7 +40649,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 285,
+          "id": 286,
           "interval": null,
           "legend": {
             "show": false
@@ -40621,7 +40754,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 286,
+          "id": 287,
           "interval": null,
           "legend": {
             "show": false
@@ -40719,7 +40852,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 287,
+          "id": 288,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -40859,7 +40992,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 288,
+          "id": 289,
           "interval": null,
           "legend": {
             "show": false
@@ -40957,7 +41090,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 289,
+          "id": 290,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -41097,7 +41230,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 290,
+          "id": 291,
           "interval": null,
           "legend": {
             "show": false
@@ -41195,7 +41328,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 291,
+          "id": 292,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -41403,7 +41536,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 292,
+          "id": 293,
           "interval": null,
           "legend": {
             "show": false
@@ -41501,7 +41634,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 293,
+          "id": 294,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -41702,7 +41835,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 294,
+          "id": 295,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -41910,7 +42043,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 295,
+          "id": 296,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -42088,7 +42221,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 296,
+          "id": 297,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -42221,7 +42354,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 297,
+          "id": 298,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -42354,7 +42487,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 298,
+          "id": 299,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -42487,7 +42620,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 299,
+          "id": 300,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -42623,7 +42756,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 300,
+      "id": 301,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -42662,7 +42795,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 301,
+          "id": 302,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -42810,7 +42943,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 302,
+          "id": 303,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -42950,7 +43083,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 303,
+          "id": 304,
           "interval": null,
           "legend": {
             "show": false
@@ -43048,7 +43181,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 304,
+          "id": 305,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -43181,7 +43314,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 305,
+          "id": 306,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -43314,7 +43447,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 306,
+          "id": 307,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -43492,7 +43625,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 307,
+          "id": 308,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -43655,7 +43788,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 308,
+          "id": 309,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -43803,7 +43936,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 309,
+          "id": 310,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -43936,7 +44069,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 310,
+          "id": 311,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -44072,7 +44205,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 311,
+      "id": 312,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -44111,7 +44244,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 312,
+          "id": 313,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -44259,7 +44392,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 313,
+          "id": 314,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -44392,7 +44525,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 314,
+          "id": 315,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -44525,7 +44658,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 315,
+          "id": 316,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -44658,7 +44791,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 316,
+          "id": 317,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -44791,7 +44924,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 317,
+          "id": 318,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -44946,7 +45079,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 318,
+          "id": 319,
           "interval": null,
           "legend": {
             "show": false
@@ -45047,7 +45180,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 319,
+      "id": 320,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -45086,7 +45219,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 320,
+          "id": 321,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -45219,7 +45352,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 321,
+          "id": 322,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -45352,7 +45485,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 322,
+          "id": 323,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -45492,7 +45625,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 323,
+          "id": 324,
           "interval": null,
           "legend": {
             "show": false
@@ -45590,7 +45723,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 324,
+          "id": 325,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -45791,7 +45924,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 325,
+          "id": 326,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -45992,7 +46125,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 326,
+          "id": 327,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -46196,7 +46329,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 327,
+      "id": 328,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -46235,7 +46368,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 328,
+          "id": 329,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -46383,7 +46516,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 329,
+          "id": 330,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -46584,7 +46717,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 330,
+          "id": 331,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -46785,7 +46918,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 331,
+          "id": 332,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -46986,7 +47119,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 332,
+          "id": 333,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -47187,7 +47320,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 333,
+          "id": 334,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -47320,7 +47453,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 334,
+          "id": 335,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -47453,7 +47586,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 335,
+          "id": 336,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -47586,7 +47719,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 336,
+          "id": 337,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -47719,7 +47852,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 337,
+          "id": 338,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -47927,7 +48060,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 338,
+          "id": 339,
           "interval": null,
           "legend": {
             "show": false
@@ -48028,7 +48161,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 339,
+      "id": 340,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -48074,7 +48207,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 340,
+          "id": 341,
           "interval": null,
           "legend": {
             "show": false
@@ -48172,7 +48305,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 341,
+          "id": 342,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -48373,7 +48506,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 342,
+          "id": 343,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -48506,7 +48639,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 343,
+          "id": 344,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -48639,7 +48772,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 344,
+          "id": 345,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -48772,7 +48905,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 345,
+          "id": 346,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -48973,7 +49106,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 346,
+          "id": 347,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -49106,7 +49239,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 347,
+          "id": 348,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -49239,7 +49372,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 348,
+          "id": 349,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -49375,7 +49508,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 349,
+      "id": 350,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -49414,7 +49547,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 350,
+          "id": 351,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -49615,7 +49748,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 351,
+          "id": 352,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -49816,7 +49949,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 352,
+          "id": 353,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -50017,7 +50150,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 353,
+          "id": 354,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -50218,7 +50351,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 354,
+          "id": 355,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -50351,7 +50484,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 355,
+          "id": 356,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -50484,7 +50617,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 356,
+          "id": 357,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -50617,7 +50750,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 357,
+          "id": 358,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -50750,7 +50883,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 358,
+          "id": 359,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -50883,7 +51016,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 359,
+          "id": 360,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -51023,7 +51156,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 360,
+          "id": 361,
           "interval": null,
           "legend": {
             "show": false
@@ -51121,7 +51254,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 361,
+          "id": 362,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -51325,7 +51458,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 362,
+      "id": 363,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -51364,7 +51497,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 363,
+          "id": 364,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -51497,7 +51630,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 364,
+          "id": 365,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -51630,7 +51763,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 365,
+          "id": 366,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -51770,7 +51903,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 366,
+          "id": 367,
           "interval": null,
           "legend": {
             "show": false
@@ -51868,7 +52001,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 367,
+          "id": 368,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -52069,7 +52202,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 368,
+          "id": 369,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -52270,7 +52403,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 369,
+          "id": 370,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -52474,7 +52607,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 370,
+      "id": 371,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -52513,7 +52646,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 371,
+          "id": 372,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -52691,7 +52824,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 372,
+          "id": 373,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -52892,7 +53025,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 373,
+          "id": 374,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -53025,7 +53158,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 374,
+          "id": 375,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -53158,7 +53291,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 375,
+          "id": 376,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -53291,7 +53424,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 376,
+          "id": 377,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -53424,7 +53557,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 377,
+          "id": 378,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -53557,7 +53690,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 378,
+          "id": 379,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -53686,7 +53819,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 379,
+          "id": 380,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -53761,7 +53894,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 380,
+          "id": 381,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -53840,7 +53973,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 381,
+          "id": 382,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -54093,7 +54226,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 382,
+          "id": 383,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -54226,7 +54359,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 383,
+          "id": 384,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -54362,7 +54495,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 384,
+      "id": 385,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -54401,7 +54534,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 385,
+          "id": 386,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -54549,7 +54682,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 386,
+          "id": 387,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -54682,7 +54815,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 387,
+          "id": 388,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -54883,7 +55016,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 388,
+          "id": 389,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -55031,7 +55164,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 389,
+          "id": 390,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -55232,7 +55365,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 390,
+          "id": 391,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -55365,7 +55498,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 391,
+          "id": 392,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -55498,7 +55631,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 392,
+          "id": 393,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -55631,7 +55764,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 393,
+          "id": 394,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -55764,7 +55897,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 394,
+          "id": 395,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -55904,7 +56037,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 395,
+          "id": 396,
           "interval": null,
           "legend": {
             "show": false
@@ -56002,7 +56135,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 396,
+          "id": 397,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -56206,7 +56339,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 397,
+      "id": 398,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -56245,7 +56378,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 398,
+          "id": 399,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -56378,7 +56511,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 399,
+          "id": 400,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -56511,7 +56644,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 400,
+          "id": 401,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -56644,7 +56777,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 401,
+          "id": 402,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -56780,7 +56913,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 402,
+      "id": 403,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -56819,7 +56952,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 403,
+          "id": 404,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -56952,7 +57085,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 404,
+          "id": 405,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -57085,7 +57218,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 405,
+          "id": 406,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -57233,7 +57366,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 406,
+          "id": 407,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -57366,7 +57499,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 407,
+          "id": 408,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -57499,7 +57632,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 408,
+          "id": 409,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -57632,7 +57765,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 409,
+          "id": 410,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -57768,7 +57901,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 410,
+      "id": 411,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -57807,7 +57940,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 411,
+          "id": 412,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -57940,7 +58073,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 412,
+          "id": 413,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -58073,7 +58206,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 413,
+          "id": 414,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -58206,7 +58339,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 414,
+          "id": 415,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -58339,7 +58472,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 415,
+          "id": 416,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -58472,7 +58605,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 416,
+          "id": 417,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -58608,7 +58741,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 417,
+      "id": 418,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -58647,7 +58780,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 418,
+          "id": 419,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -58780,7 +58913,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 419,
+          "id": 420,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -58913,7 +59046,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 420,
+          "id": 421,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -59061,7 +59194,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 421,
+          "id": 422,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -59224,7 +59357,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 422,
+          "id": 423,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -59357,7 +59490,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 423,
+          "id": 424,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -59490,7 +59623,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 424,
+          "id": 425,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -59653,7 +59786,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 425,
+          "id": 426,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -59801,7 +59934,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 426,
+          "id": 427,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -59937,7 +60070,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 427,
+      "id": 428,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -59976,7 +60109,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 428,
+          "id": 429,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -60109,7 +60242,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 429,
+          "id": 430,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -60242,7 +60375,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 430,
+          "id": 431,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -60375,7 +60508,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 431,
+          "id": 432,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -60508,7 +60641,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 432,
+          "id": 433,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -60641,7 +60774,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 433,
+          "id": 434,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -60774,7 +60907,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 434,
+          "id": 435,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -60907,7 +61040,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 435,
+          "id": 436,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -61040,7 +61173,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 436,
+          "id": 437,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -61180,7 +61313,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 437,
+          "id": 438,
           "interval": null,
           "legend": {
             "show": false
@@ -61278,7 +61411,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 438,
+          "id": 439,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -61411,7 +61544,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 439,
+          "id": 440,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -61559,7 +61692,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 440,
+          "id": 441,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -61707,7 +61840,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 441,
+          "id": 442,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -61847,7 +61980,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 442,
+          "id": 443,
           "interval": null,
           "legend": {
             "show": false
@@ -61945,7 +62078,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 443,
+          "id": 444,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -62078,7 +62211,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 444,
+          "id": 445,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -62214,7 +62347,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 445,
+      "id": 446,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -62253,7 +62386,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 446,
+          "id": 447,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -62386,7 +62519,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 447,
+          "id": 448,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -62549,7 +62682,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 448,
+          "id": 449,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -62697,7 +62830,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 449,
+          "id": 450,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -62830,7 +62963,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 450,
+          "id": 451,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -62970,7 +63103,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 451,
+          "id": 452,
           "interval": null,
           "legend": {
             "show": false
@@ -63075,7 +63208,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 452,
+          "id": 453,
           "interval": null,
           "legend": {
             "show": false
@@ -63180,7 +63313,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 453,
+          "id": 454,
           "interval": null,
           "legend": {
             "show": false
@@ -63278,7 +63411,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 454,
+          "id": 455,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -63418,7 +63551,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 455,
+          "id": 456,
           "interval": null,
           "legend": {
             "show": false
@@ -63523,7 +63656,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 456,
+          "id": 457,
           "interval": null,
           "legend": {
             "show": false
@@ -63628,7 +63761,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 457,
+          "id": 458,
           "interval": null,
           "legend": {
             "show": false
@@ -63726,7 +63859,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 458,
+          "id": 459,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -63859,7 +63992,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 459,
+          "id": 460,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -63992,7 +64125,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 460,
+          "id": 461,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -64132,7 +64265,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 461,
+          "id": 462,
           "interval": null,
           "legend": {
             "show": false
@@ -64230,7 +64363,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 462,
+          "id": 463,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -64366,7 +64499,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 463,
+      "id": 464,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -64405,7 +64538,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 464,
+          "id": 465,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -64568,7 +64701,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 465,
+          "id": 466,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -64701,7 +64834,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 466,
+          "id": 467,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -64841,7 +64974,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 467,
+          "id": 468,
           "interval": null,
           "legend": {
             "show": false
@@ -64946,7 +65079,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 468,
+          "id": 469,
           "interval": null,
           "legend": {
             "show": false
@@ -65044,7 +65177,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 469,
+          "id": 470,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -65199,7 +65332,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 470,
+          "id": 471,
           "interval": null,
           "legend": {
             "show": false
@@ -65304,7 +65437,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 471,
+          "id": 472,
           "interval": null,
           "legend": {
             "show": false
@@ -65409,7 +65542,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 472,
+          "id": 473,
           "interval": null,
           "legend": {
             "show": false
@@ -65507,7 +65640,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 473,
+          "id": 474,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -65677,7 +65810,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 474,
+          "id": 475,
           "interval": null,
           "legend": {
             "show": false
@@ -65775,7 +65908,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 475,
+          "id": 476,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -65976,7 +66109,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 476,
+          "id": 477,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -66177,7 +66310,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 477,
+          "id": 478,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -66310,7 +66443,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 478,
+          "id": 479,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -66473,7 +66606,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 479,
+          "id": 480,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -66606,7 +66739,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 480,
+          "id": 481,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -66739,7 +66872,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 481,
+          "id": 482,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -66940,7 +67073,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 482,
+          "id": 483,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -67073,7 +67206,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 483,
+          "id": 484,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -67213,7 +67346,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 484,
+          "id": 485,
           "interval": null,
           "legend": {
             "show": false
@@ -67318,7 +67451,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 485,
+          "id": 486,
           "interval": null,
           "legend": {
             "show": false
@@ -67423,7 +67556,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 486,
+          "id": 487,
           "interval": null,
           "legend": {
             "show": false
@@ -67528,7 +67661,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 487,
+          "id": 488,
           "interval": null,
           "legend": {
             "show": false
@@ -67633,7 +67766,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 488,
+          "id": 489,
           "interval": null,
           "legend": {
             "show": false
@@ -67738,7 +67871,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 489,
+          "id": 490,
           "interval": null,
           "legend": {
             "show": false
@@ -67843,7 +67976,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 490,
+          "id": 491,
           "interval": null,
           "legend": {
             "show": false
@@ -67941,7 +68074,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 491,
+          "id": 492,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -68089,7 +68222,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 492,
+          "id": 493,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -68222,7 +68355,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 493,
+          "id": 494,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -68355,7 +68488,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 494,
+          "id": 495,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -68503,7 +68636,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 495,
+          "id": 496,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -68639,7 +68772,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 496,
+      "id": 497,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -68690,7 +68823,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 497,
+          "id": 498,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -68786,7 +68919,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 498,
+          "id": 499,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -68861,7 +68994,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 499,
+          "id": 500,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -68936,7 +69069,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 500,
+          "id": 501,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -69011,7 +69144,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 501,
+          "id": 502,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -69086,7 +69219,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 502,
+          "id": 503,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -69161,7 +69294,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 503,
+          "id": 504,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -69236,7 +69369,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 504,
+          "id": 505,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
@@ -69315,7 +69448,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 505,
+          "id": 506,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -69448,7 +69581,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 506,
+          "id": 507,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -69581,7 +69714,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 507,
+          "id": 508,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -69714,7 +69847,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 508,
+          "id": 509,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -69847,7 +69980,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 509,
+          "id": 510,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -69980,7 +70113,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 510,
+          "id": 511,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -70128,7 +70261,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 511,
+          "id": 512,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -70261,7 +70394,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 512,
+          "id": 513,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -70394,7 +70527,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 513,
+          "id": 514,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -70560,7 +70693,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 514,
+          "id": 515,
           "interval": null,
           "legend": {
             "show": false
@@ -70665,7 +70798,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 515,
+          "id": 516,
           "interval": null,
           "legend": {
             "show": false
@@ -70770,7 +70903,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 516,
+          "id": 517,
           "interval": null,
           "legend": {
             "show": false
@@ -70875,7 +71008,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 517,
+          "id": 518,
           "interval": null,
           "legend": {
             "show": false
@@ -70980,7 +71113,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 518,
+          "id": 519,
           "interval": null,
           "legend": {
             "show": false
@@ -71085,7 +71218,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 519,
+          "id": 520,
           "interval": null,
           "legend": {
             "show": false
@@ -71190,7 +71323,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 520,
+          "id": 521,
           "interval": null,
           "legend": {
             "show": false
@@ -71295,7 +71428,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 521,
+          "id": 522,
           "interval": null,
           "legend": {
             "show": false
@@ -71393,7 +71526,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 522,
+          "id": 523,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -71526,7 +71659,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 523,
+          "id": 524,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -71659,7 +71792,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 524,
+          "id": 525,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -71792,7 +71925,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 525,
+          "id": 526,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -71925,7 +72058,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 526,
+          "id": 527,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -72058,7 +72191,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 527,
+          "id": 528,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -72191,7 +72324,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 528,
+          "id": 529,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -72324,7 +72457,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 529,
+          "id": 530,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -72464,7 +72597,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 530,
+          "id": 531,
           "interval": null,
           "legend": {
             "show": false
@@ -72569,7 +72702,7 @@
           "hideTimeOverride": false,
           "hideZeroBuckets": true,
           "highlightCards": true,
-          "id": 531,
+          "id": 532,
           "interval": null,
           "legend": {
             "show": false
@@ -72667,7 +72800,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 532,
+          "id": 533,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -72800,7 +72933,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 533,
+          "id": 534,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -72933,7 +73066,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 534,
+          "id": 535,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -73066,7 +73199,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 535,
+          "id": 536,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -73199,7 +73332,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 536,
+          "id": 537,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -73332,7 +73465,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 537,
+          "id": 538,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -73468,7 +73601,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 538,
+      "id": 539,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -73507,7 +73640,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 539,
+          "id": 540,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -73655,7 +73788,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 540,
+          "id": 541,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -73788,7 +73921,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 541,
+          "id": 542,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -73921,7 +74054,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 542,
+          "id": 543,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -74057,7 +74190,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 543,
+      "id": 544,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -74096,7 +74229,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 544,
+          "id": 545,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -74229,7 +74362,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 545,
+          "id": 546,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -74362,7 +74495,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 546,
+          "id": 547,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -74495,7 +74628,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 547,
+          "id": 548,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -74628,7 +74761,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 548,
+          "id": 549,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -74761,7 +74894,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 549,
+          "id": 550,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -74897,7 +75030,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 550,
+      "id": 551,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -74936,7 +75069,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 551,
+          "id": 552,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -75069,7 +75202,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 552,
+          "id": 553,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -75205,7 +75338,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 553,
+      "id": 554,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -75244,7 +75377,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 554,
+          "id": 555,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -75445,7 +75578,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 555,
+          "id": 556,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -75581,7 +75714,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 556,
+      "id": 557,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -75620,7 +75753,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 557,
+          "id": 558,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -75753,7 +75886,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 558,
+          "id": 559,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -75886,7 +76019,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 559,
+          "id": 560,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -76019,7 +76152,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 560,
+          "id": 561,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -76152,7 +76285,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 561,
+          "id": 562,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -76300,7 +76433,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 562,
+          "id": 563,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -76504,7 +76637,7 @@
       },
       "height": null,
       "hideTimeOverride": false,
-      "id": 563,
+      "id": 564,
       "interval": null,
       "links": [],
       "maxDataPoints": 100,
@@ -76543,7 +76676,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 564,
+          "id": 565,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -76676,7 +76809,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 565,
+          "id": 566,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -76809,7 +76942,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 566,
+          "id": 567,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -76942,7 +77075,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 567,
+          "id": 568,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -77075,7 +77208,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 568,
+          "id": 569,
           "interval": null,
           "isNew": true,
           "legend": {
@@ -77272,7 +77405,7 @@
           },
           "height": null,
           "hideTimeOverride": false,
-          "id": 569,
+          "id": 570,
           "interval": null,
           "links": [],
           "maxDataPoints": 100,
diff --git a/metrics/grafana/tikv_details.json.sha256 b/metrics/grafana/tikv_details.json.sha256
index 4b5f37d9f4f..1e50fbfc586 100644
--- a/metrics/grafana/tikv_details.json.sha256
+++ b/metrics/grafana/tikv_details.json.sha256
@@ -1,1 +1,1 @@
-3508479cbef7b21a2a19197e9cf12c77cc251c3cca4a3611c6c1bd27612d5e10  ./metrics/grafana/tikv_details.json
+607222008939d706ef3ecf391119e5e0fef139dd30d092b429e752902d97f26a  ./metrics/grafana/tikv_details.json
diff --git a/src/server/gc_worker/compaction_filter.rs b/src/server/gc_worker/compaction_filter.rs
index a517825b754..c415b0e5025 100644
--- a/src/server/gc_worker/compaction_filter.rs
+++ b/src/server/gc_worker/compaction_filter.rs
@@ -288,10 +288,12 @@ impl CompactionFilterFactory for WriteCompactionFilterFactory {
 
 pub struct DeleteBatch<B> {
     pub batch: Either<B, Vec<Key>>,
+    pub smallest_key: Option<Key>,
+    pub largest_key: Option<Key>,
 }
 
 impl<B: WriteBatch> DeleteBatch<B> {
-    fn new<EK>(db: &Option<EK>) -> Self
+    pub fn new<EK>(db: &Option<EK>) -> Self
     where
         EK: KvEngine<WriteBatch = B>,
     {
@@ -300,11 +302,34 @@ impl<B: WriteBatch> DeleteBatch<B> {
                 Some(db) => Either::Left(db.write_batch_with_cap(DEFAULT_DELETE_BATCH_SIZE)),
                 None => Either::Right(Vec::with_capacity(64)),
             },
+            smallest_key: None,
+            largest_key: None,
         }
     }
 
     // `key` has prefix `DATA_KEY`.
-    fn delete(&mut self, key: &[u8], ts: TimeStamp) -> Result<(), String> {
+    pub fn delete(&mut self, key: &[u8], ts: TimeStamp) -> Result<(), String> {
+        match &self.smallest_key {
+            Some(smallest) => {
+                debug_assert!(key >= smallest.as_encoded().as_slice());
+            }
+            None => {
+                // Smallest key is the first key because compaction processes keys in asending
+                // order.
+                self.smallest_key = Some(Key::from_encoded_slice(key));
+            }
+        }
+        match &self.largest_key {
+            Some(largest) => {
+                debug_assert!(key >= largest.as_encoded().as_slice());
+                self.largest_key = Some(Key::from_encoded_slice(key));
+            }
+            None => {
+                self.largest_key = Some(Key::from_encoded_slice(key));
+            }
+        }
+        debug_assert_le!(self.smallest_key, self.largest_key);
+
         match &mut self.batch {
             Either::Left(batch) => {
                 let key = Key::from_encoded_slice(key).append_ts(ts);
@@ -555,6 +580,34 @@ impl WriteCompactionFilter {
         if self.write_batch.count() > DEFAULT_DELETE_BATCH_COUNT || force {
             let err = match &mut self.write_batch.batch {
                 Either::Left(wb) => {
+                    // Acquire latch to prevent concurrency with ingestion operations
+                    // using RocksDB IngestExternalFileOptions.allow_write = true.
+                    let _region_inject_latch_guard = match self.engine.as_ref() {
+                        Some(engine) => {
+                            let smallest_key = self
+                                .write_batch
+                                .smallest_key
+                                .as_ref()
+                                .unwrap()
+                                .as_encoded()
+                                .clone();
+                            let largest_key = self
+                                .write_batch
+                                .largest_key
+                                .as_ref()
+                                .unwrap()
+                                .as_encoded()
+                                .clone();
+                            Some(
+                                engine
+                                    .ingest_latch
+                                    .acquire(smallest_key, keys::next_key(&largest_key)),
+                            )
+                        }
+                        None => None,
+                    };
+                    fail_point!("compaction_filter_ingest_latch_acquired_flush");
+
                     let mut wopts = WriteOptions::default();
                     wopts.set_no_slowdown(true);
                     match do_flush(wb, &wopts) {
@@ -699,6 +752,7 @@ impl CompactionFilter for WriteCompactionFilter {
         value: &[u8],
         value_type: CompactionFilterValueType,
     ) -> CompactionFilterDecision {
+        fail_point!("before_compaction_filter");
         if self.encountered_errors {
             // If there are already some errors, do nothing.
             return CompactionFilterDecision::Keep;
@@ -1242,4 +1296,55 @@ pub mod tests {
         gc_runner.safe_point(200).gc(&raw_engine);
         must_get_none(&mut engine, b"zkey", 200);
     }
+
+    #[test]
+    fn test_delete_batch_smallest_and_largest_key_update() {
+        let mut cfg = DbConfig::default();
+        cfg.writecf.disable_auto_compactions = true;
+        cfg.writecf.dynamic_level_bytes = false;
+
+        let dir = tempfile::TempDir::new().unwrap();
+        let builder = TestEngineBuilder::new().path(dir.path());
+        let engine = builder.build_with_cfg(&cfg).unwrap();
+        let raw_engine = engine.get_rocksdb();
+
+        let mut delete_batch = DeleteBatch::new(&Some(raw_engine.clone()));
+
+        let key1 = b"key1";
+        let key2 = b"key2";
+        let key3 = b"key3";
+        let ts1 = TimeStamp::new(100);
+        let ts2 = TimeStamp::new(200);
+        let ts3 = TimeStamp::new(300);
+
+        delete_batch.delete(key1, ts1).unwrap();
+        assert_eq!(
+            delete_batch.smallest_key.as_ref().unwrap().as_encoded(),
+            key1
+        );
+        assert_eq!(
+            delete_batch.largest_key.as_ref().unwrap().as_encoded(),
+            key1
+        );
+
+        delete_batch.delete(key2, ts2).unwrap();
+        assert_eq!(
+            delete_batch.smallest_key.as_ref().unwrap().as_encoded(),
+            key1
+        );
+        assert_eq!(
+            delete_batch.largest_key.as_ref().unwrap().as_encoded(),
+            key2
+        );
+
+        delete_batch.delete(key3, ts3).unwrap();
+        assert_eq!(
+            delete_batch.smallest_key.as_ref().unwrap().as_encoded(),
+            key1
+        );
+        assert_eq!(
+            delete_batch.largest_key.as_ref().unwrap().as_encoded(),
+            key3
+        );
+    }
 }
diff --git a/src/server/gc_worker/gc_worker.rs b/src/server/gc_worker/gc_worker.rs
index 3fff47fab42..b216fd80046 100644
--- a/src/server/gc_worker/gc_worker.rs
+++ b/src/server/gc_worker/gc_worker.rs
@@ -17,8 +17,8 @@ use collections::HashMap;
 use concurrency_manager::ConcurrencyManager;
 use engine_rocks::{FlowInfo, RocksEngine};
 use engine_traits::{
-    raw_ttl::ttl_current_ts, DeleteStrategy, Error as EngineError, KvEngine, MiscExt, Range,
-    WriteBatch, WriteOptions, CF_DEFAULT, CF_LOCK, CF_WRITE,
+    raw_ttl::ttl_current_ts, DeleteStrategy, Error as EngineError, ImportExt, KvEngine, MiscExt,
+    Range, WriteBatch, WriteOptions, CF_DEFAULT, CF_LOCK, CF_WRITE,
 };
 use file_system::{IoType, WithIoType};
 use futures::executor::block_on;
@@ -1054,6 +1054,19 @@ impl<E: Engine> GcRunnerCore<E> {
                 id,
                 region_info_provider,
             } => {
+                let smallest_key = wb.smallest_key.as_ref().unwrap().as_encoded().as_slice();
+                let largest_key = wb.largest_key.as_ref().unwrap().as_encoded().as_slice();
+                // unwrap() is safe as None only occurs in multi-rocksdb version.
+                let kv_engine = self.engine.kv_engine().unwrap();
+                // Acquire latch to prevent concurrency with ingestion operations
+                // using RocksDB IngestExternalFileOptions.allow_write = true.
+                let _ingest_latch_guard = kv_engine.acquire_ingest_latch(Range {
+                    start_key: smallest_key,
+                    end_key: keys::next_key(largest_key).as_slice(),
+                });
+
+                fail_point!("after_gc_orphan_versions_ingest_latch_acquired");
+
                 let count = wb.count();
                 match wb.batch {
                     Either::Left(mut wb) => {
@@ -1570,11 +1583,10 @@ pub mod test_gc_worker {
 
 #[cfg(test)]
 mod tests {
-
     use std::{
         collections::{BTreeMap, BTreeSet},
         path::Path,
-        sync::mpsc,
+        sync::{atomic::AtomicBool, mpsc},
         thread,
         time::Duration,
     };
@@ -2910,4 +2922,128 @@ mod tests {
 
         assert_eq!(gc_worker.get_worker_thread_count(), 2);
     }
+
+    // This test verifies that the GcOrphanVersions task is blocked by concurrent
+    // ingest operations.
+    #[test]
+    fn test_gc_orphan_version_blocked_by_ingest() {
+        let store_id = 1;
+        let engine = TestEngineBuilder::new().build().unwrap();
+        let prefixed_engine = PrefixedEngine(engine.clone());
+        let (tx, _rx) = mpsc::channel();
+        let cfg = GcConfig::default();
+        let coprocessor_host = CoprocessorHost::default();
+        let mut runner = GcRunnerCore::new(
+            store_id,
+            prefixed_engine.clone(),
+            tx,
+            GcWorkerConfigManager(Arc::new(VersionTrack::new(cfg.clone())), None)
+                .0
+                .tracker("gc-worker".to_owned()),
+            cfg,
+            coprocessor_host,
+        );
+
+        let kv_engine = engine.kv_engine().unwrap();
+        let _ingest_latch_guard = kv_engine.acquire_ingest_latch(Range {
+            start_key: b"a",
+            end_key: b"b",
+        });
+
+        let gc_task_completed = Arc::new(AtomicBool::new(false));
+        let gc_task_completed_clone = gc_task_completed.clone();
+        let gc_handle = thread::spawn(move || {
+            let mut wb = DeleteBatch::new(&Some(engine.kv_engine().unwrap()));
+            wb.delete(b"a", 100_u64.into()).unwrap();
+            wb.delete(b"b", 101_u64.into()).unwrap();
+            let region_info_provider = Arc::new(MockRegionInfoProvider::new(vec![]));
+            let task = GcTask::OrphanVersions {
+                wb,
+                region_info_provider,
+                id: 0,
+            };
+            runner.run(task);
+            gc_task_completed_clone.store(true, Ordering::SeqCst);
+        });
+
+        thread::sleep(Duration::from_millis(500));
+        // The GC task must remain pending as _ingest_latch_guard is holding the latch.
+        assert!(
+            !gc_task_completed.load(Ordering::SeqCst),
+            "GC task completed before dropping the guard"
+        );
+        drop(_ingest_latch_guard);
+        gc_handle.join().expect("Gc thread panicked");
+        // The GC task must have finished as _ingest_latch_guard has released the latch.
+        assert!(
+            gc_task_completed.load(Ordering::SeqCst),
+            "GC task did not complete as expected"
+        );
+    }
+
+    // This test verifies that the GcOrphanVersions task blocks the ingest
+    // operation.
+    #[cfg(feature = "failpoints")]
+    #[test]
+    fn test_gc_orphan_version_blocks_ingest() {
+        let store_id = 1;
+        let engine = TestEngineBuilder::new().build().unwrap();
+        let prefixed_engine = PrefixedEngine(engine.clone());
+        let (tx, _rx) = mpsc::channel();
+        let cfg = GcConfig::default();
+        let coprocessor_host = CoprocessorHost::default();
+        let mut runner = GcRunnerCore::new(
+            store_id,
+            prefixed_engine.clone(),
+            tx,
+            GcWorkerConfigManager(Arc::new(VersionTrack::new(cfg.clone())), None)
+                .0
+                .tracker("gc-worker".to_owned()),
+            cfg,
+            coprocessor_host,
+        );
+
+        let engine_clone = engine.clone();
+        let gc_handle = thread::spawn(move || {
+            fail::cfg("after_gc_orphan_versions_ingest_latch_acquired", "pause").unwrap();
+            let mut wb = DeleteBatch::new(&Some(engine_clone.kv_engine().unwrap()));
+            wb.delete(b"a", 100_u64.into()).unwrap();
+            wb.delete(b"b", 101_u64.into()).unwrap();
+            let region_info_provider = Arc::new(MockRegionInfoProvider::new(vec![]));
+            let task = GcTask::OrphanVersions {
+                wb,
+                region_info_provider,
+                id: 0,
+            };
+            runner.run(task);
+        });
+
+        // Wait gc_handle to start.
+        thread::sleep(Duration::from_millis(500));
+        let ingest_task_completed = Arc::new(AtomicBool::new(false));
+        let ingest_task_completed_clone = ingest_task_completed.clone();
+        let ingest_handle = thread::spawn(move || {
+            let kv_engine = engine.kv_engine().unwrap();
+            let _ingest_latch_guard = kv_engine.acquire_ingest_latch(Range {
+                start_key: b"a",
+                end_key: b"b",
+            });
+            ingest_task_completed_clone.store(true, Ordering::SeqCst);
+        });
+        thread::sleep(Duration::from_millis(500));
+        // The ingest task must remain pending as _gc_latch_guard is holding the latch.
+        assert!(
+            !ingest_task_completed.load(Ordering::SeqCst),
+            "Ingest task completed before dropping the GC latch guard"
+        );
+        fail::remove("after_gc_orphan_versions_ingest_latch_acquired");
+        thread::sleep(Duration::from_millis(500));
+        gc_handle.join().expect("Gc thread panicked");
+        ingest_handle.join().expect("Ingest thread panicked");
+        // The ingest task must have finished as _gc_latch_guard has released the latch.
+        assert!(
+            ingest_task_completed.load(Ordering::SeqCst),
+            "Ingest task did not complete as expected"
+        );
+    }
 }


Test patch:
diff --git a/components/test_raftstore/src/server.rs b/components/test_raftstore/src/server.rs
index a023f1541a2..aa852b91fa4 100644
--- a/components/test_raftstore/src/server.rs
+++ b/components/test_raftstore/src/server.rs
@@ -993,7 +993,7 @@ pub fn must_new_and_configure_cluster(
     must_new_and_configure_cluster_mul(1, configure)
 }
 
-fn must_new_and_configure_cluster_mul(
+pub fn must_new_and_configure_cluster_mul(
     count: usize,
     mut configure: impl FnMut(&mut Cluster<ServerCluster>),
 ) -> (Cluster<ServerCluster>, metapb::Peer, Context) {
diff --git a/tests/failpoints/cases/mod.rs b/tests/failpoints/cases/mod.rs
index ca6770b046f..186f733b75a 100644
--- a/tests/failpoints/cases/mod.rs
+++ b/tests/failpoints/cases/mod.rs
@@ -34,6 +34,7 @@ mod test_replica_stale_read;
 mod test_server;
 mod test_snap;
 mod test_split_region;
+mod test_sst_ingest;
 mod test_sst_recovery;
 mod test_stale_peer;
 mod test_stale_read;
diff --git a/tests/failpoints/cases/test_sst_ingest.rs b/tests/failpoints/cases/test_sst_ingest.rs
new file mode 100644
index 00000000000..c8deb951143
--- /dev/null
+++ b/tests/failpoints/cases/test_sst_ingest.rs
@@ -0,0 +1,468 @@
+// Copyright 2020 TiKV Project Authors. Licensed under Apache-2.0.
+
+use std::{
+    sync::{
+        mpsc::{sync_channel, Receiver, RecvTimeoutError},
+        Arc,
+    },
+    thread,
+    thread::JoinHandle,
+    time::Duration,
+};
+
+use engine_traits::MiscExt;
+use grpcio::{ChannelBuilder, Environment};
+use kvproto::{
+    kvrpcpb::{Context, Op},
+    metapb::{Peer, Region},
+    tikvpb::TikvClient,
+};
+use test_pd_client::TestPdClient;
+use test_raftstore::*;
+use tikv::server::gc_worker::TestGcRunner;
+use tikv_util::HandyRwLock;
+
+// Prepares test data for verifying the behavior of the compaction filter.
+//
+// This function creates 3 regions and simulates some key-value writes with
+// timestamps. Specifically, it splits the keyspace into the following regions:
+// - Region A: Covers the range `[-infinite, zb)`, where key `za` has values
+//   written at timestamps 101 and 103.
+// - Region B: Covers the range `[zb, zc)`, where key `zb` has values written at
+//   timestamps 101 and 103.
+// - Region C: Covers the range `[zc, +infinite)`.
+//
+// The function also flushes the data to disk for debugging purposes, allowing
+// tools like `tikv-ctl` to inspect the on-disk data.
+fn prepare_data_used_by_compaction_filter(
+    client: &TikvClient,
+    ctx: &Context,
+    cluster: &mut Cluster<ServerCluster>,
+) {
+    let large_value = vec![b'x'; 300];
+
+    {
+        let keys = vec![b"a".to_vec(), b"b".to_vec()];
+        for start_ts in [101, 103] {
+            let commit_ts = start_ts + 1;
+
+            for pk in &keys {
+                let muts = vec![new_mutation(Op::Put, pk.as_slice(), &large_value)];
+                must_kv_prewrite(client, ctx.clone(), muts, pk.clone(), start_ts);
+            }
+            must_kv_commit(
+                client,
+                ctx.clone(),
+                keys.clone(),
+                start_ts,
+                commit_ts,
+                commit_ts,
+            );
+        }
+    }
+
+    // Region c needs to contain data to trigger the apply-snapshot process.
+    // However, this data will not trigger the compaction filter, resulting in
+    // only one version being created.
+    let pk = b"c";
+    let muts = vec![new_mutation(Op::Put, pk.as_slice(), &large_value)];
+    must_kv_prewrite(client, ctx.clone(), muts, pk.to_vec(), 101);
+    must_kv_commit(client, ctx.clone(), vec![pk.to_vec()], 101, 102, 102);
+
+    let region_a = cluster.get_region(b"a");
+    cluster.must_split(&region_a, "b".as_bytes());
+    let region_b = cluster.get_region(b"b");
+    cluster.must_split(&region_b, "c".as_bytes());
+    let _ = cluster.get_region(b"c");
+    cluster.check_regions_number(3);
+
+    // Flush data for debugging so that `tikv-ctl` can be used to inspect the data.
+    cluster
+        .get_engine(1)
+        .flush_cfs(&["default", "write"], true)
+        .unwrap();
+    cluster
+        .get_engine(2)
+        .flush_cfs(&["default", "write"], true)
+        .unwrap();
+    cluster
+        .get_engine(3)
+        .flush_cfs(&["default", "write"], true)
+        .unwrap();
+}
+
+fn setup_cluster(
+    region_to_migrate: &[u8],
+) -> (Cluster<ServerCluster>, Region, Peer, Arc<TestPdClient>) {
+    let env = Arc::new(Environment::new(1));
+    let (mut cluster, leader, ctx) = must_new_cluster_mul(3);
+    let channel =
+        ChannelBuilder::new(env).connect(&cluster.sim.rl().get_addr(leader.get_store_id()));
+    let client = TikvClient::new(channel);
+    let pd_client = cluster.pd_client.clone();
+    pd_client.disable_default_operator();
+
+    prepare_data_used_by_compaction_filter(&client, &ctx, &mut cluster);
+
+    // Get the region dynamically based on the `region_to_migrate` parameter.
+    let region = cluster.get_region(region_to_migrate);
+    let peer = region.get_peers()[1].clone();
+
+    (cluster, region, peer, pd_client)
+}
+
+fn start_region_migrate(
+    region_id: u64,
+    pd_client: Arc<TestPdClient>,
+    peer: Peer,
+) -> JoinHandle<()> {
+    thread::spawn(move || {
+        // Force SST ingestion instead of using RocksDB writes.
+        fail::cfg("apply_cf_without_ingest_false", "return").unwrap();
+        // Disabled for scenarios that are not relevant.
+        fail::cfg("before_clean_stale_ranges", "return").unwrap();
+
+        pd_client.must_remove_peer(region_id, peer.clone());
+        pd_client.must_add_peer(region_id, peer.clone());
+    })
+}
+
+fn start_compaction_filter(cluster: &Cluster<ServerCluster>, store_id: u64) -> JoinHandle<()> {
+    let gc_engine = cluster.get_engine(store_id);
+    thread::spawn(move || {
+        let mut gc_runner = TestGcRunner::new(200);
+        gc_runner.gc(&gc_engine);
+    })
+}
+
+fn verify_pending(rx: &Receiver<bool>, duration: u64) {
+    assert_eq!(
+        rx.recv_timeout(Duration::from_millis(duration)),
+        Err(RecvTimeoutError::Timeout)
+    );
+}
+
+fn verify_completed(rx: &Receiver<bool>) {
+    assert_eq!(rx.recv_timeout(Duration::from_millis(500)), Ok(true));
+}
+
+// Test whether ingestion blocks the compaction filter based on
+// whether they have overlapping keys.
+//
+// Steps:
+// - Prepare 3 regions with test data and ensure the environment is set up.
+// - Simulate a apply-snapshot process that acquires the range latch and pauses.
+// - Start the compaction filter GC and verify that it cannot acquire the latch
+//   while the apply-snapshot process holds it.
+// - Resume the apply-snapshot ingestion process and verify that the compaction
+//   filter GC can acquire the latch and complete its operation.
+fn blocked_by_ingest_test(region_to_migrate: &[u8], ingest_type: &str) {
+    let (cluster, region, peer, pd_client) = setup_cluster(region_to_migrate);
+    let region_id = region.id;
+
+    // If testing `clean_overlap_ingest`, skip `apply_snapshot`; otherwise, skip
+    // `clean_overlap_ingest`.
+    if ingest_type == "clean_overlap_ingest" {
+        fail::cfg("apply_snap_cleanup_range", "return").unwrap();
+        fail::cfg("manually_set_max_delete_count_by_key", "return").unwrap();
+    } else {
+        fail::cfg("before_clean_overlap_ranges", "return").unwrap();
+    }
+
+    // Start and pause the apply-snapshot process.
+    fail::cfg("after_apply_snapshot_ingest_latch_acquired", "pause").unwrap();
+    let apply_snap_handle = start_region_migrate(region_id, pd_client, peer.clone());
+
+    // Wait for snapshot to acquire the latch.
+    sleep_ms(500);
+
+    let (tx, rx) = sync_channel(0);
+    fail::cfg_callback("compaction_filter_ingest_latch_acquired_flush", move || {
+        // Ignore the return value because the ingest process triggers twice. At the
+        // second trigger, the channel might already be dropped, causing send to fail.
+        // However, this does not affect the test functionality.
+        let _ = tx.send(true);
+    })
+    .unwrap();
+
+    // Start GC, which might be blocked by the apply-snapshot thread, depending
+    // on whether the ranges overlap.
+    let gc_handle = start_compaction_filter(&cluster, peer.store_id);
+
+    if region_to_migrate != b"c" {
+        // Other regions overlap, so the GC thread will be blocked.
+        verify_pending(&rx, 500);
+    } else {
+        // Region c does not overlap with the snapshot process.
+        verify_completed(&rx);
+        fail::remove("after_apply_snapshot_ingest_latch_acquired");
+        gc_handle.join().expect("GC thread panicked");
+        apply_snap_handle
+            .join()
+            .expect("apply snapshot thread panicked");
+        fail::remove("compaction_filter_ingest_latch_acquired_flush");
+        return;
+    }
+
+    // Resume the snapshot process.
+    fail::remove("after_apply_snapshot_ingest_latch_acquired");
+    // Wait for the snapshot process to release the latch.
+    sleep_ms(500);
+    // The GC thread can continue after the snapshot process releases the latch.
+    verify_completed(&rx);
+    gc_handle.join().expect("GC thread panicked");
+    apply_snap_handle
+        .join()
+        .expect("apply snapshot thread panicked");
+    fail::remove("compaction_filter_ingest_latch_acquired_flush");
+}
+
+// Test Region A: Verify that the compaction filter is blocked because it
+// overlaps with the region's data.
+#[test]
+fn test_compaction_filter_blocked_by_snapshot_ingest_basic() {
+    blocked_by_ingest_test(b"a", "snapshot_ingest");
+}
+
+// Test Region B: Verify that the compaction filter is blocked because its
+// `end_key` matches the `Region.start_key` and overlaps with this region's
+// data.
+#[test]
+fn test_compaction_filter_blocked_by_snapshot_ingest_boundary() {
+    blocked_by_ingest_test(b"b", "snapshot_ingest");
+}
+
+// Test Region C: Verify that the compaction filter is not blocked because it
+// does not overlap with this region's data.
+#[test]
+fn test_compaction_filter_not_blocked_by_snapshot_ingest() {
+    blocked_by_ingest_test(b"c", "snapshot_ingest");
+}
+
+// Similar to `test_compaction_filter_blocked_by_snapshot_ingest_basic`, but
+// tests the `clean_overlap_range`.
+#[test]
+fn test_compaction_filter_blocked_by_clean_overlap_ingest_basic() {
+    blocked_by_ingest_test(b"a", "clean_overlap_ingest");
+}
+
+// Similar to `test_compaction_filter_blocked_by_snapshot_ingest_boundary`, but
+// tests the `clean_overlap_range`.
+#[test]
+fn test_compaction_filter_blocked_by_clean_overlap_ingest_boundary() {
+    blocked_by_ingest_test(b"b", "clean_overlap_ingest");
+}
+
+// Similar to `test_compaction_filter_not_blocked_by_snapshot_ingest`, but
+// tests the `clean_overlap_range`.
+#[test]
+fn test_compaction_filter_not_blocked_by_clean_overlap_ingest() {
+    blocked_by_ingest_test(b"c", "clean_overlap_ingest");
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// The following tests is similar to above tests, but tests the reverse scenario
+// where compaction filter blocks ingestion.
+///////////////////////////////////////////////////////////////////////////////
+
+fn blocks_ingest_test(region_to_migrate: &[u8], ingest_type: &str) {
+    let (cluster, region, peer, pd_client) = setup_cluster(region_to_migrate);
+
+    // Start and pause the GC thread.
+    fail::cfg("compaction_filter_ingest_latch_acquired_flush", "pause").unwrap();
+    let gc_handle = start_compaction_filter(&cluster, peer.store_id);
+
+    let (tx, rx) = sync_channel(0);
+    fail::cfg_callback("after_apply_snapshot_ingest_latch_acquired", move || {
+        // Ignore the return value because the ingest process triggers twice. At the
+        // second trigger, the channel might already be dropped, causing send to fail.
+        // However, this does not affect the test functionality.
+        let _ = tx.send(true);
+    })
+    .unwrap();
+
+    let region_id = region.id;
+
+    // If testing `clean_overlap_ingest`, skip `apply_snapshot`; otherwise, skip
+    // `clean_overlap_ingest`.
+    if ingest_type == "clean_overlap_ingest" {
+        fail::cfg("apply_snap_cleanup_range", "return").unwrap();
+        fail::cfg("manually_set_max_delete_count_by_key", "return").unwrap();
+    } else {
+        fail::cfg("before_clean_overlap_ranges", "return").unwrap();
+    }
+    // Start the apply-snapshot thread, which might be blocked by the GC thread,
+    // depending on whether the ranges overlap.
+    let apply_snap_handle = start_region_migrate(region_id, pd_client, peer.clone());
+    // Wait for snapshot to acquire the latch.
+    sleep_ms(500);
+    if region_to_migrate != b"c" {
+        // Other regions overlap, so the snapshot process will be blocked.
+        verify_pending(&rx, 500);
+    } else {
+        // Region c does not overlap with the GC thread.
+        verify_completed(&rx);
+        fail::remove("compaction_filter_ingest_latch_acquired_flush");
+        gc_handle.join().expect("GC thread panicked");
+        apply_snap_handle
+            .join()
+            .expect("apply-snapshot thread panicked");
+        fail::remove("after_apply_snapshot_ingest_latch_acquired");
+        return;
+    }
+
+    // Resume the compaction filter process.
+    fail::remove("compaction_filter_ingest_latch_acquired_flush");
+    // Wait for the GC thread to release the latch.
+    sleep_ms(500);
+    // The apply-snapshot thread can continue after the GC thread releases the
+    // latch.
+    verify_completed(&rx);
+    gc_handle.join().expect("GC thread panicked");
+    apply_snap_handle
+        .join()
+        .expect("apply snapshot thread panicked");
+    fail::remove("after_apply_snapshot_ingest_latch_acquired");
+}
+
+#[test]
+fn test_compaction_filter_blocks_snapshot_ingest_basic() {
+    blocks_ingest_test(b"a", "snapshot_ingest");
+}
+
+#[test]
+fn test_compaction_filter_blocks_snapshot_ingest_boundary() {
+    blocks_ingest_test(b"b", "snapshot_ingest");
+}
+
+#[test]
+fn test_compaction_filter_not_blocks_snapshot_ingest() {
+    blocks_ingest_test(b"c", "snapshot_ingest");
+}
+
+#[test]
+fn test_compaction_filter_blocks_clean_overlap_ingest_basic() {
+    blocks_ingest_test(b"a", "clean_overlap_ingest");
+}
+
+#[test]
+fn test_compaction_filter_blocks_clean_overlap_ingest_boundary() {
+    blocks_ingest_test(b"b", "clean_overlap_ingest");
+}
+
+#[test]
+fn test_compaction_filter_not_blocks_clean_overlap_ingest() {
+    blocks_ingest_test(b"c", "clean_overlap_ingest");
+}
+
+// This test verifies that when a region is migrated out and then migrated back,
+// the destroy-peer task must complete before the apply-snapshot task.
+//
+// Currently, since the region worker operates on a single thread, when a
+// region is migrated out and back again, the destroy-peer task is enqueued
+// before the apply-snapshot task, ensuring sequential execution without
+// concurrency.
+//
+// As the apply-snapshot ingestion uses RocksDB
+// IngestExternalFileOption.allow_write = true, concurrent writes are not
+// expected during apply-snapshot ingestion. Therefore, this test ensures that
+// there are no concurrent writes from destroy-peer during this process.
+#[test]
+fn test_apply_snapshot_must_wait_destroy_peer() {
+    let (mut cluster, ..) = must_new_cluster_mul(3);
+
+    cluster.must_put(b"k1", b"v1");
+    let peer = new_peer(2, 2);
+    let pk = b"k1".to_vec();
+    let region = cluster.get_region(&pk);
+    let region_id = region.get_id();
+    let pd_client = cluster.pd_client.clone();
+    pd_client.disable_default_operator();
+
+    // Start the destroy-peer process, which will pause at
+    // before_clean_stale_ranges.
+    let pd_client_clone1 = pd_client.clone();
+    let peer_clone1 = peer.clone();
+    let destroy_handle = thread::spawn(move || {
+        fail::cfg("before_clean_stale_ranges", "pause").unwrap();
+        pd_client_clone1.must_remove_peer(region_id, peer_clone1);
+    });
+    // Wait for the destroy-peer process to pause.
+    sleep_ms(500);
+
+    let (tx, rx) = sync_channel(0);
+    fail::cfg_callback("apply_snapshot_finished", move || {
+        tx.send(true).unwrap();
+    })
+    .unwrap();
+
+    let pd_client_clone2 = pd_client.clone();
+    let peer_clone2 = peer.clone();
+    let apply_handle = thread::spawn(move || {
+        pd_client_clone2.must_add_peer(region_id, peer_clone2);
+    });
+
+    // Verify that apply-snapshot is pending.
+    assert_eq!(
+        rx.recv_timeout(Duration::from_millis(500)),
+        Err(RecvTimeoutError::Timeout)
+    );
+
+    // Resume the destroy-peer process.
+    fail::remove("before_clean_stale_ranges");
+    // Verify that apply-snapshot is finished
+    assert_eq!(rx.recv_timeout(Duration::from_millis(1000)), Ok(true));
+    destroy_handle.join().expect("destroy handle panicked");
+    apply_handle.join().expect("apply handle panicked");
+}
+
+// This test ensures that the destroy-peer process will wait for the ongoing
+// foreground writes of the same region to finish.
+//
+// Since test_apply_snapshot_must_wait_destroy_peer() has already verified that
+// when a region is migrated out and then back, the apply-snapshot process will
+// wait for the destroy-peer process to complete first, it can be considered
+// that apply-snapshot will also wait for the ongoing foreground writes to
+// finish.
+#[test]
+fn test_destroy_peer_must_wait_ongoing_foreground_writes() {
+    let (mut cluster, ..) = must_new_cluster_mul(3);
+    // Use peer_id 3 since on_apply_write_cmd only works on peer 3.
+    let peer = new_peer(3, 3);
+    let pk = b"k1".to_vec();
+    let region = cluster.get_region(&pk);
+    let region_id = region.get_id();
+    let pd_client = cluster.pd_client.clone();
+    pd_client.disable_default_operator();
+
+    fail::cfg("on_apply_write_cmd", "pause").unwrap();
+    cluster.put(b"k1", b"v1").unwrap();
+    // Wait for foreground writes to pause.
+    sleep_ms(500);
+
+    let (tx, rx) = sync_channel(0);
+    fail::cfg_callback("raft_store_after_destroy_peer", move || {
+        tx.send(true).unwrap();
+    })
+    .unwrap();
+
+    // Start the destroy-peer process, which will pause at
+    // before_clean_stale_ranges.
+    let pd_client_clone1 = pd_client.clone();
+    let peer_clone1 = peer.clone();
+    let destroy_handle = thread::spawn(move || {
+        pd_client_clone1.must_remove_peer(region_id, peer_clone1);
+    });
+
+    // Verify that apply-snapshot is pending.
+    assert_eq!(
+        rx.recv_timeout(Duration::from_millis(500)),
+        Err(RecvTimeoutError::Timeout)
+    );
+
+    // Resume the foreground writes.
+    fail::remove("on_apply_write_cmd");
+    // Verify that apply-snapshot is finished
+    assert_eq!(rx.recv_timeout(Duration::from_millis(500)), Ok(true));
+    destroy_handle.join().expect("destroy thread panicked");
+}
diff --git a/tests/integrations/storage/test_titan.rs b/tests/integrations/storage/test_titan.rs
index a99f0383c81..5e6ed39b3f8 100644
--- a/tests/integrations/storage/test_titan.rs
+++ b/tests/integrations/storage/test_titan.rs
@@ -413,13 +413,14 @@ fn test_delete_files_in_range_for_titan() {
         .iter()
         .map(|s| s.as_str())
         .collect::<Vec<&str>>();
-    apply_sst_cf_files_by_ingest(&tmp_file_paths, &engines1.kv, CF_DEFAULT).unwrap();
+    apply_sst_cf_files_by_ingest(&tmp_file_paths, &engines1.kv, CF_DEFAULT, vec![], vec![])
+        .unwrap();
     let tmp_file_paths = cf_file_write.tmp_file_paths();
     let tmp_file_paths = tmp_file_paths
         .iter()
         .map(|s| s.as_str())
         .collect::<Vec<&str>>();
-    apply_sst_cf_files_by_ingest(&tmp_file_paths, &engines1.kv, CF_WRITE).unwrap();
+    apply_sst_cf_files_by_ingest(&tmp_file_paths, &engines1.kv, CF_WRITE, vec![], vec![]).unwrap();
 
     // Do scan on other DB.
     let mut r = Region::default();