Repository: lancedb/lance
Pull request #3847: feat!: introduce fragment reuse index to defer compaction index remap
Author: login: jackye1995
Labels: name: enhancement
description: New feature or request
name: python
name: breaking-change
Created: 2025-05-19T06:29:23Z | Merged: 2025-05-29T23:28:50Z

BREAKING CHANGE: Signature of `dataset::optimize::commit_compaction` has renamed `options` as `remap_options`, and takes additional `CompactionOptions` as `options`. The `dataset::transaction::Operation::Rewrite` adds a new field `frag_reuse_index: Option<Index>` that holds optional fragment reuse index metadata if present.

This PR introduces a new index, fragment reuse index, to store compaction row address information after compactions. The index will be created when compaction has option defer_index_remap enabled.

The index is named as "fragment reuse" since the ultimate goal is to allow query to reuse old indexed fragments in replacement of new unindexed fragments for index queries. An alternative was to name it a "remap index", but that name was too confusing as the term "remap" is also a verb with functions like `remap_index` already in the codebase.

Closes #3833 #3834 


Linked issue #3833: [Remap Separation] Define an index structure for storing row ID map after a compaction

Diff:
diff --git a/protos/table.proto b/protos/table.proto
index 0885f88598e..91bfcd48e84 100644
--- a/protos/table.proto
+++ b/protos/table.proto
@@ -369,4 +369,35 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  oneof content {
+    // if < 200KB, store the content inline, otherwise store the InlineContent bytes in external file
+    InlineContent inline = 1;
+    ExternalFile external = 2;
+  }
+
+  message InlineContent {
+    repeated Version versions = 1;
+  }
+
+  message Version {
+    // The dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
+
+    // A roaring treemap of the changed row addresses.
+    // When combined with the old fragment IDs and new fragment IDs,
+    // it can recover the full mapping of old row addresses to either new row addresses or deleted.
+    // this mapping can then be used to remap indexes or satisfy index queries for the new unindexed fragments.
+    bytes changed_row_addrs = 2;
+
+    // Store the full content of old fragments to be used for indexed queries
+    // This should be exactly in the order that a rewrite_files takes in the old fragments
+    repeated DataFragment old_fragments = 3;
+
+    // This must be exactly in the order that a rewrite_files produces the new fragments
+    repeated uint64 new_fragments = 4;
+  }
+}
\ No newline at end of file
diff --git a/python/src/dataset/optimize.rs b/python/src/dataset/optimize.rs
index 1fc536194f1..a4d64d23b78 100644
--- a/python/src/dataset/optimize.rs
+++ b/python/src/dataset/optimize.rs
@@ -545,10 +545,13 @@ impl PyCompaction {
         let dataset = Python::with_gil(|py| dataset_ref.borrow(py).clone());
         let rewrites: Vec<RewriteResult> = rewrites.into_iter().map(|r| r.0).collect();
         let mut new_ds = dataset.ds.as_ref().clone();
+        // TODO: pass compaction option from plan and execute time
+        let options: CompactionOptions = CompactionOptions::default();
         let fut = commit_compaction(
             &mut new_ds,
             rewrites,
             Arc::new(DatasetIndexRemapperOptions::default()),
+            &options,
         );
         let metrics = RT
             .block_on(None, fut)?
diff --git a/python/src/transaction.rs b/python/src/transaction.rs
index cb9ac0fbe65..a23bf855af4 100644
--- a/python/src/transaction.rs
+++ b/python/src/transaction.rs
@@ -118,6 +118,8 @@ impl FromPyObject<'_> for PyLance<Operation> {
                 let op = Operation::Rewrite {
                     groups,
                     rewritten_indices,
+                    // TODO: pass frag_reuse_index when available
+                    frag_reuse_index: None,
                 };
                 Ok(Self(op))
             }
diff --git a/rust/lance-core/src/utils/tracing.rs b/rust/lance-core/src/utils/tracing.rs
index f7b581badb7..ec59e9a3396 100644
--- a/rust/lance-core/src/utils/tracing.rs
+++ b/rust/lance-core/src/utils/tracing.rs
@@ -70,6 +70,7 @@ pub const TRACE_FILE_CREATE: &str = "create";
 pub const TRACE_IO_EVENTS: &str = "lance::io_events";
 pub const IO_TYPE_OPEN_SCALAR: &str = "open_scalar_index";
 pub const IO_TYPE_OPEN_VECTOR: &str = "open_vector_index";
+pub const IO_TYPE_OPEN_FRAG_REUSE: &str = "open_frag_reuse_index";
 pub const IO_TYPE_LOAD_VECTOR_PART: &str = "load_vector_part";
 pub const IO_TYPE_LOAD_SCALAR_PART: &str = "load_scalar_part";
 pub const TRACE_EXECUTION: &str = "lance::execution";
diff --git a/rust/lance-index/src/frag_reuse.rs b/rust/lance-index/src/frag_reuse.rs
new file mode 100644
index 00000000000..2464c88232f
--- /dev/null
+++ b/rust/lance-index/src/frag_reuse.rs
@@ -0,0 +1,227 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::{Index, IndexType};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use itertools::Itertools;
+use lance_core::{Error, Result};
+use lance_table::format::pb::fragment_reuse_index_details::InlineContent;
+use lance_table::format::{pb, ExternalFile, Fragment};
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+
+pub const FRAG_REUSE_INDEX_NAME: &str = "__lance_frag_reuse";
+pub const FRAG_REUSE_DETAILS_FILE_NAME: &str = "details.binpb";
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseVersion {
+    pub dataset_version: u64,
+    pub changed_row_addrs: Vec<u8>,
+    pub old_frags: Vec<Fragment>,
+    pub new_frags: Vec<u64>,
+}
+
+impl From<&FragReuseVersion> for pb::fragment_reuse_index_details::Version {
+    fn from(version: &FragReuseVersion) -> Self {
+        Self {
+            dataset_version: version.dataset_version,
+            changed_row_addrs: version.changed_row_addrs.clone(),
+            old_fragments: version
+                .old_frags
+                .iter()
+                .map(pb::DataFragment::from)
+                .collect::<Vec<_>>(),
+            new_fragments: version.new_frags.clone(),
+        }
+    }
+}
+
+impl TryFrom<pb::fragment_reuse_index_details::Version> for FragReuseVersion {
+    type Error = Error;
+
+    fn try_from(version: pb::fragment_reuse_index_details::Version) -> Result<Self> {
+        Ok(Self {
+            dataset_version: version.dataset_version,
+            changed_row_addrs: version.changed_row_addrs,
+            old_frags: version
+                .old_fragments
+                .into_iter()
+                .map(Fragment::try_from)
+                .collect::<Result<_>>()?,
+            new_frags: version.new_fragments,
+        })
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub enum FragReuseIndexDetailsContentType {
+    Inline(FragReuseIndexDetails),
+    External(ExternalFile),
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseIndexDetails {
+    pub versions: Vec<FragReuseVersion>,
+}
+
+impl From<&FragReuseIndexDetails> for InlineContent {
+    fn from(details: &FragReuseIndexDetails) -> Self {
+        Self {
+            versions: details
+                .versions
+                .iter()
+                .map(|m| m.into())
+                // sort from oldest to latest version
+                .sorted_by_key(|v: &pb::fragment_reuse_index_details::Version| v.dataset_version)
+                .collect(),
+        }
+    }
+}
+
+impl TryFrom<InlineContent> for FragReuseIndexDetails {
+    type Error = Error;
+
+    fn try_from(content: InlineContent) -> Result<Self> {
+        Ok(Self {
+            versions: content
+                .versions
+                .into_iter()
+                .map(|m| m.try_into())
+                .collect::<Result<Vec<_>>>()?,
+        })
+    }
+}
+
+/// An index that stores row ID maps.
+/// A row ID map describes the mapping from old row address to new address after compactions.
+/// Each version contains the mapping for one round of compaction.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseIndex {
+    pub row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+    pub details: FragReuseIndexDetails,
+}
+
+impl FragReuseIndex {
+    pub fn new(
+        row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+        details: FragReuseIndexDetails,
+    ) -> Self {
+        Self {
+            row_id_maps,
+            details,
+        }
+    }
+}
+
+#[derive(Serialize)]
+struct FragReuseStatistics {
+    num_versions: usize,
+}
+
+#[async_trait]
+impl Index for FragReuseIndex {
+    fn as_any(&self) -> &dyn Any {
+        self
+    }
+
+    fn as_index(self: Arc<Self>) -> Arc<dyn Index> {
+        self
+    }
+
+    fn as_vector_index(self: Arc<Self>) -> Result<Arc<dyn crate::vector::VectorIndex>> {
+        Err(Error::NotSupported {
+            source: "FragReuseIndex is not a vector index".into(),
+            location: location!(),
+        })
+    }
+
+    fn statistics(&self) -> Result<serde_json::Value> {
+        let stats = FragReuseStatistics {
+            num_versions: self.details.versions.len(),
+        };
+        serde_json::to_value(stats).map_err(|e| Error::Internal {
+            message: format!("failed to serialize fragment reuse index statistics: {}", e),
+            location: location!(),
+        })
+    }
+
+    async fn prewarm(&self) -> Result<()> {
+        Ok(())
+    }
+
+    fn index_type(&self) -> IndexType {
+        IndexType::FragmentReuse
+    }
+
+    async fn calculate_included_frags(&self) -> Result<RoaringBitmap> {
+        unimplemented!()
+    }
+}
+
+#[cfg(test)]
+pub mod tests {
+
+    use super::*;
+
+    #[tokio::test]
+    async fn test_serialize_deserialize_index_details() {
+        // Create sample FragReuseVersions with different dataset versions
+        let version1 = FragReuseVersion {
+            dataset_version: 2,
+            changed_row_addrs: vec![1, 2, 3],
+            old_frags: vec![Fragment {
+                id: 1,
+                files: vec![],
+                deletion_file: None,
+                row_id_meta: None,
+                physical_rows: None,
+            }],
+            new_frags: vec![2, 3],
+        };
+
+        let version2 = FragReuseVersion {
+            dataset_version: 1,
+            changed_row_addrs: vec![4, 5, 6],
+            old_frags: vec![Fragment {
+                id: 2,
+                files: vec![],
+                deletion_file: None,
+                row_id_meta: None,
+                physical_rows: None,
+            }],
+            new_frags: vec![4, 5],
+        };
+
+        // Create FragReuseIndexDetails with versions in reverse order
+        let details = FragReuseIndexDetails {
+            versions: vec![version1, version2],
+        };
+
+        // Convert to protobuf format
+        let inline_content: InlineContent = (&details).into();
+
+        // Convert back to FragReuseIndexDetails
+        let roundtrip_details = FragReuseIndexDetails::try_from(inline_content).unwrap();
+
+        // Verify the roundtrip
+        assert_eq!(roundtrip_details.versions.len(), 2);
+
+        // Verify versions are sorted by dataset_version (oldest to latest)
+        assert_eq!(roundtrip_details.versions[0].dataset_version, 1);
+        assert_eq!(
+            roundtrip_details.versions[0].changed_row_addrs,
+            vec![4, 5, 6]
+        );
+        assert_eq!(roundtrip_details.versions[0].new_frags, vec![4, 5]);
+
+        assert_eq!(roundtrip_details.versions[1].dataset_version, 2);
+        assert_eq!(
+            roundtrip_details.versions[1].changed_row_addrs,
+            vec![1, 2, 3]
+        );
+        assert_eq!(roundtrip_details.versions[1].new_frags, vec![2, 3]);
+    }
+}
diff --git a/rust/lance-index/src/lib.rs b/rust/lance-index/src/lib.rs
index d8725848ede..298a7b4e2e8 100644
--- a/rust/lance-index/src/lib.rs
+++ b/rust/lance-index/src/lib.rs
@@ -19,12 +19,14 @@ use serde::{Deserialize, Serialize};
 use snafu::location;
 use std::convert::TryFrom;
 
+pub mod frag_reuse;
 pub mod metrics;
 pub mod optimize;
 pub mod prefilter;
 pub mod scalar;
 pub mod traits;
 pub mod vector;
+
 pub use crate::traits::*;
 
 pub const INDEX_FILE_NAME: &str = "index.idx";
@@ -87,6 +89,8 @@ pub enum IndexType {
 
     NGram = 5, // NGram
 
+    FragmentReuse = 6,
+
     // 100+ and up for vector index.
     /// Flat vector index.
     Vector = 100, // Legacy vector index, alias to IvfPq
@@ -105,6 +109,7 @@ impl std::fmt::Display for IndexType {
             Self::LabelList => write!(f, "LabelList"),
             Self::Inverted => write!(f, "Inverted"),
             Self::NGram => write!(f, "NGram"),
+            Self::FragmentReuse => write!(f, "FragmentReuse"),
             Self::Vector | Self::IvfPq => write!(f, "IVF_PQ"),
             Self::IvfFlat => write!(f, "IVF_FLAT"),
             Self::IvfSq => write!(f, "IVF_SQ"),
diff --git a/rust/lance/src/dataset/optimize.rs b/rust/lance/src/dataset/optimize.rs
index a493cd88312..efe767f15bd 100644
--- a/rust/lance/src/dataset/optimize.rs
+++ b/rust/lance/src/dataset/optimize.rs
@@ -82,6 +82,7 @@
 //! they can be committed in any order.
 use std::borrow::Cow;
 use std::collections::HashMap;
+use std::io::Cursor;
 use std::ops::{AddAssign, Range};
 use std::sync::{Arc, RwLock};
 
@@ -105,8 +106,9 @@ use super::transaction::{Operation, RewriteGroup, RewrittenIndex, Transaction};
 use super::utils::make_rowid_capture_stream;
 use super::{write_fragments_internal, WriteMode, WriteParams};
 
-mod remapping;
+pub mod remapping;
 
+use crate::index::frag_reuse::build_new_frag_reuse_index;
 use crate::io::deletion::read_dataset_deletion_file;
 pub use remapping::{IgnoreRemap, IndexRemapper, IndexRemapperOptions, RemappedIndex};
 
@@ -147,6 +149,10 @@ pub struct CompactionOptions {
     /// specified then the default (see
     /// [`crate::dataset::Scanner::batch_size`]) will be used.
     pub batch_size: Option<usize>,
+    /// Whether to defer remapping indices during compaction. If true, indices will
+    /// not be remapped during this compaction operation. Instead, the fragment reuse index
+    /// is updated and will be used to perform remapping later.
+    pub defer_index_remap: bool,
 }
 
 impl Default for CompactionOptions {
@@ -160,6 +166,7 @@ impl Default for CompactionOptions {
             num_threads: None,
             max_bytes_per_file: None,
             batch_size: None,
+            defer_index_remap: false,
         }
     }
 }
@@ -232,7 +239,7 @@ pub async fn compact_files(
 
     let completed_tasks: Vec<RewriteResult> = result_stream.try_collect().await?;
     let remap_options = remap_options.unwrap_or(Arc::new(DatasetIndexRemapperOptions::default()));
-    let metrics = commit_compaction(dataset, completed_tasks, remap_options).await?;
+    let metrics = commit_compaction(dataset, completed_tasks, remap_options, &options).await?;
 
     Ok(metrics)
 }
@@ -581,7 +588,13 @@ pub struct RewriteResult {
     pub read_version: u64,
     /// The original fragments being replaced
     pub original_fragments: Vec<Fragment>,
-    pub row_id_map: HashMap<u64, Option<u64>>,
+    /// A HashMap of original row IDs to new row IDs or None (deleted)
+    /// Only set when index remap is done as a part of the compaction
+    pub row_id_map: Option<HashMap<u64, Option<u64>>>,
+    /// the changed row addresses in the original fragment
+    /// in the form of serialized RoaringTreemap
+    /// Only set when index remap is deferred after compaction
+    pub changed_row_addrs: Option<Vec<u8>>,
 }
 
 async fn reserve_fragment_ids(
@@ -636,7 +649,8 @@ async fn rewrite_files(
             new_fragments: Vec::new(),
             read_version: dataset.manifest.version,
             original_fragments: task.fragments,
-            row_id_map: HashMap::new(),
+            row_id_map: None,
+            changed_row_addrs: None,
         });
     }
 
@@ -726,7 +740,7 @@ async fn rewrite_files(
 
     log::info!("Compaction task {}: file written", task_id);
 
-    let row_id_map = if let Some(row_ids) = row_ids {
+    let (row_id_map, changed_row_addrs) = if let Some(row_ids) = row_ids {
         let row_ids = Arc::try_unwrap(row_ids)
             .expect("Row ids lock still owned")
             .into_inner()
@@ -738,12 +752,26 @@ async fn rewrite_files(
         );
         reserve_fragment_ids(&dataset, new_fragments.iter_mut()).await?;
 
-        remapping::transpose_row_ids(row_ids, &fragments, &new_fragments)
+        if options.defer_index_remap {
+            let mut changed_row_addrs = Vec::with_capacity(row_ids.serialized_size());
+            row_ids.serialize_into(&mut changed_row_addrs)?;
+            (None, Some(changed_row_addrs))
+        } else {
+            let row_id_map = remapping::transpose_row_ids(row_ids, &fragments, &new_fragments);
+            (Some(row_id_map), None)
+        }
     } else {
         log::info!("Compaction task {}: rechunking stable row ids", task_id);
         rechunk_stable_row_ids(dataset.as_ref(), &mut new_fragments, &fragments).await?;
 
-        HashMap::new()
+        if options.defer_index_remap {
+            let no_addrs = RoaringTreemap::new();
+            let mut serialized_no_addrs = Vec::with_capacity(no_addrs.serialized_size());
+            no_addrs.serialize_into(&mut serialized_no_addrs)?;
+            (None, Some(serialized_no_addrs))
+        } else {
+            (Some(HashMap::new()), None)
+        }
     };
 
     metrics.files_removed = task
@@ -766,6 +794,7 @@ async fn rewrite_files(
         read_version: dataset.manifest.version,
         original_fragments: task.fragments,
         row_id_map,
+        changed_row_addrs,
     })
 }
 
@@ -839,34 +868,50 @@ async fn rechunk_stable_row_ids(
 pub async fn commit_compaction(
     dataset: &mut Dataset,
     completed_tasks: Vec<RewriteResult>,
-    options: Arc<dyn IndexRemapperOptions>,
+    remap_options: Arc<dyn IndexRemapperOptions>,
+    options: &CompactionOptions,
 ) -> Result<CompactionMetrics> {
     if completed_tasks.is_empty() {
         return Ok(CompactionMetrics::default());
     }
 
     // If we aren't using move-stable row ids, then we need to remap indices.
-    let needs_remapping = !dataset.manifest.uses_move_stable_row_ids();
+    let needs_remapping =
+        !dataset.manifest.uses_move_stable_row_ids() && !options.defer_index_remap;
 
     let mut rewrite_groups = Vec::with_capacity(completed_tasks.len());
     let mut metrics = CompactionMetrics::default();
 
     let mut row_id_map: HashMap<u64, Option<u64>> = HashMap::default();
+    let mut changed_row_addrs: RoaringTreemap = RoaringTreemap::new();
+    let mut old_fragments: Vec<Fragment> = Vec::new();
+    let mut new_fragment_ids: Vec<u64> = Vec::new();
+    let mut new_fragment_bitmap: RoaringBitmap = RoaringBitmap::new();
 
     for task in completed_tasks {
         metrics += task.metrics;
         let rewrite_group = RewriteGroup {
-            old_fragments: task.original_fragments,
-            new_fragments: task.new_fragments,
+            old_fragments: task.original_fragments.clone(),
+            new_fragments: task.new_fragments.clone(),
         };
         if needs_remapping {
-            row_id_map.extend(task.row_id_map);
+            row_id_map.extend(task.row_id_map.unwrap());
+        } else if options.defer_index_remap {
+            let task_changed_row_addr_bytes = task.changed_row_addrs.unwrap();
+            let mut cursor = Cursor::new(&task_changed_row_addr_bytes);
+            let task_changed_row_addrs = RoaringTreemap::deserialize_from(&mut cursor)?;
+            changed_row_addrs |= task_changed_row_addrs;
+            old_fragments.extend(task.original_fragments.clone());
+            task.new_fragments.iter().for_each(|frag| {
+                new_fragment_ids.push(frag.id);
+                new_fragment_bitmap.insert(frag.id as u32);
+            });
         }
         rewrite_groups.push(rewrite_group);
     }
 
     let rewritten_indices = if needs_remapping {
-        let index_remapper = options.create_remapper(dataset)?;
+        let index_remapper = remap_options.create_remapper(dataset)?;
         let affected_ids = rewrite_groups
             .iter()
             .flat_map(|group| group.old_fragments.iter().map(|frag| frag.id))
@@ -893,11 +938,27 @@ pub async fn commit_compaction(
         Vec::new()
     };
 
+    let frag_reuse_index = if options.defer_index_remap {
+        Some(
+            build_new_frag_reuse_index(
+                dataset,
+                old_fragments,
+                new_fragment_ids,
+                new_fragment_bitmap,
+                changed_row_addrs,
+            )
+            .await?,
+        )
+    } else {
+        None
+    };
+
     let transaction = Transaction::new(
         dataset.manifest.version,
         Operation::Rewrite {
             groups: rewrite_groups,
             rewritten_indices,
+            frag_reuse_index,
         },
         // TODO: Add a blob compaction pass
         /*blob_op= */ None,
@@ -930,7 +991,10 @@ mod tests {
     use tempfile::tempdir;
     use uuid::Uuid;
 
+    use crate::dataset::optimize::remapping::transpose_row_ids;
+    use crate::index::frag_reuse::{load_frag_reuse_index_details, open_frag_reuse_index};
     use crate::index::vector::VectorIndexParams;
+    use lance_index::frag_reuse::FRAG_REUSE_INDEX_NAME;
 
     use self::remapping::RemappedIndex;
 
@@ -1567,6 +1631,7 @@ mod tests {
             &mut dataset,
             vec![results.pop().unwrap()],
             Arc::new(IgnoreRemap::default()),
+            &options,
         )
         .await
         .unwrap();
@@ -1582,9 +1647,14 @@ mod tests {
         }
 
         // Can commit the remaining tasks
-        commit_compaction(&mut dataset, results, Arc::new(IgnoreRemap::default()))
-            .await
-            .unwrap();
+        commit_compaction(
+            &mut dataset,
+            results,
+            Arc::new(IgnoreRemap::default()),
+            &options,
+        )
+        .await
+        .unwrap();
         if use_stable_row_id {
             // 1 commit for reserve fragments and 1 for final commit, both
             // from the call to commit_compaction
@@ -1700,4 +1770,303 @@ mod tests {
         let after_scalar_result = scalar_query(&dataset).await;
         assert_eq!(before_scalar_result, after_scalar_result);
     }
+
+    #[tokio::test]
+    async fn test_defer_index_remap() {
+        let mut data_gen = BatchGenerator::new()
+            .col(Box::new(
+                RandomVector::new().vec_width(128).named("vec".to_owned()),
+            ))
+            .col(Box::new(IncrementingInt32::new().named("i".to_owned())));
+
+        let mut dataset = Dataset::write(
+            data_gen.batch(6_000),
+            "memory://test/table",
+            Some(WriteParams {
+                max_rows_per_file: 1_000, // 6 files
+                ..Default::default()
+            }),
+        )
+        .await
+        .unwrap();
+
+        // Create another same dataset to mimic behavior without deferred index remap
+        let mut data_gen2 = BatchGenerator::new()
+            .col(Box::new(
+                RandomVector::new().vec_width(128).named("vec".to_owned()),
+            ))
+            .col(Box::new(IncrementingInt32::new().named("i".to_owned())));
+
+        let mut dataset2 = Dataset::write(
+            data_gen2.batch(6_000),
+            "memory://test/table",
+            Some(WriteParams {
+                max_rows_per_file: 1_000, // 6 files
+                ..Default::default()
+            }),
+        )
+        .await
+        .unwrap();
+
+        // Delete some rows to create deletions
+        dataset.delete("i < 500").await.unwrap();
+        dataset2.delete("i < 500").await.unwrap();
+
+        // Create a scalar index to check this is not touched
+        dataset
+            .create_index(
+                &["i"],
+                IndexType::Scalar,
+                Some("scalar".into()),
+                &ScalarIndexParams::default(),
+                false,
+            )
+            .await
+            .unwrap();
+
+        // Verify the initial state - no fragment reuse index should exist
+        let initial_indices = dataset.load_indices().await.unwrap();
+        assert_eq!(initial_indices.len(), 1);
+        assert_eq!(initial_indices[0].name, "scalar");
+
+        // Store the original scalar index UUID for comparison
+        let original_scalar_uuid = initial_indices[0].uuid;
+
+        // Plan and execute compaction manually
+        let options = CompactionOptions {
+            target_rows_per_fragment: 2_000,
+            defer_index_remap: true,
+            ..Default::default()
+        };
+        let options2 = CompactionOptions {
+            target_rows_per_fragment: 2_000,
+            defer_index_remap: false,
+            ..Default::default()
+        };
+
+        let plan = plan_compaction(&dataset, &options).await.unwrap();
+        let plan2 = plan_compaction(&dataset2, &options2).await.unwrap();
+
+        let mut expected_all_old_frag_ids = Vec::new();
+        let mut expected_all_new_frag_ids = Vec::new();
+        let mut expected_all_new_frag_bitmap = RoaringBitmap::new();
+        let mut expected_all_row_id_map = HashMap::new();
+        let mut deferred_results = Vec::new();
+
+        for (task, task2) in plan.tasks().iter().zip(plan2.tasks()) {
+            let deferred_result = rewrite_files(Cow::Borrowed(&dataset), task.clone(), &options)
+                .await
+                .unwrap();
+            let immediate_result =
+                rewrite_files(Cow::Borrowed(&dataset2), task2.clone(), &options2)
+                    .await
+                    .unwrap();
+
+            // Verify RewriteResult for deferred index remap
+            assert!(deferred_result.row_id_map.is_none());
+            assert!(deferred_result.changed_row_addrs.is_some());
+            assert!(!deferred_result
+                .changed_row_addrs
+                .as_ref()
+                .unwrap()
+                .is_empty());
+            assert!(!deferred_result.original_fragments.is_empty());
+            assert!(!deferred_result.new_fragments.is_empty());
+
+            // Verify RewriteResult for immediate index remap
+            assert!(immediate_result.changed_row_addrs.is_none());
+            assert!(!immediate_result.original_fragments.is_empty());
+            assert!(!immediate_result.new_fragments.is_empty());
+            assert!(immediate_result.row_id_map.is_some());
+
+            // Deserialize the changed_row_addrs from the deferred result
+            let changed_row_addrs_bytes = deferred_result.changed_row_addrs.as_ref().unwrap();
+            let mut cursor = Cursor::new(changed_row_addrs_bytes);
+            let changed_row_addrs = RoaringTreemap::deserialize_from(&mut cursor).unwrap();
+
+            // Use transpose_row_ids to convert changed_row_addrs to row_id_map
+            let transposed_map = transpose_row_ids(
+                changed_row_addrs,
+                &deferred_result.original_fragments,
+                &deferred_result.new_fragments,
+            );
+
+            // Compare with the immediate result's row_id_map
+            let immediate_map = immediate_result.row_id_map.as_ref().unwrap();
+            assert_eq!(transposed_map.len(), immediate_map.len());
+            for (old_row_id, new_row_id) in &transposed_map {
+                assert_eq!(
+                    immediate_map.get(old_row_id),
+                    Some(new_row_id),
+                    "Row ID mapping should be identical: {} -> {:?}",
+                    old_row_id,
+                    new_row_id
+                );
+            }
+
+            // Store result for further comparison against frag reuse index
+            deferred_results.push(deferred_result);
+            immediate_result.new_fragments.iter().for_each(|frag| {
+                expected_all_new_frag_bitmap.insert(frag.id as u32);
+            });
+            expected_all_new_frag_ids.extend(
+                immediate_result
+                    .new_fragments
+                    .iter()
+                    .map(|s| s.id)
+                    .collect::<Vec<_>>(),
+            );
+            expected_all_old_frag_ids.extend(
+                immediate_result
+                    .original_fragments
+                    .iter()
+                    .map(|s| s.id)
+                    .collect::<Vec<_>>(),
+            );
+            expected_all_row_id_map.extend(immediate_result.row_id_map.unwrap());
+        }
+
+        // Now commit the first compaction (using deferred results)
+        let first_metrics = commit_compaction(
+            &mut dataset,
+            deferred_results.clone(),
+            Arc::new(DatasetIndexRemapperOptions::default()),
+            &options,
+        )
+        .await
+        .unwrap();
+
+        // Verify compaction happened
+        assert!(first_metrics.fragments_removed > 0);
+        assert!(first_metrics.fragments_added > 0);
+
+        // Verify fragment reuse index was created
+        let indices_after_compact = dataset.load_indices().await.unwrap();
+        let frag_reuse_indices: Vec<_> = indices_after_compact
+            .iter()
+            .filter(|idx| idx.name == FRAG_REUSE_INDEX_NAME)
+            .collect();
+        assert_eq!(frag_reuse_indices.len(), 1);
+
+        // Load and verify the fragment reuse index content
+        let frag_reuse_index_meta = frag_reuse_indices[0];
+        assert_eq!(
+            frag_reuse_index_meta.fragment_bitmap.clone().unwrap(),
+            expected_all_new_frag_bitmap
+        );
+        let frag_reuse_details = load_frag_reuse_index_details(&dataset, frag_reuse_index_meta)
+            .await
+            .unwrap();
+        let frag_reuse_index =
+            open_frag_reuse_index(frag_reuse_details.as_ref(), dataset.fragments().as_slice())
+                .await
+                .unwrap();
+
+        // Verify the index has one version with the correct dataset version
+        let compaction_version = &frag_reuse_index.details.versions[0];
+        assert_eq!(frag_reuse_index.details.versions.len(), 1);
+        assert_eq!(
+            compaction_version.dataset_version,
+            frag_reuse_index_meta.dataset_version
+        );
+
+        // Verify the index compaction version information matches the RewriteResults
+        let changed_row_addr_bytes = &compaction_version.changed_row_addrs;
+        let mut cursor = Cursor::new(&changed_row_addr_bytes);
+        let changed_row_addrs = RoaringTreemap::deserialize_from(&mut cursor).unwrap();
+
+        assert_eq!(
+            compaction_version
+                .old_frags
+                .iter()
+                .map(|s| s.id)
+                .collect::<Vec<_>>(),
+            expected_all_old_frag_ids
+        );
+        assert_eq!(compaction_version.new_frags, expected_all_new_frag_ids);
+
+        let new_fragments = deferred_results
+            .into_iter()
+            .flat_map(|r| r.new_fragments.into_iter())
+            .collect::<Vec<_>>();
+        let transposed_map = transpose_row_ids(
+            changed_row_addrs,
+            &compaction_version.old_frags,
+            new_fragments.as_slice(),
+        );
+        assert_eq!(transposed_map, expected_all_row_id_map);
+
+        // Verify the scalar index UUID is unchanged (it should not be remapped yet)
+        let current_scalar_index = indices_after_compact
+            .iter()
+            .find(|idx| idx.name == "scalar")
+            .unwrap();
+        assert_eq!(current_scalar_index.uuid, original_scalar_uuid);
+    }
+
+    #[tokio::test]
+    async fn test_defer_index_remap_multiple_compactions() {
+        let mut data_gen = BatchGenerator::new()
+            .col(Box::new(
+                RandomVector::new().vec_width(128).named("vec".to_owned()),
+            ))
+            .col(Box::new(IncrementingInt32::new().named("i".to_owned())));
+
+        let mut dataset = Dataset::write(
+            data_gen.batch(6_000),
+            "memory://test/table",
+            Some(WriteParams {
+                max_rows_per_file: 1_000, // 6 files
+                ..Default::default()
+            }),
+        )
+        .await
+        .unwrap();
+
+        let options = CompactionOptions {
+            target_rows_per_fragment: 2_000,
+            defer_index_remap: true,
+            ..Default::default()
+        };
+
+        let mut compact_read_versions = Vec::new();
+        for i in 0..10 {
+            dataset
+                .delete(&format!("i < {}", 500 * (i + 1)))
+                .await
+                .unwrap();
+            compact_read_versions.push(dataset.manifest.version);
+            compact_files(&mut dataset, options.clone(), None)
+                .await
+                .unwrap();
+
+            let indices_after_compact = dataset.load_indices().await.unwrap();
+            let frag_reuse_indices: Vec<_> = indices_after_compact
+                .iter()
+                .filter(|idx| idx.name == FRAG_REUSE_INDEX_NAME)
+                .collect();
+            assert_eq!(frag_reuse_indices.len(), 1);
+
+            // Load and verify the fragment reuse index content
+            let frag_reuse_index_meta = frag_reuse_indices[0];
+            let frag_reuse_details = load_frag_reuse_index_details(&dataset, frag_reuse_index_meta)
+                .await
+                .unwrap();
+            let frag_reuse_index =
+                open_frag_reuse_index(frag_reuse_details.as_ref(), dataset.fragments().as_slice())
+                    .await
+                    .unwrap();
+
+            // Verify the index has one version with the correct dataset version
+            assert_eq!(
+                frag_reuse_index
+                    .details
+                    .versions
+                    .iter()
+                    .map(|v| v.dataset_version)
+                    .collect::<Vec<_>>(),
+                compact_read_versions
+            );
+        }
+    }
 }
diff --git a/rust/lance/src/dataset/transaction.rs b/rust/lance/src/dataset/transaction.rs
index 1ea90e4a884..1f5722df262 100644
--- a/rust/lance/src/dataset/transaction.rs
+++ b/rust/lance/src/dataset/transaction.rs
@@ -50,10 +50,14 @@ use std::{
     sync::Arc,
 };
 
+use super::ManifestWriteConfig;
+use crate::utils::temporal::timestamp_to_nanos;
 use deepsize::DeepSizeOf;
 use lance_core::{datatypes::Schema, Error, Result};
 use lance_file::{datatypes::Fields, version::LanceFileVersion};
+use lance_index::frag_reuse::FRAG_REUSE_INDEX_NAME;
 use lance_io::object_store::ObjectStore;
+use lance_table::feature_flags::{apply_feature_flags, FLAG_MOVE_STABLE_ROW_IDS};
 use lance_table::{
     format::{
         pb::{self, IndexMetadata},
@@ -70,10 +74,6 @@ use roaring::RoaringBitmap;
 use snafu::location;
 use uuid::Uuid;
 
-use super::ManifestWriteConfig;
-use crate::utils::temporal::timestamp_to_nanos;
-use lance_table::feature_flags::{apply_feature_flags, FLAG_MOVE_STABLE_ROW_IDS};
-
 /// A change to a dataset that can be retried
 ///
 /// This contains enough information to be able to build the next manifest,
@@ -144,6 +144,8 @@ pub enum Operation {
         groups: Vec<RewriteGroup>,
         /// Indices that have been updated with the new row addresses
         rewritten_indices: Vec<RewrittenIndex>,
+        /// The fragment reuse index to be created or updated to
+        frag_reuse_index: Option<Index>,
     },
     /// Replace data in a column in the dataset with a new data. This is used for
     /// null column population where we replace an entirely null column with a
@@ -293,12 +295,18 @@ impl PartialEq for Operation {
                 Self::Rewrite {
                     groups: a_groups,
                     rewritten_indices: a_indices,
+                    frag_reuse_index: a_frag_reuse_index,
                 },
                 Self::Rewrite {
                     groups: b_groups,
                     rewritten_indices: b_indices,
+                    frag_reuse_index: b_frag_reuse_index,
                 },
-            ) => compare_vec(a_groups, b_groups) && compare_vec(a_indices, b_indices),
+            ) => {
+                compare_vec(a_groups, b_groups)
+                    && compare_vec(a_indices, b_indices)
+                    && a_frag_reuse_index == b_frag_reuse_index
+            }
             (
                 Self::Merge {
                     fragments: a_fragments,
@@ -1443,6 +1451,7 @@ impl Transaction {
             Operation::Rewrite {
                 ref groups,
                 ref rewritten_indices,
+                ref frag_reuse_index,
             } => {
                 final_fragments.extend(maybe_existing_fragments?.clone());
                 let current_version = current_manifest.map(|m| m.version).unwrap_or_default();
@@ -1465,6 +1474,11 @@ impl Transaction {
                 } else {
                     Self::handle_rewrite_indices(&mut final_indices, rewritten_indices, groups)?;
                 }
+
+                if let Some(frag_reuse_index) = frag_reuse_index {
+                    final_indices.retain(|idx| idx.name != frag_reuse_index.name);
+                    final_indices.push(frag_reuse_index.clone());
+                }
             }
             Operation::CreateIndex {
                 new_indices,
@@ -1759,6 +1773,7 @@ impl Transaction {
                 .fields
                 .iter()
                 .all(|field_id| field_ids.contains(field_id))
+                || existing_index.name == FRAG_REUSE_INDEX_NAME
         });
 
         let fragment_ids = fragments.iter().map(|f| f.id).collect::<HashSet<_>>();
@@ -1771,6 +1786,7 @@ impl Transaction {
                 .as_ref()
                 .map(|bitmap| bitmap.iter().any(|id| fragment_ids.contains(&(id as u64))))
                 .unwrap_or(true)
+                || existing_index.name == FRAG_REUSE_INDEX_NAME
         });
     }
 
@@ -2014,6 +2030,7 @@ impl TryFrom<pb::Transaction> for Transaction {
                 Operation::Rewrite {
                     groups,
                     rewritten_indices,
+                    frag_reuse_index: None,
                 }
             }
             Some(pb::transaction::Operation::CreateIndex(pb::transaction::CreateIndex {
@@ -2254,6 +2271,7 @@ impl From<&Transaction> for pb::Transaction {
             Operation::Rewrite {
                 groups,
                 rewritten_indices,
+                frag_reuse_index: _,
             } => pb::transaction::Operation::Rewrite(pb::transaction::Rewrite {
                 groups: groups
                     .iter()
@@ -2528,6 +2546,7 @@ mod tests {
                     new_fragments: vec![fragment1.clone()],
                 }],
                 rewritten_indices: vec![],
+                frag_reuse_index: None,
             },
             Operation::ReserveFragments { num_fragments: 3 },
             Operation::Update {
@@ -2650,6 +2669,7 @@ mod tests {
                         new_fragments: vec![fragment0.clone()],
                     }],
                     rewritten_indices: Vec::new(),
+                    frag_reuse_index: None,
                 },
                 [
                     Compatible,    // append
@@ -2671,6 +2691,7 @@ mod tests {
                         new_fragments: vec![fragment0.clone()],
                     }],
                     rewritten_indices: Vec::new(),
+                    frag_reuse_index: None,
                 },
                 [
                     Compatible,    // append
diff --git a/rust/lance/src/index.rs b/rust/lance/src/index.rs
index 7024e6a0384..7cb8adcb07c 100644
--- a/rust/lance/src/index.rs
+++ b/rust/lance/src/index.rs
@@ -14,10 +14,13 @@ use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
 use futures::{stream, StreamExt, TryStreamExt};
 use itertools::Itertools;
 use lance_core::utils::parse::str_is_truthy;
-use lance_core::utils::tracing::{IO_TYPE_OPEN_SCALAR, IO_TYPE_OPEN_VECTOR, TRACE_IO_EVENTS};
+use lance_core::utils::tracing::{
+    IO_TYPE_OPEN_FRAG_REUSE, IO_TYPE_OPEN_SCALAR, IO_TYPE_OPEN_VECTOR, TRACE_IO_EVENTS,
+};
 use lance_file::reader::FileReader;
 use lance_file::v2;
 use lance_file::v2::reader::FileReaderOptions;
+use lance_index::frag_reuse::{FragReuseIndex, FRAG_REUSE_INDEX_NAME};
 use lance_index::pb::index::Implementation;
 use lance_index::scalar::expression::{
     FtsQueryParser, IndexInformationProvider, LabelListQueryParser, MultiQueryParser,
@@ -65,6 +68,7 @@ use vector::utils::get_vector_type;
 
 pub(crate) mod append;
 pub(crate) mod cache;
+pub mod frag_reuse;
 pub mod prefilter;
 pub mod scalar;
 pub mod vector;
@@ -360,6 +364,13 @@ impl DatasetIndexExt for Dataset {
                     .await?;
                 vector_index_details()
             }
+            (IndexType::FragmentReuse, _) => {
+                return Err(Error::Index {
+                    message: "Fragment reuse index can only be created through compaction"
+                        .to_string(),
+                    location: location!(),
+                })
+            }
             (index_type, index_name) => {
                 return Err(Error::Index {
                     message: format!(
@@ -555,6 +566,7 @@ impl DatasetIndexExt for Dataset {
                 indices_to_optimize
                     .as_ref()
                     .is_none_or(|names| names.contains(&idx.name))
+                    && idx.name != FRAG_REUSE_INDEX_NAME
             })
             .map(|idx| (idx.name.clone(), idx))
             .into_group_map();
@@ -803,6 +815,14 @@ pub trait DatasetIndexInternalExt: DatasetIndexExt {
         uuid: &str,
         metrics: &dyn MetricsCollector,
     ) -> Result<Arc<dyn VectorIndex>>;
+
+    /// Opens the fragment reuse index
+    async fn open_frag_reuse_index(
+        &self,
+        uuid: &str,
+        metrics: &dyn MetricsCollector,
+    ) -> Result<Arc<FragReuseIndex>>;
+
     /// Loads information about all the available scalar indices on the dataset
     async fn scalar_index_info(&self) -> Result<ScalarIndexInfo>;
 
@@ -828,6 +848,9 @@ impl DatasetIndexInternalExt for Dataset {
         if let Some(index) = self.session.index_cache.get_vector(uuid) {
             return Ok(index.as_index());
         }
+        if let Some(index) = self.session.index_cache.get_frag_reuse(uuid) {
+            return Ok(index.as_index());
+        }
 
         // Sometimes we want to open an index and we don't care if it is a scalar or vector index.
         // For example, we might want to get statistics for an index, regardless of type.
@@ -1043,6 +1066,34 @@ impl DatasetIndexInternalExt for Dataset {
         Ok(index)
     }
 
+    async fn open_frag_reuse_index(
+        &self,
+        uuid: &str,
+        metrics: &dyn MetricsCollector,
+    ) -> Result<Arc<FragReuseIndex>> {
+        if let Some(index) = self.session.index_cache.get_frag_reuse(uuid) {
+            log::debug!("Found vector index in cache uuid: {}", uuid);
+            return Ok(index);
+        }
+
+        let index_meta = self.load_index(uuid).await?.ok_or_else(|| Error::Index {
+            message: format!("Index with id {} does not exist", uuid),
+            location: location!(),
+        })?;
+        let index_details = frag_reuse::load_frag_reuse_index_details(self, &index_meta).await?;
+        let index =
+            frag_reuse::open_frag_reuse_index(index_details.as_ref(), self.fragments().as_slice())
+                .await?;
+
+        info!(target: TRACE_IO_EVENTS, index_uuid=uuid, type=IO_TYPE_OPEN_FRAG_REUSE);
+        metrics.record_index_load();
+
+        self.session
+            .index_cache
+            .insert_frag_reuse(uuid, index.clone());
+        Ok(index)
+    }
+
     #[instrument(level = "trace", skip_all)]
     async fn scalar_index_info(&self) -> Result<ScalarIndexInfo> {
         let indices = self.load_indices().await?;
diff --git a/rust/lance/src/index/cache.rs b/rust/lance/src/index/cache.rs
index 44e18b15069..b55e0758c61 100644
--- a/rust/lance/src/index/cache.rs
+++ b/rust/lance/src/index/cache.rs
@@ -12,6 +12,7 @@ use lance_index::{
 use lance_table::format::Index;
 use moka::sync::Cache;
 
+use lance_index::frag_reuse::FragReuseIndex;
 use std::sync::atomic::{AtomicU64, Ordering};
 
 #[derive(Debug, Default, DeepSizeOf)]
@@ -35,6 +36,7 @@ pub struct IndexCache {
     // TODO: Can we merge these two caches into one for uniform memory management?
     scalar_cache: Arc<Cache<String, Arc<dyn ScalarIndex>>>,
     vector_cache: Arc<Cache<String, Arc<dyn VectorIndex>>>,
+    frag_reuse_cache: Arc<Cache<String, Arc<FragReuseIndex>>>,
     // this is for v3 index, sadly we can't use the same cache as the vector index for now
     vector_partition_cache: Arc<Cache<String, Arc<dyn VectorIndexCacheEntry>>>,
 
@@ -67,6 +69,11 @@ impl DeepSizeOf for IndexCache {
                 .iter()
                 .map(|(_, v)| v.deep_size_of_children(context))
                 .sum::<usize>()
+            + self
+                .frag_reuse_cache
+                .iter()
+                .map(|(_, v)| v.deep_size_of_children(context))
+                .sum::<usize>()
             + self
                 .metadata_cache
                 .iter()
@@ -82,6 +89,8 @@ impl IndexCache {
             scalar_cache: Arc::new(Cache::new(capacity as u64)),
             vector_cache: Arc::new(Cache::new(capacity as u64)),
             vector_partition_cache: Arc::new(Cache::new(capacity as u64)),
+            // there is always 1 fragment reuse index that should be used
+            frag_reuse_cache: Arc::new(Cache::new(1)),
             metadata_cache: Arc::new(Cache::new(capacity as u64)),
             type_cache: Arc::new(Cache::new(capacity as u64)),
             cache_stats: Arc::new(CacheStats::default()),
@@ -100,6 +109,9 @@ impl IndexCache {
         self.vector_partition_cache.invalidate_all();
         self.vector_partition_cache.run_pending_tasks();
 
+        self.frag_reuse_cache.invalidate_all();
+        self.frag_reuse_cache.run_pending_tasks();
+
         self.metadata_cache.invalidate_all();
         self.metadata_cache.run_pending_tasks();
 
@@ -117,10 +129,12 @@ impl IndexCache {
         self.scalar_cache.run_pending_tasks();
         self.vector_cache.run_pending_tasks();
         self.vector_partition_cache.run_pending_tasks();
+        self.frag_reuse_cache.run_pending_tasks();
         self.metadata_cache.run_pending_tasks();
         (self.scalar_cache.entry_count()
             + self.vector_cache.entry_count()
             + self.vector_partition_cache.entry_count()
+            + self.frag_reuse_cache.entry_count()
             + self.metadata_cache.entry_count()) as usize
     }
 
@@ -128,6 +142,7 @@ impl IndexCache {
         (self.scalar_cache.entry_count()
             + self.vector_cache.entry_count()
             + self.vector_partition_cache.entry_count()
+            + self.frag_reuse_cache.entry_count()
             + self.metadata_cache.entry_count()) as usize
     }
 
@@ -172,6 +187,16 @@ impl IndexCache {
         }
     }
 
+    pub(crate) fn get_frag_reuse(&self, key: &str) -> Option<Arc<FragReuseIndex>> {
+        if let Some(index) = self.frag_reuse_cache.get(key) {
+            self.cache_stats.record_hit();
+            Some(index)
+        } else {
+            self.cache_stats.record_miss();
+            None
+        }
+    }
+
     /// Insert a new entry into the cache.
     pub(crate) fn insert_scalar(&self, key: &str, index: Arc<dyn ScalarIndex>) {
         self.scalar_cache.insert(key.to_string(), index);
@@ -181,6 +206,10 @@ impl IndexCache {
         self.vector_cache.insert(key.to_string(), index);
     }
 
+    pub(crate) fn insert_frag_reuse(&self, key: &str, index: Arc<FragReuseIndex>) {
+        self.frag_reuse_cache.insert(key.to_string(), index);
+    }
+
     pub(crate) fn insert_vector_partition(&self, key: &str, index: Arc<dyn VectorIndexCacheEntry>) {
         self.vector_partition_cache.insert(key.to_string(), index);
     }
diff --git a/rust/lance/src/index/frag_reuse.rs b/rust/lance/src/index/frag_reuse.rs
new file mode 100644
index 00000000000..b63be195fc2
--- /dev/null
+++ b/rust/lance/src/index/frag_reuse.rs
@@ -0,0 +1,171 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::dataset::optimize::remapping::transpose_row_ids;
+use crate::Dataset;
+use lance_core::Error;
+use lance_index::frag_reuse::{
+    FragReuseIndex, FragReuseIndexDetails, FragReuseVersion, FRAG_REUSE_DETAILS_FILE_NAME,
+    FRAG_REUSE_INDEX_NAME,
+};
+use lance_index::DatasetIndexExt;
+use lance_table::format::pb::fragment_reuse_index_details::{Content, InlineContent};
+use lance_table::format::pb::{ExternalFile, FragmentReuseIndexDetails};
+use lance_table::format::{Fragment, Index};
+use prost::Message;
+use roaring::{RoaringBitmap, RoaringTreemap};
+use snafu::location;
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::sync::Arc;
+use tokio::io::AsyncWriteExt;
+
+/// Load fragment reuse index details from index metadata
+pub async fn load_frag_reuse_index_details(
+    dataset: &Dataset,
+    index: &Index,
+) -> lance_core::Result<Arc<FragReuseIndexDetails>> {
+    let details_any = index.index_details.clone();
+    if details_any.is_none()
+        || !details_any
+            .as_ref()
+            .unwrap()
+            .type_url
+            .ends_with("FragmentReuseIndexDetails")
+    {
+        return Err(Error::Index {
+            message: "Index details is not for the fragment reuse index".into(),
+            location: location!(),
+        });
+    }
+
+    let proto = details_any.unwrap().to_msg::<FragmentReuseIndexDetails>()?;
+    match &proto.content {
+        None => Err(Error::Index {
+            message: "Index details content is not found".into(),
+            location: location!(),
+        }),
+        Some(Content::Inline(content)) => {
+            Ok(Arc::new(FragReuseIndexDetails::try_from(content.clone())?))
+        }
+        Some(Content::External(external_file)) => {
+            let file_path = dataset
+                .base
+                .child(index.uuid.to_string())
+                .child(external_file.path.clone());
+
+            // the file content will be cached in the index cache later
+            // so we do not put it to the file cache
+            let range = external_file.offset as usize
+                ..(external_file.offset as usize + external_file.size as usize);
+            let data = dataset
+                .object_store
+                .open(&file_path)
+                .await?
+                .get_range(range)
+                .await?;
+
+            let pb_sequence = InlineContent::decode(data)?;
+            Ok(Arc::new(FragReuseIndexDetails::try_from(pb_sequence)?))
+        }
+    }
+}
+
+/// open fragment reuse index based on its metadata details
+pub async fn open_frag_reuse_index(
+    details: &FragReuseIndexDetails,
+    dataset_fragments: &[Fragment],
+) -> lance_core::Result<Arc<FragReuseIndex>> {
+    let mut row_id_maps: Vec<HashMap<u64, Option<u64>>> =
+        Vec::with_capacity(details.versions.len());
+    for version in &details.versions {
+        let mut new_fragments_in_version = Vec::with_capacity(version.new_frags.len());
+        dataset_fragments
+            .iter()
+            .filter(|f| version.new_frags.contains(&f.id))
+            .for_each(|f| new_fragments_in_version.push(f.clone()));
+
+        let cursor = Cursor::new(&version.changed_row_addrs);
+        let changed_row_addrs = RoaringTreemap::deserialize_from(cursor).unwrap();
+
+        let row_id_map = transpose_row_ids(
+            changed_row_addrs,
+            &version.old_frags,
+            new_fragments_in_version.as_slice(),
+        );
+        row_id_maps.push(row_id_map);
+    }
+
+    Ok(Arc::new(FragReuseIndex::new(row_id_maps, details.clone())))
+}
+
+pub async fn build_new_frag_reuse_index(
+    dataset: &mut Dataset,
+    old_fragments: Vec<Fragment>,
+    new_fragment_ids: Vec<u64>,
+    new_fragment_bitmap: RoaringBitmap,
+    changed_row_addrs: RoaringTreemap,
+) -> lance_core::Result<Index> {
+    let index_id = uuid::Uuid::new_v4();
+    let mut serialized = Vec::with_capacity(changed_row_addrs.serialized_size());
+    changed_row_addrs.serialize_into(&mut serialized)?;
+
+    let new_version = FragReuseVersion {
+        dataset_version: dataset.manifest.version,
+        old_frags: old_fragments.clone(),
+        new_frags: new_fragment_ids,
+        changed_row_addrs: serialized,
+    };
+
+    let index_meta = dataset.load_indices().await.map(|indices| {
+        indices
+            .iter()
+            .find(|idx| idx.name == FRAG_REUSE_INDEX_NAME)
+            .cloned()
+    })?;
+
+    let new_index_details = match index_meta {
+        None => FragReuseIndexDetails {
+            versions: Vec::from([new_version]),
+        },
+        Some(index_meta) => {
+            let current_details = load_frag_reuse_index_details(dataset, &index_meta).await?;
+            let mut versions = current_details.versions.clone();
+            versions.push(new_version);
+            FragReuseIndexDetails { versions }
+        }
+    };
+
+    let new_index_details_proto = InlineContent::from(&new_index_details);
+    let proto = if new_index_details_proto.encoded_len() > 204800 {
+        let file_path = dataset
+            .indices_dir()
+            .child(index_id.to_string())
+            .child(FRAG_REUSE_DETAILS_FILE_NAME);
+        let mut writer = dataset.object_store.create(&file_path).await?;
+        writer
+            .write_all(new_index_details_proto.encode_to_vec().as_slice())
+            .await?;
+        let external_file = ExternalFile {
+            path: file_path.to_string(),
+            offset: 0,
+            size: new_index_details_proto.encoded_len() as u64,
+        };
+        FragmentReuseIndexDetails {
+            content: Some(Content::External(external_file)),
+        }
+    } else {
+        FragmentReuseIndexDetails {
+            content: Some(Content::Inline(new_index_details_proto)),
+        }
+    };
+
+    Ok(Index {
+        uuid: index_id,
+        name: FRAG_REUSE_INDEX_NAME.to_string(),
+        fields: vec![],
+        dataset_version: dataset.manifest.version,
+        fragment_bitmap: Some(new_fragment_bitmap),
+        index_details: Some(prost_types::Any::from_msg(&proto)?),
+    })
+}


Reviews:
user: login: wjones127
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T17:15:04Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T17:59:42Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T18:33:45Z
user: login: wjones127
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T21:41:44Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:05:24Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:05:51Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:08:15Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:08:57Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:12:55Z
user: login: wjones127
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:23:57Z
user: login: wjones127
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:24:36Z
user: login: wjones127
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:24:47Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:31:27Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-29T22:35:15Z
user: login: ztorchan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-09-12T01:59:03Z
user: login: yanghua
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-09-12T02:58:48Z
user: login: jackye1995
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-09-12T03:01:23Z

Review comments:
diff_hunk: @@ -0,0 +1,421 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::scalar::IndexStore;
+use crate::{Index, IndexType};
+use arrow_array::{Array, RecordBatch, UInt64Array};
+use arrow_schema::{DataType, Field, Schema};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use futures;
+use futures::future::join_all;
+use lance_core::{Error, Result};
+use lance_table::format::pb;
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+use uuid::Uuid;
+
+const BATCH_SIZE: usize = i32::MAX as usize - 1024 * 1024;
+
+const MAX_ROWS_PER_CHUNK: usize = 2 * 1024;
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseVersion {
+    pub dataset_version: u64,
+    pub row_id_map_path: String,
+}
+
+impl From<&FragReuseVersion> for pb::fragment_reuse_index_details::Version {
+    fn from(version: &FragReuseVersion) -> Self {
+        Self {
+            dataset_version: version.dataset_version,
+            row_id_map_path: version.row_id_map_path.clone(),
+        }
+    }
+}
+
+impl TryFrom<pb::fragment_reuse_index_details::Version> for FragReuseVersion {
+    type Error = Error;
+
+    fn try_from(version: pb::fragment_reuse_index_details::Version) -> Result<Self> {
+        Ok(Self {
+            dataset_version: version.dataset_version,
+            row_id_map_path: version.row_id_map_path,
+        })
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseIndexDetails {
+    pub versions: Vec<FragReuseVersion>,
+}
+
+impl From<&FragReuseIndexDetails> for pb::FragmentReuseIndexDetails {
+    fn from(details: &FragReuseIndexDetails) -> Self {
+        Self {
+            versions: details.versions.iter().map(|m| m.into()).collect(),
+        }
+    }
+}
+
+impl TryFrom<pb::FragmentReuseIndexDetails> for FragReuseIndexDetails {
+    type Error = Error;
+
+    fn try_from(details: pb::FragmentReuseIndexDetails) -> Result<Self> {
+        Ok(Self {
+            versions: details
+                .versions
+                .into_iter()
+                .map(|m| m.try_into())
+                .collect::<Result<Vec<_>>>()?,
+        })
+    }
+}
+
+/// An index that stores row ID maps.
+/// A row ID map describes the mapping from old row address to new address after compactions.
+/// Each version contains the mapping for one round of compaction.
+#[derive(Debug, Clone)]
+pub struct FragReuseIndex {
+    pub details: FragReuseIndexDetails,
+
+    // row ID map of all compaction rounds
+    // ordered from oldest to newest
+    pub row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+
+    store: Arc<dyn IndexStore>,
+}
+
+impl DeepSizeOf for FragReuseIndex {
+    fn deep_size_of_children(&self, context: &mut deepsize::Context) -> usize {
+        self.details.deep_size_of_children(context) + self.store.deep_size_of_children(context)
+    }
+}
+
+impl FragReuseIndex {
+    pub fn new(
+        row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+        details: FragReuseIndexDetails,
+        store: Arc<dyn IndexStore>,
+    ) -> Self {
+        Self {
+            row_id_maps,
+            details,
+            store,
+        }
+    }
+
+    pub async fn load(
+        details: FragReuseIndexDetails,
+        store: Arc<dyn IndexStore>,
+    ) -> Result<Arc<Self>> {
+        let row_id_maps = join_all(
+            details
+                .versions
+                .iter()
+                .map(|version| load_row_id_map(&version.row_id_map_path, store.clone())),
+        )
+        .await
+        .into_iter()
+        .collect::<Result<Vec<_>>>()?;
+
+        Ok(Arc::new(Self::new(row_id_maps, details, store)))
+    }
+}
+
+#[derive(Serialize)]
+struct FragReuseStatistics {
+    num_versions: usize,
+    num_rows: usize,
+}
+
+#[async_trait]
+impl Index for FragReuseIndex {
+    fn as_any(&self) -> &dyn Any {
+        self
+    }
+
+    fn as_index(self: Arc<Self>) -> Arc<dyn Index> {
+        self
+    }
+
+    fn as_vector_index(self: Arc<Self>) -> Result<Arc<dyn crate::vector::VectorIndex>> {
+        Err(Error::NotSupported {
+            source: "FragReuseIndex is not a vector index".into(),
+            location: location!(),
+        })
+    }
+
+    fn statistics(&self) -> Result<serde_json::Value> {
+        let stats = FragReuseStatistics {
+            num_versions: self.details.versions.len(),
+            num_rows: self.details.versions.len(),
+        };
+        serde_json::to_value(stats).map_err(|e| Error::Internal {
+            message: format!("failed to serialize fragment reuse index statistics: {}", e),
+            location: location!(),
+        })
+    }
+
+    async fn prewarm(&self) -> Result<()> {
+        // TODO: can preload the row ID map
+        Ok(())
+    }
+
+    fn index_type(&self) -> IndexType {
+        IndexType::FragmentReuse
+    }
+
+    async fn calculate_included_frags(&self) -> Result<RoaringBitmap> {
+        unimplemented!()
+    }
+}
+
+/// Save a row ID map to a Lance index file.
+pub async fn save_row_id_map(
+    index_store: Arc<dyn IndexStore>,
+    dataset_version: &u64,
+    row_id_map: &HashMap<u64, Option<u64>>,
path: rust/lance-index/src/reuse.rs
user: login: wjones127
body: This structure feels a bit unfortunate given that the common case is mapping sorted contiguous ranges to other sorted contiguous ranges. Have you considered re-using some of the `U64Segment` or `RowIdSequence` machinery?
created_at: 2025-05-19T17:07:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,421 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::scalar::IndexStore;
+use crate::{Index, IndexType};
+use arrow_array::{Array, RecordBatch, UInt64Array};
+use arrow_schema::{DataType, Field, Schema};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use futures;
+use futures::future::join_all;
+use lance_core::{Error, Result};
+use lance_table::format::pb;
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+use uuid::Uuid;
+
+const BATCH_SIZE: usize = i32::MAX as usize - 1024 * 1024;
+
+const MAX_ROWS_PER_CHUNK: usize = 2 * 1024;
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseVersion {
+    pub dataset_version: u64,
+    pub row_id_map_path: String,
+}
+
+impl From<&FragReuseVersion> for pb::fragment_reuse_index_details::Version {
+    fn from(version: &FragReuseVersion) -> Self {
+        Self {
+            dataset_version: version.dataset_version,
+            row_id_map_path: version.row_id_map_path.clone(),
+        }
+    }
+}
+
+impl TryFrom<pb::fragment_reuse_index_details::Version> for FragReuseVersion {
+    type Error = Error;
+
+    fn try_from(version: pb::fragment_reuse_index_details::Version) -> Result<Self> {
+        Ok(Self {
+            dataset_version: version.dataset_version,
+            row_id_map_path: version.row_id_map_path,
+        })
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseIndexDetails {
+    pub versions: Vec<FragReuseVersion>,
+}
+
+impl From<&FragReuseIndexDetails> for pb::FragmentReuseIndexDetails {
+    fn from(details: &FragReuseIndexDetails) -> Self {
+        Self {
+            versions: details.versions.iter().map(|m| m.into()).collect(),
+        }
+    }
+}
+
+impl TryFrom<pb::FragmentReuseIndexDetails> for FragReuseIndexDetails {
+    type Error = Error;
+
+    fn try_from(details: pb::FragmentReuseIndexDetails) -> Result<Self> {
+        Ok(Self {
+            versions: details
+                .versions
+                .into_iter()
+                .map(|m| m.try_into())
+                .collect::<Result<Vec<_>>>()?,
+        })
+    }
+}
+
+/// An index that stores row ID maps.
+/// A row ID map describes the mapping from old row address to new address after compactions.
+/// Each version contains the mapping for one round of compaction.
+#[derive(Debug, Clone)]
+pub struct FragReuseIndex {
+    pub details: FragReuseIndexDetails,
+
+    // row ID map of all compaction rounds
+    // ordered from oldest to newest
+    pub row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+
+    store: Arc<dyn IndexStore>,
+}
+
+impl DeepSizeOf for FragReuseIndex {
+    fn deep_size_of_children(&self, context: &mut deepsize::Context) -> usize {
+        self.details.deep_size_of_children(context) + self.store.deep_size_of_children(context)
+    }
+}
+
+impl FragReuseIndex {
+    pub fn new(
+        row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+        details: FragReuseIndexDetails,
+        store: Arc<dyn IndexStore>,
+    ) -> Self {
+        Self {
+            row_id_maps,
+            details,
+            store,
+        }
+    }
+
+    pub async fn load(
+        details: FragReuseIndexDetails,
+        store: Arc<dyn IndexStore>,
+    ) -> Result<Arc<Self>> {
+        let row_id_maps = join_all(
+            details
+                .versions
+                .iter()
+                .map(|version| load_row_id_map(&version.row_id_map_path, store.clone())),
+        )
+        .await
+        .into_iter()
+        .collect::<Result<Vec<_>>>()?;
+
+        Ok(Arc::new(Self::new(row_id_maps, details, store)))
+    }
+}
+
+#[derive(Serialize)]
+struct FragReuseStatistics {
+    num_versions: usize,
+    num_rows: usize,
+}
+
+#[async_trait]
+impl Index for FragReuseIndex {
+    fn as_any(&self) -> &dyn Any {
+        self
+    }
+
+    fn as_index(self: Arc<Self>) -> Arc<dyn Index> {
+        self
+    }
+
+    fn as_vector_index(self: Arc<Self>) -> Result<Arc<dyn crate::vector::VectorIndex>> {
+        Err(Error::NotSupported {
+            source: "FragReuseIndex is not a vector index".into(),
+            location: location!(),
+        })
+    }
+
+    fn statistics(&self) -> Result<serde_json::Value> {
+        let stats = FragReuseStatistics {
+            num_versions: self.details.versions.len(),
+            num_rows: self.details.versions.len(),
+        };
+        serde_json::to_value(stats).map_err(|e| Error::Internal {
+            message: format!("failed to serialize fragment reuse index statistics: {}", e),
+            location: location!(),
+        })
+    }
+
+    async fn prewarm(&self) -> Result<()> {
+        // TODO: can preload the row ID map
+        Ok(())
+    }
+
+    fn index_type(&self) -> IndexType {
+        IndexType::FragmentReuse
+    }
+
+    async fn calculate_included_frags(&self) -> Result<RoaringBitmap> {
+        unimplemented!()
+    }
+}
+
+/// Save a row ID map to a Lance index file.
+pub async fn save_row_id_map(
+    index_store: Arc<dyn IndexStore>,
+    dataset_version: &u64,
+    row_id_map: &HashMap<u64, Option<u64>>,
+) -> Result<String> {
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("old_row_id", DataType::UInt64, false),
+        Field::new("new_row_id", DataType::UInt64, true),
+    ]));
+
+    // need a UUID in case of concurrent compactions writing
+    let file_path = format!("{}_row_id_map_{}.lance", dataset_version, Uuid::new_v4());
+    let mut index_file = index_store
+        .new_index_file(&file_path, schema.clone())
+        .await?;
+
+    let mut old_row_ids = Vec::new();
+    let mut new_row_ids = Vec::new();
+    let mut rows = 0;
+
+    for (old_row_id, new_row_id) in row_id_map.into_iter() {
+        // flush current batch
+        if rows > BATCH_SIZE {
+            let batch = RecordBatch::try_new(
+                schema.clone(),
+                vec![
+                    Arc::new(UInt64Array::from(old_row_ids.clone())),
+                    Arc::new(UInt64Array::from(new_row_ids.clone())),
+                ],
+            )?;
+
+            index_file.write_record_batch(batch).await?;
+
+            old_row_ids.clear();
+            new_row_ids.clear();
+            rows = 0;
+        }
+
+        old_row_ids.push(*old_row_id);
+        new_row_ids.push(*new_row_id);
+        rows += 1;
+    }
+
+    if !old_row_ids.is_empty() {
+        // write remaining
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(UInt64Array::from(old_row_ids)),
+                Arc::new(UInt64Array::from(new_row_ids)),
+            ],
+        )?;
+
+        index_file.write_record_batch(batch).await?;
+    }
+
+    index_file.finish().await?;
+    Ok(file_path)
+}
+
+pub async fn load_row_id_map(
+    file_path: &str,
+    store: Arc<dyn IndexStore>,
+) -> Result<HashMap<u64, Option<u64>>> {
path: rust/lance-index/src/reuse.rs
user: login: wjones127
body: If we are going to use this within the conflict resolution code path, I think we should aggressive in limiting IO requests. You can add a unit test that measures the number of read and write IO requests using `StatsHolder`. Here is an example test that does this:

https://github.com/lancedb/lance/blob/01962adcd8d188a1dd1a9daa5ca5ebcc524d9b47/rust/lance/src/index.rs#L1835

Might need to store the file size of the index file in the metadata so that you can skip the additional head request when opening the index file.
created_at: 2025-05-19T17:13:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -369,4 +369,17 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  message Version {
+    // version of the dataset corresponds to the dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
path: protos/table.proto
user: login: wjones127
body: If we want to use this for conflict resolution, we should probably add this index in the same commit as compaction.
created_at: 2025-05-19T17:14:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -369,4 +369,17 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  message Version {
+    // version of the dataset corresponds to the dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
path: protos/table.proto
user: login: jackye1995
body: Yes that is the goal, this should be the same dataset version as the compaction
created_at: 2025-05-19T17:59:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,421 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::scalar::IndexStore;
+use crate::{Index, IndexType};
+use arrow_array::{Array, RecordBatch, UInt64Array};
+use arrow_schema::{DataType, Field, Schema};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use futures;
+use futures::future::join_all;
+use lance_core::{Error, Result};
+use lance_table::format::pb;
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+use uuid::Uuid;
+
+const BATCH_SIZE: usize = i32::MAX as usize - 1024 * 1024;
+
+const MAX_ROWS_PER_CHUNK: usize = 2 * 1024;
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseVersion {
+    pub dataset_version: u64,
+    pub row_id_map_path: String,
+}
+
+impl From<&FragReuseVersion> for pb::fragment_reuse_index_details::Version {
+    fn from(version: &FragReuseVersion) -> Self {
+        Self {
+            dataset_version: version.dataset_version,
+            row_id_map_path: version.row_id_map_path.clone(),
+        }
+    }
+}
+
+impl TryFrom<pb::fragment_reuse_index_details::Version> for FragReuseVersion {
+    type Error = Error;
+
+    fn try_from(version: pb::fragment_reuse_index_details::Version) -> Result<Self> {
+        Ok(Self {
+            dataset_version: version.dataset_version,
+            row_id_map_path: version.row_id_map_path,
+        })
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct FragReuseIndexDetails {
+    pub versions: Vec<FragReuseVersion>,
+}
+
+impl From<&FragReuseIndexDetails> for pb::FragmentReuseIndexDetails {
+    fn from(details: &FragReuseIndexDetails) -> Self {
+        Self {
+            versions: details.versions.iter().map(|m| m.into()).collect(),
+        }
+    }
+}
+
+impl TryFrom<pb::FragmentReuseIndexDetails> for FragReuseIndexDetails {
+    type Error = Error;
+
+    fn try_from(details: pb::FragmentReuseIndexDetails) -> Result<Self> {
+        Ok(Self {
+            versions: details
+                .versions
+                .into_iter()
+                .map(|m| m.try_into())
+                .collect::<Result<Vec<_>>>()?,
+        })
+    }
+}
+
+/// An index that stores row ID maps.
+/// A row ID map describes the mapping from old row address to new address after compactions.
+/// Each version contains the mapping for one round of compaction.
+#[derive(Debug, Clone)]
+pub struct FragReuseIndex {
+    pub details: FragReuseIndexDetails,
+
+    // row ID map of all compaction rounds
+    // ordered from oldest to newest
+    pub row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+
+    store: Arc<dyn IndexStore>,
+}
+
+impl DeepSizeOf for FragReuseIndex {
+    fn deep_size_of_children(&self, context: &mut deepsize::Context) -> usize {
+        self.details.deep_size_of_children(context) + self.store.deep_size_of_children(context)
+    }
+}
+
+impl FragReuseIndex {
+    pub fn new(
+        row_id_maps: Vec<HashMap<u64, Option<u64>>>,
+        details: FragReuseIndexDetails,
+        store: Arc<dyn IndexStore>,
+    ) -> Self {
+        Self {
+            row_id_maps,
+            details,
+            store,
+        }
+    }
+
+    pub async fn load(
+        details: FragReuseIndexDetails,
+        store: Arc<dyn IndexStore>,
+    ) -> Result<Arc<Self>> {
+        let row_id_maps = join_all(
+            details
+                .versions
+                .iter()
+                .map(|version| load_row_id_map(&version.row_id_map_path, store.clone())),
+        )
+        .await
+        .into_iter()
+        .collect::<Result<Vec<_>>>()?;
+
+        Ok(Arc::new(Self::new(row_id_maps, details, store)))
+    }
+}
+
+#[derive(Serialize)]
+struct FragReuseStatistics {
+    num_versions: usize,
+    num_rows: usize,
+}
+
+#[async_trait]
+impl Index for FragReuseIndex {
+    fn as_any(&self) -> &dyn Any {
+        self
+    }
+
+    fn as_index(self: Arc<Self>) -> Arc<dyn Index> {
+        self
+    }
+
+    fn as_vector_index(self: Arc<Self>) -> Result<Arc<dyn crate::vector::VectorIndex>> {
+        Err(Error::NotSupported {
+            source: "FragReuseIndex is not a vector index".into(),
+            location: location!(),
+        })
+    }
+
+    fn statistics(&self) -> Result<serde_json::Value> {
+        let stats = FragReuseStatistics {
+            num_versions: self.details.versions.len(),
+            num_rows: self.details.versions.len(),
+        };
+        serde_json::to_value(stats).map_err(|e| Error::Internal {
+            message: format!("failed to serialize fragment reuse index statistics: {}", e),
+            location: location!(),
+        })
+    }
+
+    async fn prewarm(&self) -> Result<()> {
+        // TODO: can preload the row ID map
+        Ok(())
+    }
+
+    fn index_type(&self) -> IndexType {
+        IndexType::FragmentReuse
+    }
+
+    async fn calculate_included_frags(&self) -> Result<RoaringBitmap> {
+        unimplemented!()
+    }
+}
+
+/// Save a row ID map to a Lance index file.
+pub async fn save_row_id_map(
+    index_store: Arc<dyn IndexStore>,
+    dataset_version: &u64,
+    row_id_map: &HashMap<u64, Option<u64>>,
path: rust/lance-index/src/reuse.rs
user: login: jackye1995
body: Yeah I used `HashMap` becuase these row ID maps come from the `RewriteResult` of `optimize.rewrite_files`, at that point it's already a hashmap and the sorting information is already lost. But I can make this storage more efficient with the data structure you mentioned, and then make the upstream changes in `optimize` based on what we arrive at here.
created_at: 2025-05-19T18:33:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -369,4 +369,31 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  oneof content {
+    // if < 200KB, store the content inline, otherwise store the InlineContent bytes in external file
+    InlineContent inline = 1;
+    ExternalFile external = 2;
+  }
+
+  message InlineContent {
+    repeated Version versions = 1;
+  }
+
+  message Version {
+    // The dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
+
+    // a roaring treemap of the changed row addresses
+    bytes changed_row_addrs = 2;
path: protos/table.proto
user: login: wjones127
body: I feel like this could be explained more, like how it is used. I feel like a roaring treemap acts as a set, but we likely want a mapping?
created_at: 2025-05-29T16:29:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,227 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::{Index, IndexType};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use itertools::Itertools;
+use lance_core::{Error, Result};
+use lance_table::format::pb::fragment_reuse_index_details::InlineContent;
+use lance_table::format::{pb, ExternalFile, Fragment};
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+
+pub const FRAG_REUSE_INDEX_NAME: &str = "__lance_frag_reuse";
path: rust/lance-index/src/frag_reuse.rs
user: login: wjones127
body: Are we going to ignore this index type in the in `list_indices()` output? 
created_at: 2025-05-29T20:30:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1759,6 +1773,7 @@ impl Transaction {
                 .fields
                 .iter()
                 .all(|field_id| field_ids.contains(field_id))
+                || existing_index.name == FRAG_REUSE_INDEX_NAME
path: rust/lance/src/dataset/transaction.rs
user: login: wjones127
body: Does this mean the frag reuse indices will never go away? How long will we retain them?
created_at: 2025-05-29T21:34:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,6 +36,7 @@ pub struct IndexCache {
     // TODO: Can we merge these two caches into one for uniform memory management?
     scalar_cache: Arc<Cache<String, Arc<dyn ScalarIndex>>>,
     vector_cache: Arc<Cache<String, Arc<dyn VectorIndex>>>,
+    frag_reuse_cache: Arc<Cache<String, Arc<FragReuseIndex>>>,
path: rust/lance/src/index/cache.rs
user: login: wjones127
body: 🤣 😭  I will consolidate these soon...
created_at: 2025-05-29T21:36:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,175 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::dataset::optimize::remapping::transpose_row_ids;
+use crate::Dataset;
+use lance_core::Error;
+use lance_index::frag_reuse::{
+    FragReuseIndex, FragReuseIndexDetails, FragReuseVersion, FRAG_REUSE_DETAILS_FILE_NAME,
+    FRAG_REUSE_INDEX_NAME,
+};
+use lance_index::DatasetIndexExt;
+use lance_table::format::pb::fragment_reuse_index_details::{Content, InlineContent};
+use lance_table::format::pb::{ExternalFile, FragmentReuseIndexDetails};
+use lance_table::format::{Fragment, Index};
+use prost::Message;
+use roaring::{RoaringBitmap, RoaringTreemap};
+use snafu::location;
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::sync::Arc;
+use tokio::io::AsyncWriteExt;
+
+/// Load fragment reuse index details from index metadata
+pub async fn load_frag_reuse_index_details(
+    dataset: &Dataset,
+    index: &Index,
+) -> lance_core::Result<Arc<FragReuseIndexDetails>> {
+    let details_any = index.index_details.clone();
+    if details_any.is_none()
+        || !details_any
+            .as_ref()
+            .unwrap()
+            .type_url
+            .ends_with("FragmentReuseIndexDetails")
+    {
+        return Err(Error::Index {
+            message: "Index details is not for the fragment reuse index".into(),
+            location: location!(),
+        });
+    }
+
+    let proto = details_any.unwrap().to_msg::<FragmentReuseIndexDetails>()?;
+    match &proto.content {
+        None => Err(Error::Index {
+            message: "Index details content is not found".into(),
+            location: location!(),
+        }),
+        Some(Content::Inline(content)) => {
+            Ok(Arc::new(FragReuseIndexDetails::try_from(content.clone())?))
+        }
+        Some(Content::External(external_file)) => {
+            let file_path = dataset
+                .base
+                .child(index.uuid.to_string())
+                .child(external_file.path.clone());
+
+            dataset
+                .session
+                .file_metadata_cache
+                .get_or_insert(&file_path, |_path| async {
+                    let range = external_file.offset as usize
+                        ..(external_file.offset as usize + external_file.size as usize);
+                    let data = dataset
+                        .object_store
+                        .open(&file_path)
+                        .await?
+                        .get_range(range)
+                        .await?;
+
+                    let pb_sequence = InlineContent::decode(data)?;
+                    FragReuseIndexDetails::try_from(pb_sequence)
path: rust/lance/src/index/frag_reuse.rs
user: login: wjones127
body: Are we double caching this data? Won't it also be stored in the index cache?
created_at: 2025-05-29T21:39:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -369,4 +369,31 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  oneof content {
+    // if < 200KB, store the content inline, otherwise store the InlineContent bytes in external file
+    InlineContent inline = 1;
+    ExternalFile external = 2;
+  }
+
+  message InlineContent {
+    repeated Version versions = 1;
+  }
+
+  message Version {
+    // The dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
+
+    // a roaring treemap of the changed row addresses
+    bytes changed_row_addrs = 2;
path: protos/table.proto
user: login: wjones127
body: Or is the intended use "if you have row addrs in here, then use the old fragments instead of the new fragments while scanning / taking"?
created_at: 2025-05-29T21:41:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -369,4 +369,31 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  oneof content {
+    // if < 200KB, store the content inline, otherwise store the InlineContent bytes in external file
+    InlineContent inline = 1;
+    ExternalFile external = 2;
+  }
+
+  message InlineContent {
+    repeated Version versions = 1;
+  }
+
+  message Version {
+    // The dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
+
+    // a roaring treemap of the changed row addresses
+    bytes changed_row_addrs = 2;
path: protos/table.proto
user: login: jackye1995
body: the changed_row_addrs + old fragment ids + new fragment ids can be used to together recover the full old address to new address mapping, and that is used for separated remap (another PR) and use for scanning/taking at read path (another PR). I can add the details there.
created_at: 2025-05-29T22:05:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,227 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::{Index, IndexType};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use itertools::Itertools;
+use lance_core::{Error, Result};
+use lance_table::format::pb::fragment_reuse_index_details::InlineContent;
+use lance_table::format::{pb, ExternalFile, Fragment};
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+
+pub const FRAG_REUSE_INDEX_NAME: &str = "__lance_frag_reuse";
path: rust/lance-index/src/frag_reuse.rs
user: login: jackye1995
body: No, because we still need to be able to discover this index.
created_at: 2025-05-29T22:05:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1759,6 +1773,7 @@ impl Transaction {
                 .fields
                 .iter()
                 .all(|field_id| field_ids.contains(field_id))
+                || existing_index.name == FRAG_REUSE_INDEX_NAME
path: rust/lance/src/dataset/transaction.rs
user: login: jackye1995
body: after all the indexes are remapped it can go away, so depending on the reindexing process. But given we want to continuously compact and reindex at the same time, this intermediate state might always exist.
created_at: 2025-05-29T22:08:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,6 +36,7 @@ pub struct IndexCache {
     // TODO: Can we merge these two caches into one for uniform memory management?
     scalar_cache: Arc<Cache<String, Arc<dyn ScalarIndex>>>,
     vector_cache: Arc<Cache<String, Arc<dyn VectorIndex>>>,
+    frag_reuse_cache: Arc<Cache<String, Arc<FragReuseIndex>>>,
path: rust/lance/src/index/cache.rs
user: login: jackye1995
body: yeah I saw the TODO so was thinking it could be done together, currently I am just keeping things consistent...
created_at: 2025-05-29T22:08:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,175 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::dataset::optimize::remapping::transpose_row_ids;
+use crate::Dataset;
+use lance_core::Error;
+use lance_index::frag_reuse::{
+    FragReuseIndex, FragReuseIndexDetails, FragReuseVersion, FRAG_REUSE_DETAILS_FILE_NAME,
+    FRAG_REUSE_INDEX_NAME,
+};
+use lance_index::DatasetIndexExt;
+use lance_table::format::pb::fragment_reuse_index_details::{Content, InlineContent};
+use lance_table::format::pb::{ExternalFile, FragmentReuseIndexDetails};
+use lance_table::format::{Fragment, Index};
+use prost::Message;
+use roaring::{RoaringBitmap, RoaringTreemap};
+use snafu::location;
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::sync::Arc;
+use tokio::io::AsyncWriteExt;
+
+/// Load fragment reuse index details from index metadata
+pub async fn load_frag_reuse_index_details(
+    dataset: &Dataset,
+    index: &Index,
+) -> lance_core::Result<Arc<FragReuseIndexDetails>> {
+    let details_any = index.index_details.clone();
+    if details_any.is_none()
+        || !details_any
+            .as_ref()
+            .unwrap()
+            .type_url
+            .ends_with("FragmentReuseIndexDetails")
+    {
+        return Err(Error::Index {
+            message: "Index details is not for the fragment reuse index".into(),
+            location: location!(),
+        });
+    }
+
+    let proto = details_any.unwrap().to_msg::<FragmentReuseIndexDetails>()?;
+    match &proto.content {
+        None => Err(Error::Index {
+            message: "Index details content is not found".into(),
+            location: location!(),
+        }),
+        Some(Content::Inline(content)) => {
+            Ok(Arc::new(FragReuseIndexDetails::try_from(content.clone())?))
+        }
+        Some(Content::External(external_file)) => {
+            let file_path = dataset
+                .base
+                .child(index.uuid.to_string())
+                .child(external_file.path.clone());
+
+            dataset
+                .session
+                .file_metadata_cache
+                .get_or_insert(&file_path, |_path| async {
+                    let range = external_file.offset as usize
+                        ..(external_file.offset as usize + external_file.size as usize);
+                    let data = dataset
+                        .object_store
+                        .open(&file_path)
+                        .await?
+                        .get_range(range)
+                        .await?;
+
+                    let pb_sequence = InlineContent::decode(data)?;
+                    FragReuseIndexDetails::try_from(pb_sequence)
path: rust/lance/src/index/frag_reuse.rs
user: login: jackye1995
body: yeah you are right, the content will be in the index cache anyway so this is not necessary. Let me remove
created_at: 2025-05-29T22:12:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,227 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::{Index, IndexType};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use itertools::Itertools;
+use lance_core::{Error, Result};
+use lance_table::format::pb::fragment_reuse_index_details::InlineContent;
+use lance_table::format::{pb, ExternalFile, Fragment};
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+
+pub const FRAG_REUSE_INDEX_NAME: &str = "__lance_frag_reuse";
path: rust/lance-index/src/frag_reuse.rs
user: login: wjones127
body: Yeah, I'm just thinking about whether it will confuse users. We should be careful when we bring this up to lancedb. I think it might error if `list_indices()` returns an index type it doesn't recognize.
created_at: 2025-05-29T22:23:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -369,4 +369,34 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  oneof content {
+    // if < 200KB, store the content inline, otherwise store the InlineContent bytes in external file
+    InlineContent inline = 1;
+    ExternalFile external = 2;
+  }
+
+  message InlineContent {
+    repeated Version versions = 1;
+  }
+
+  message Version {
+    // The dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
+
+    // A roaring treemap of the changed row addresses.
+    // When combined with the old fragment IDs and new fragment IDs,
+    // it can recover the full mapping of old row addresses to either new row addresses or deleted.
+    // this mapping can then be used to remap indexes or satisfy index queries for the new unindexed fragments.
path: protos/table.proto
user: login: wjones127
body: Does this assume that no re-ordering happens?
created_at: 2025-05-29T22:24:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,227 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+use crate::{Index, IndexType};
+use async_trait::async_trait;
+use deepsize::DeepSizeOf;
+use itertools::Itertools;
+use lance_core::{Error, Result};
+use lance_table::format::pb::fragment_reuse_index_details::InlineContent;
+use lance_table::format::{pb, ExternalFile, Fragment};
+use roaring::RoaringBitmap;
+use serde::{Deserialize, Serialize};
+use snafu::location;
+use std::{any::Any, collections::HashMap, sync::Arc};
+
+pub const FRAG_REUSE_INDEX_NAME: &str = "__lance_frag_reuse";
path: rust/lance-index/src/frag_reuse.rs
user: login: jackye1995
body: I see let me create an issue to track this
created_at: 2025-05-29T22:31:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -369,4 +369,34 @@ message BitmapIndexDetails {}
 message LabelListIndexDetails {}
 message InvertedIndexDetails {}
 message NGramIndexDetails {}
-message VectorIndexDetails {}
\ No newline at end of file
+message VectorIndexDetails {}
+
+message FragmentReuseIndexDetails {
+
+  oneof content {
+    // if < 200KB, store the content inline, otherwise store the InlineContent bytes in external file
+    InlineContent inline = 1;
+    ExternalFile external = 2;
+  }
+
+  message InlineContent {
+    repeated Version versions = 1;
+  }
+
+  message Version {
+    // The dataset_version at the time the index adds this version entry
+    uint64 dataset_version = 1;
+
+    // A roaring treemap of the changed row addresses.
+    // When combined with the old fragment IDs and new fragment IDs,
+    // it can recover the full mapping of old row addresses to either new row addresses or deleted.
+    // this mapping can then be used to remap indexes or satisfy index queries for the new unindexed fragments.
path: protos/table.proto
user: login: jackye1995
body: yes the order for which the old and new fragments are stored must preserve what happened at the rewrite time. That's actually a good point, the comment below should be updated, since structures like roaring bitmap cannot be used to store this, only a list. Let me update the comments.
created_at: 2025-05-29T22:35:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -738,12 +752,26 @@ async fn rewrite_files(
         );
         reserve_fragment_ids(&dataset, new_fragments.iter_mut()).await?;
 
-        remapping::transpose_row_ids(row_ids, &fragments, &new_fragments)
+        if options.defer_index_remap {
+            let mut changed_row_addrs = Vec::with_capacity(row_ids.serialized_size());
+            row_ids.serialize_into(&mut changed_row_addrs)?;
+            (None, Some(changed_row_addrs))
+        } else {
+            let row_id_map = remapping::transpose_row_ids(row_ids, &fragments, &new_fragments);
+            (Some(row_id_map), None)
+        }
     } else {
         log::info!("Compaction task {}: rechunking stable row ids", task_id);
         rechunk_stable_row_ids(dataset.as_ref(), &mut new_fragments, &fragments).await?;
 
-        HashMap::new()
+        if options.defer_index_remap {
path: rust/lance/src/dataset/optimize.rs
user: login: ztorchan
body: Hello, It seems that there will be a empty FragReuseIndex when use_stable_row_id && defer_index_remap.
Why we need the empty FragReuseIndex?
created_at: 2025-09-12T01:59:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -738,12 +752,26 @@ async fn rewrite_files(
         );
         reserve_fragment_ids(&dataset, new_fragments.iter_mut()).await?;
 
-        remapping::transpose_row_ids(row_ids, &fragments, &new_fragments)
+        if options.defer_index_remap {
+            let mut changed_row_addrs = Vec::with_capacity(row_ids.serialized_size());
+            row_ids.serialize_into(&mut changed_row_addrs)?;
+            (None, Some(changed_row_addrs))
+        } else {
+            let row_id_map = remapping::transpose_row_ids(row_ids, &fragments, &new_fragments);
+            (Some(row_id_map), None)
+        }
     } else {
         log::info!("Compaction task {}: rechunking stable row ids", task_id);
         rechunk_stable_row_ids(dataset.as_ref(), &mut new_fragments, &fragments).await?;
 
-        HashMap::new()
+        if options.defer_index_remap {
path: rust/lance/src/dataset/optimize.rs
user: login: yanghua
body: Hi @ztorchan I would suggest you raise a new issue to file a problem you have encountered. This work was done before the stable row id feature. And recently we have changed some logic for realizing the update-stable row ID.  
created_at: 2025-09-12T02:58:48Z
author_association: COLLABORATOR
diff_hunk: @@ -738,12 +752,26 @@ async fn rewrite_files(
         );
         reserve_fragment_ids(&dataset, new_fragments.iter_mut()).await?;
 
-        remapping::transpose_row_ids(row_ids, &fragments, &new_fragments)
+        if options.defer_index_remap {
+            let mut changed_row_addrs = Vec::with_capacity(row_ids.serialized_size());
+            row_ids.serialize_into(&mut changed_row_addrs)?;
+            (None, Some(changed_row_addrs))
+        } else {
+            let row_id_map = remapping::transpose_row_ids(row_ids, &fragments, &new_fragments);
+            (Some(row_id_map), None)
+        }
     } else {
         log::info!("Compaction task {}: rechunking stable row ids", task_id);
         rechunk_stable_row_ids(dataset.as_ref(), &mut new_fragments, &fragments).await?;
 
-        HashMap::new()
+        if options.defer_index_remap {
path: rust/lance/src/dataset/optimize.rs
user: login: jackye1995
body: Yeah let's discuss it separately. But overall I think maybe we should use a separated flag here as use_stable_row_id_for_index. Because that's what this flag is doing here, and it assumes either you enable that flag and you don't need to remap, or you enable the other flag then you defer the remap. 
created_at: 2025-09-12T03:01:23Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov-commenter
created_at: 2025-05-19T07:02:40Z
body: ## [Codecov](https://app.codecov.io/gh/lancedb/lance/pull/3847?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) Report
Attention: Patch coverage is `79.15452%` with `143 lines` in your changes missing coverage. Please review.
> Project coverage is 78.29%. Comparing base [(`bf2f72e`)](https://app.codecov.io/gh/lancedb/lance/commit/bf2f72ec2645701ce810bb81b6b651c4fa06977a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) to head [(`3905fe5`)](https://app.codecov.io/gh/lancedb/lance/commit/3905fe5734f8744b325eeef061bd032f14694f30?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb).

| [Files with missing lines](https://app.codecov.io/gh/lancedb/lance/pull/3847?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Patch % | Lines |
|---|---|---|
| [rust/lance/src/index/frag\_reuse.rs](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&filepath=rust%2Flance%2Fsrc%2Findex%2Ffrag_reuse.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#diff-cnVzdC9sYW5jZS9zcmMvaW5kZXgvZnJhZ19yZXVzZS5ycw==) | 61.41% | [42 Missing and 7 partials :warning: ](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) |
| [rust/lance-index/src/frag\_reuse.rs](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&filepath=rust%2Flance-index%2Fsrc%2Ffrag_reuse.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#diff-cnVzdC9sYW5jZS1pbmRleC9zcmMvZnJhZ19yZXVzZS5ycw==) | 75.17% | [33 Missing and 2 partials :warning: ](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) |
| [rust/lance/src/index.rs](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&filepath=rust%2Flance%2Fsrc%2Findex.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#diff-cnVzdC9sYW5jZS9zcmMvaW5kZXgucnM=) | 6.66% | [27 Missing and 1 partial :warning: ](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) |
| [rust/lance/src/dataset/optimize.rs](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&filepath=rust%2Flance%2Fsrc%2Fdataset%2Foptimize.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#diff-cnVzdC9sYW5jZS9zcmMvZGF0YXNldC9vcHRpbWl6ZS5ycw==) | 96.53% | [6 Missing and 6 partials :warning: ](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) |
| [rust/lance/src/index/cache.rs](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&filepath=rust%2Flance%2Fsrc%2Findex%2Fcache.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#diff-cnVzdC9sYW5jZS9zcmMvaW5kZXgvY2FjaGUucnM=) | 52.17% | [10 Missing and 1 partial :warning: ](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) |
| [rust/lance/src/dataset/transaction.rs](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&filepath=rust%2Flance%2Fsrc%2Fdataset%2Ftransaction.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#diff-cnVzdC9sYW5jZS9zcmMvZGF0YXNldC90cmFuc2FjdGlvbi5ycw==) | 61.11% | [7 Missing :warning: ](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) |
| [rust/lance-index/src/lib.rs](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&filepath=rust%2Flance-index%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#diff-cnVzdC9sYW5jZS1pbmRleC9zcmMvbGliLnJz) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/lancedb/lance/pull/3847?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) |

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff            @@
##             main    #3847    +/-   ##
========================================
  Coverage   78.29%   78.29%            
========================================
  Files         281      283     +2     
  Lines      108112   108783   +671     
  Branches   108112   108783   +671     
========================================
+ Hits        84641    85167   +526     
- Misses      20126    20254   +128     
- Partials     3345     3362    +17     
```

| [Flag](https://app.codecov.io/gh/lancedb/lance/pull/3847/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/lancedb/lance/pull/3847/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | `78.29% <79.15%> (+<0.01%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/lancedb/lance/pull/3847?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: jackye1995
created_at: 2025-05-29T17:05:14Z
body: looks like this breaks backwards compatibility, will do the version bump while the CI is running
author_association: CONTRIBUTOR
user: login: jackye1995
created_at: 2025-05-29T17:47:52Z
body: Given #3882 is also breaking, I will wait to see if this PR meets the next release, if not I will bump the versions.
author_association: CONTRIBUTOR