Repository: chroma-core/chroma
Pull request #4466: [ENH]: add operator for garbage collection to list all files used by collection version
Author: login: codetheweb
Created: 2025-05-06T22:02:17Z | Merged: 2025-05-23T02:06:56Z

## Description of changes

Does what the title says. Given a version file and a version, the operator constructs a complete list of files used by the collection version (including block files by traversing all sparse indices).

## Test plan

_How are these changes tested?_

- [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust

Tested in the next PR.

## Documentation Changes

_Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_

n/a


Diff:
diff --git a/rust/blockstore/src/arrow/provider.rs b/rust/blockstore/src/arrow/provider.rs
index 1540fc3e5ab..e421cecf6de 100644
--- a/rust/blockstore/src/arrow/provider.rs
+++ b/rust/blockstore/src/arrow/provider.rs
@@ -469,6 +469,15 @@ pub struct RootManager {
     prefetched_roots: Arc<parking_lot::Mutex<HashMap<Uuid, Duration>>>,
 }
 
+impl std::fmt::Debug for RootManager {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("RootManager")
+            .field("cache", &self.cache)
+            .field("storage", &self.storage)
+            .finish()
+    }
+}
+
 impl RootManager {
     pub fn new(storage: Storage, cache: Box<dyn PersistentCache<Uuid, RootReader>>) -> Self {
         let cache: Arc<dyn PersistentCache<Uuid, RootReader>> = cache.into();
diff --git a/rust/garbage_collector/src/operators/list_files_at_version.rs b/rust/garbage_collector/src/operators/list_files_at_version.rs
new file mode 100644
index 00000000000..5bff14e9ed0
--- /dev/null
+++ b/rust/garbage_collector/src/operators/list_files_at_version.rs
@@ -0,0 +1,160 @@
+use async_trait::async_trait;
+use chroma_blockstore::{arrow::provider::RootManagerError, RootManager};
+use chroma_system::{Operator, OperatorType};
+use chroma_types::{chroma_proto::CollectionVersionFile, CollectionUuid, HNSW_PATH};
+use std::{collections::HashSet, str::FromStr};
+use thiserror::Error;
+use tokio::task::{JoinError, JoinSet};
+use uuid::Uuid;
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionInput {
+    root_manager: RootManager,
+    version_file: CollectionVersionFile,
+    version: i64,
+}
+
+impl ListFilesAtVersionInput {
+    pub fn new(
+        root_manager: RootManager,
+        version_file: CollectionVersionFile,
+        version: i64,
+    ) -> Self {
+        Self {
+            root_manager,
+            version_file,
+            version,
+        }
+    }
+}
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionOutput {
+    pub collection_id: CollectionUuid,
+    pub version: i64,
+    pub file_paths: HashSet<String>,
+}
+
+#[derive(Debug, Error)]
+pub enum ListFilesAtVersionError {
+    #[error("Version history field missing")]
+    VersionHistoryMissing,
+    #[error("Version {0} not found")]
+    VersionNotFound(i64),
+    #[error("Invalid UUID: {0}")]
+    InvalidUuid(uuid::Error),
+    #[error("Sparse index fetch task failed: {0}")]
+    SparseIndexTaskFailed(JoinError),
+    #[error("Error fetching block IDs for sparse index: {0}")]
+    FetchBlockIdsError(#[from] RootManagerError),
+    #[error("Version file missing collection ID")]
+    VersionFileMissingCollectionId,
+}
+
+#[derive(Clone, Debug)]
+pub struct ListFilesAtVersionsOperator {}
+
+#[async_trait]
+impl Operator<ListFilesAtVersionInput, ListFilesAtVersionOutput> for ListFilesAtVersionsOperator {
+    type Error = ListFilesAtVersionError;
+
+    fn get_type(&self) -> OperatorType {
+        OperatorType::IO
+    }
+
+    async fn run(
+        &self,
+        input: &ListFilesAtVersionInput,
+    ) -> Result<ListFilesAtVersionOutput, Self::Error> {
+        let collection_id = CollectionUuid::from_str(
+            &input
+                .version_file
+                .collection_info_immutable
+                .as_ref()
+                .ok_or_else(|| ListFilesAtVersionError::VersionFileMissingCollectionId)?
+                .collection_id,
+        )
+        .map_err(ListFilesAtVersionError::InvalidUuid)?;
+
+        let version_history = input
+            .version_file
+            .version_history
+            .as_ref()
+            .ok_or_else(|| ListFilesAtVersionError::VersionHistoryMissing)?;
+
+        let mut file_paths = HashSet::new();
+        let mut sparse_index_ids = HashSet::new();
+
+        let version = version_history
+            .versions
+            .iter()
+            .find(|v| v.version == input.version)
+            .ok_or_else(|| ListFilesAtVersionError::VersionNotFound(input.version))?;
+
+        tracing::debug!(
+            "Listing files at version {} for collection {}.",
+            version.version,
+            collection_id,
+        );
+        tracing::trace!(
+            "Processing version {:#?} for collection {}",
+            version,
+            collection_id
+        );
+
+        if let Some(segment_info) = &version.segment_info {
+            for segment in &segment_info.segment_compaction_info {
+                for (file_type, segment_paths) in &segment.file_paths {
+                    if file_type == "hnsw_index" || file_type == HNSW_PATH {
+                        for path in &segment_paths.paths {
+                            for hnsw_file in [
+                                "header.bin",
+                                "data_level0.bin",
+                                "length.bin",
+                                "link_lists.bin",
+                            ] {
+                                // Path construction here will need to be updated after the upcoming collection file prefix changes.
+                                file_paths.insert(format!("hnsw/{}/{}", path, hnsw_file));
+                            }
+                        }
+                    } else {
+                        // Must be a sparse index
+                        for path in &segment_paths.paths {
+                            // Path construction here will need to be updated after the upcoming collection file prefix changes.
+                            file_paths.insert(format!("sparse_index/{}", path));
+
+                            let sparse_index_id = Uuid::parse_str(path)
+                                .map_err(ListFilesAtVersionError::InvalidUuid)?;
+
+                            sparse_index_ids.insert(sparse_index_id);
+                        }
+                    }
+                }
+            }
+        }
+
+        let mut block_id_tasks = JoinSet::new();
+        for sparse_index_id in sparse_index_ids {
+            let root_manager = input.root_manager.clone();
+            block_id_tasks
+                .spawn(async move { root_manager.get_all_block_ids(&sparse_index_id).await });
+        }
+
+        while let Some(res) = block_id_tasks.join_next().await {
+            let block_ids = res
+                .map_err(ListFilesAtVersionError::SparseIndexTaskFailed)?
+                .map_err(ListFilesAtVersionError::FetchBlockIdsError)?;
+
+            for block_id in block_ids {
+                // Path construction here will need to be updated after the upcoming collection file prefix changes.
+                file_paths.insert(format!("block/{}", block_id));
+            }
+        }
+
+        Ok(ListFilesAtVersionOutput {
+            collection_id,
+            version: input.version,
+            file_paths,
+        })
+    }
+}
diff --git a/rust/garbage_collector/src/operators/mod.rs b/rust/garbage_collector/src/operators/mod.rs
index 144752ca1f9..703bbe0c274 100644
--- a/rust/garbage_collector/src/operators/mod.rs
+++ b/rust/garbage_collector/src/operators/mod.rs
@@ -8,4 +8,5 @@ pub mod fetch_lineage_file;
 pub mod fetch_sparse_index_files;
 pub mod fetch_version_file;
 pub mod get_version_file_paths;
+pub mod list_files_at_version;
 pub mod mark_versions_at_sysdb;


Reviews:
user: login: Sicheng-Pan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T23:33:03Z
user: login: codetheweb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T17:38:51Z
user: login: propel-code-bot[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T17:35:43Z
user: login: sanketkedia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T20:43:31Z
user: login: sanketkedia
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T20:51:18Z
user: login: codetheweb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T20:53:29Z

Review comments:
diff_hunk: @@ -0,0 +1,144 @@
+use async_trait::async_trait;
+use chroma_blockstore::{arrow::provider::RootManagerError, RootManager};
+use chroma_system::{Operator, OperatorType};
+use chroma_types::{chroma_proto::CollectionVersionFile, CollectionUuid, HNSW_PATH};
+use std::{collections::HashSet, str::FromStr};
+use thiserror::Error;
+use tokio::task::{JoinError, JoinSet};
+use uuid::Uuid;
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionInput {
+    root_manager: RootManager,
+    version_file: CollectionVersionFile,
+    version: i64,
+}
+
+impl ListFilesAtVersionInput {
+    pub fn new(
+        root_manager: RootManager,
+        version_file: CollectionVersionFile,
+        version: i64,
+    ) -> Self {
+        Self {
+            root_manager,
+            version_file,
+            version,
+        }
+    }
+}
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionOutput {
+    pub collection_id: CollectionUuid,
+    pub version: i64,
+    pub file_paths: HashSet<String>,
+}
+
+#[derive(Debug, Error)]
+pub enum ListFilesAtVersionsError {
+    #[error("Version history field missing")]
+    VersionHistoryMissing,
+    #[error("Version {0} not found")]
+    VersionNotFound(i64),
+    #[error("Invalid UUID: {0}")]
+    InvalidUuid(uuid::Error),
+    #[error("Sparse index fetch task failed: {0}")]
+    SparseIndexTaskFailed(JoinError),
+    #[error("Error fetching block IDs for sparse index: {0}")]
+    FetchBlockIdsError(#[from] RootManagerError),
+    #[error("Version file missing collection ID")]
+    VersionFileMissingCollectionId,
+}
+
+#[derive(Clone, Debug)]
+pub struct ListFilesAtVersionsOperator {}
+
+#[async_trait]
+impl Operator<ListFilesAtVersionInput, ListFilesAtVersionOutput> for ListFilesAtVersionsOperator {
+    type Error = ListFilesAtVersionsError;
+
+    fn get_type(&self) -> OperatorType {
+        OperatorType::IO
+    }
+
+    async fn run(
+        &self,
+        input: &ListFilesAtVersionInput,
+    ) -> Result<ListFilesAtVersionOutput, Self::Error> {
+        let version_history = input
+            .version_file
+            .version_history
+            .as_ref()
+            .ok_or_else(|| ListFilesAtVersionsError::VersionHistoryMissing)?;
+
+        let mut file_paths = HashSet::new();
+        let mut sparse_index_ids = HashSet::new();
+
+        let version = version_history
+            .versions
+            .iter()
+            .find(|v| v.version == input.version)
+            .ok_or_else(|| ListFilesAtVersionsError::VersionNotFound(input.version))?;
+
+        if let Some(segment_info) = &version.segment_info {
+            for segment in &segment_info.segment_compaction_info {
+                for (file_type, segment_paths) in &segment.file_paths {
+                    if file_type == "hnsw_index" || file_type == HNSW_PATH {
+                        for path in &segment_paths.paths {
+                            for hnsw_file in [
+                                "header.bin",
+                                "data_level0.bin",
+                                "length.bin",
+                                "link_lists.bin",
+                            ] {
+                                file_paths.insert(format!("hnsw/{}/{}", path, hnsw_file));
+                            }
+                        }
+                    } else {
+                        // Must be a sparse index
+                        for path in &segment_paths.paths {
+                            file_paths.insert(format!("sparse_index/{}", path));
+
+                            let sparse_index_id = Uuid::parse_str(path)
+                                .map_err(ListFilesAtVersionsError::InvalidUuid)?;
+
+                            sparse_index_ids.insert(sparse_index_id);
+                        }
+                    }
+                }
+            }
+        }
+
+        let mut block_id_tasks = JoinSet::new();
+        for sparse_index_id in sparse_index_ids {
+            let root_manager = input.root_manager.clone();
+            block_id_tasks
+                .spawn(async move { root_manager.get_all_block_ids(&sparse_index_id).await });
+        }
+
+        while let Some(res) = block_id_tasks.join_next().await {
+            let block_ids = res
+                .map_err(ListFilesAtVersionsError::SparseIndexTaskFailed)?
+                .map_err(ListFilesAtVersionsError::FetchBlockIdsError)?;
+
+            for block_id in block_ids {
+                file_paths.insert(format!("block/{}", block_id));
path: rust/garbage_collector/src/operators/list_files_at_version.rs
user: login: Sicheng-Pan
body: is the path here subject to change?
created_at: 2025-05-08T23:33:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,144 @@
+use async_trait::async_trait;
+use chroma_blockstore::{arrow::provider::RootManagerError, RootManager};
+use chroma_system::{Operator, OperatorType};
+use chroma_types::{chroma_proto::CollectionVersionFile, CollectionUuid, HNSW_PATH};
+use std::{collections::HashSet, str::FromStr};
+use thiserror::Error;
+use tokio::task::{JoinError, JoinSet};
+use uuid::Uuid;
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionInput {
+    root_manager: RootManager,
+    version_file: CollectionVersionFile,
+    version: i64,
+}
+
+impl ListFilesAtVersionInput {
+    pub fn new(
+        root_manager: RootManager,
+        version_file: CollectionVersionFile,
+        version: i64,
+    ) -> Self {
+        Self {
+            root_manager,
+            version_file,
+            version,
+        }
+    }
+}
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionOutput {
+    pub collection_id: CollectionUuid,
+    pub version: i64,
+    pub file_paths: HashSet<String>,
+}
+
+#[derive(Debug, Error)]
+pub enum ListFilesAtVersionsError {
+    #[error("Version history field missing")]
+    VersionHistoryMissing,
+    #[error("Version {0} not found")]
+    VersionNotFound(i64),
+    #[error("Invalid UUID: {0}")]
+    InvalidUuid(uuid::Error),
+    #[error("Sparse index fetch task failed: {0}")]
+    SparseIndexTaskFailed(JoinError),
+    #[error("Error fetching block IDs for sparse index: {0}")]
+    FetchBlockIdsError(#[from] RootManagerError),
+    #[error("Version file missing collection ID")]
+    VersionFileMissingCollectionId,
+}
+
+#[derive(Clone, Debug)]
+pub struct ListFilesAtVersionsOperator {}
+
+#[async_trait]
+impl Operator<ListFilesAtVersionInput, ListFilesAtVersionOutput> for ListFilesAtVersionsOperator {
+    type Error = ListFilesAtVersionsError;
+
+    fn get_type(&self) -> OperatorType {
+        OperatorType::IO
+    }
+
+    async fn run(
+        &self,
+        input: &ListFilesAtVersionInput,
+    ) -> Result<ListFilesAtVersionOutput, Self::Error> {
+        let version_history = input
+            .version_file
+            .version_history
+            .as_ref()
+            .ok_or_else(|| ListFilesAtVersionsError::VersionHistoryMissing)?;
+
+        let mut file_paths = HashSet::new();
+        let mut sparse_index_ids = HashSet::new();
+
+        let version = version_history
+            .versions
+            .iter()
+            .find(|v| v.version == input.version)
+            .ok_or_else(|| ListFilesAtVersionsError::VersionNotFound(input.version))?;
+
+        if let Some(segment_info) = &version.segment_info {
+            for segment in &segment_info.segment_compaction_info {
+                for (file_type, segment_paths) in &segment.file_paths {
+                    if file_type == "hnsw_index" || file_type == HNSW_PATH {
+                        for path in &segment_paths.paths {
+                            for hnsw_file in [
+                                "header.bin",
+                                "data_level0.bin",
+                                "length.bin",
+                                "link_lists.bin",
+                            ] {
+                                file_paths.insert(format!("hnsw/{}/{}", path, hnsw_file));
+                            }
+                        }
+                    } else {
+                        // Must be a sparse index
+                        for path in &segment_paths.paths {
+                            file_paths.insert(format!("sparse_index/{}", path));
+
+                            let sparse_index_id = Uuid::parse_str(path)
+                                .map_err(ListFilesAtVersionsError::InvalidUuid)?;
+
+                            sparse_index_ids.insert(sparse_index_id);
+                        }
+                    }
+                }
+            }
+        }
+
+        let mut block_id_tasks = JoinSet::new();
+        for sparse_index_id in sparse_index_ids {
+            let root_manager = input.root_manager.clone();
+            block_id_tasks
+                .spawn(async move { root_manager.get_all_block_ids(&sparse_index_id).await });
+        }
+
+        while let Some(res) = block_id_tasks.join_next().await {
+            let block_ids = res
+                .map_err(ListFilesAtVersionsError::SparseIndexTaskFailed)?
+                .map_err(ListFilesAtVersionsError::FetchBlockIdsError)?;
+
+            for block_id in block_ids {
+                file_paths.insert(format!("block/{}", block_id));
path: rust/garbage_collector/src/operators/list_files_at_version.rs
user: login: codetheweb
body: yes, can leave a comment
created_at: 2025-05-09T17:38:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -469,6 +469,15 @@ pub struct RootManager {
     prefetched_roots: Arc<parking_lot::Mutex<HashMap<Uuid, Duration>>>,
 }
 
+impl std::fmt::Debug for RootManager {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("RootManager")
+            .field("cache", &self.cache)
+            .field("storage", &self.storage)
+            .finish()
path: rust/blockstore/src/arrow/provider.rs
user: login: propel-code-bot[bot]
body: [**BestPractice**]

In the `fmt` implementation for `RootManager`, you're not including the `prefetched_roots` field in the debug output. For completeness in the debug representation, consider including it with an appropriate format that doesn't expose sensitive information.
created_at: 2025-05-13T17:35:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,160 @@
+use async_trait::async_trait;
+use chroma_blockstore::{arrow::provider::RootManagerError, RootManager};
+use chroma_system::{Operator, OperatorType};
+use chroma_types::{chroma_proto::CollectionVersionFile, CollectionUuid, HNSW_PATH};
+use std::{collections::HashSet, str::FromStr};
+use thiserror::Error;
+use tokio::task::{JoinError, JoinSet};
+use uuid::Uuid;
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionInput {
+    root_manager: RootManager,
+    version_file: CollectionVersionFile,
+    version: i64,
+}
+
+impl ListFilesAtVersionInput {
+    pub fn new(
+        root_manager: RootManager,
+        version_file: CollectionVersionFile,
+        version: i64,
+    ) -> Self {
+        Self {
+            root_manager,
+            version_file,
+            version,
+        }
+    }
+}
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionOutput {
+    pub collection_id: CollectionUuid,
+    pub version: i64,
+    pub file_paths: HashSet<String>,
+}
+
+#[derive(Debug, Error)]
+pub enum ListFilesAtVersionError {
+    #[error("Version history field missing")]
+    VersionHistoryMissing,
+    #[error("Version {0} not found")]
+    VersionNotFound(i64),
+    #[error("Invalid UUID: {0}")]
+    InvalidUuid(uuid::Error),
+    #[error("Sparse index fetch task failed: {0}")]
+    SparseIndexTaskFailed(JoinError),
+    #[error("Error fetching block IDs for sparse index: {0}")]
+    FetchBlockIdsError(#[from] RootManagerError),
+    #[error("Version file missing collection ID")]
+    VersionFileMissingCollectionId,
+}
+
+#[derive(Clone, Debug)]
+pub struct ListFilesAtVersionsOperator {}
+
+#[async_trait]
+impl Operator<ListFilesAtVersionInput, ListFilesAtVersionOutput> for ListFilesAtVersionsOperator {
+    type Error = ListFilesAtVersionError;
+
+    fn get_type(&self) -> OperatorType {
+        OperatorType::IO
+    }
+
+    async fn run(
+        &self,
+        input: &ListFilesAtVersionInput,
+    ) -> Result<ListFilesAtVersionOutput, Self::Error> {
+        let collection_id = CollectionUuid::from_str(
+            &input
+                .version_file
+                .collection_info_immutable
+                .as_ref()
+                .ok_or_else(|| ListFilesAtVersionError::VersionFileMissingCollectionId)?
+                .collection_id,
+        )
+        .map_err(ListFilesAtVersionError::InvalidUuid)?;
+
+        let version_history = input
+            .version_file
+            .version_history
+            .as_ref()
+            .ok_or_else(|| ListFilesAtVersionError::VersionHistoryMissing)?;
+
+        let mut file_paths = HashSet::new();
+        let mut sparse_index_ids = HashSet::new();
+
+        let version = version_history
+            .versions
+            .iter()
+            .find(|v| v.version == input.version)
+            .ok_or_else(|| ListFilesAtVersionError::VersionNotFound(input.version))?;
+
+        tracing::debug!(
+            "Listing files at version {} for collection {}.",
+            version.version,
+            collection_id,
+        );
+        tracing::trace!(
path: rust/garbage_collector/src/operators/list_files_at_version.rs
user: login: sanketkedia
body: why two traces here?
created_at: 2025-05-22T20:43:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,160 @@
+use async_trait::async_trait;
+use chroma_blockstore::{arrow::provider::RootManagerError, RootManager};
+use chroma_system::{Operator, OperatorType};
+use chroma_types::{chroma_proto::CollectionVersionFile, CollectionUuid, HNSW_PATH};
+use std::{collections::HashSet, str::FromStr};
+use thiserror::Error;
+use tokio::task::{JoinError, JoinSet};
+use uuid::Uuid;
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionInput {
+    root_manager: RootManager,
+    version_file: CollectionVersionFile,
+    version: i64,
+}
+
+impl ListFilesAtVersionInput {
+    pub fn new(
+        root_manager: RootManager,
+        version_file: CollectionVersionFile,
+        version: i64,
+    ) -> Self {
+        Self {
+            root_manager,
+            version_file,
+            version,
+        }
+    }
+}
+
+#[derive(Debug)]
+pub struct ListFilesAtVersionOutput {
+    pub collection_id: CollectionUuid,
+    pub version: i64,
+    pub file_paths: HashSet<String>,
+}
+
+#[derive(Debug, Error)]
+pub enum ListFilesAtVersionError {
+    #[error("Version history field missing")]
+    VersionHistoryMissing,
+    #[error("Version {0} not found")]
+    VersionNotFound(i64),
+    #[error("Invalid UUID: {0}")]
+    InvalidUuid(uuid::Error),
+    #[error("Sparse index fetch task failed: {0}")]
+    SparseIndexTaskFailed(JoinError),
+    #[error("Error fetching block IDs for sparse index: {0}")]
+    FetchBlockIdsError(#[from] RootManagerError),
+    #[error("Version file missing collection ID")]
+    VersionFileMissingCollectionId,
+}
+
+#[derive(Clone, Debug)]
+pub struct ListFilesAtVersionsOperator {}
+
+#[async_trait]
+impl Operator<ListFilesAtVersionInput, ListFilesAtVersionOutput> for ListFilesAtVersionsOperator {
+    type Error = ListFilesAtVersionError;
+
+    fn get_type(&self) -> OperatorType {
+        OperatorType::IO
+    }
+
+    async fn run(
+        &self,
+        input: &ListFilesAtVersionInput,
+    ) -> Result<ListFilesAtVersionOutput, Self::Error> {
+        let collection_id = CollectionUuid::from_str(
+            &input
+                .version_file
+                .collection_info_immutable
+                .as_ref()
+                .ok_or_else(|| ListFilesAtVersionError::VersionFileMissingCollectionId)?
+                .collection_id,
+        )
+        .map_err(ListFilesAtVersionError::InvalidUuid)?;
+
+        let version_history = input
+            .version_file
+            .version_history
+            .as_ref()
+            .ok_or_else(|| ListFilesAtVersionError::VersionHistoryMissing)?;
+
+        let mut file_paths = HashSet::new();
+        let mut sparse_index_ids = HashSet::new();
+
+        let version = version_history
+            .versions
+            .iter()
+            .find(|v| v.version == input.version)
+            .ok_or_else(|| ListFilesAtVersionError::VersionNotFound(input.version))?;
+
+        tracing::debug!(
+            "Listing files at version {} for collection {}.",
+            version.version,
+            collection_id,
+        );
+        tracing::trace!(
path: rust/garbage_collector/src/operators/list_files_at_version.rs
user: login: codetheweb
body: they have different log levels
the second one is very verbose
created_at: 2025-05-22T20:53:29Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-06T22:02:26Z
body: # Reviewer Checklist
  Please leverage this checklist to ensure your code review is thorough before approving
  ## Testing, Bugs, Errors, Logs, Documentation
  - [ ] Can you think of any use case in which the code does not behave as intended? Have they been tested?
  - [ ] Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  - [ ] If appropriate, are there adequate property based tests?
  - [ ] If appropriate, are there adequate unit tests?
  - [ ] Should any logging, debugging, tracing information be added or removed?
  - [ ] Are error messages user-friendly?
  - [ ] Have all documentation changes needed been made?
  - [ ] Have all non-obvious changes been commented?
  ## System Compatibility
  - [ ] Are there any potential impacts on other parts of the system or backward compatibility?
  - [ ] Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?
  ## Quality
  - [ ] Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)
author_association: NONE
user: login: codetheweb
created_at: 2025-05-06T22:02:35Z
body: * **#4671** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4671?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4627** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4627?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4605** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4605?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4607** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4607?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4586** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4586?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4468** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4468?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4579** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4579?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4466** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4466?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a> 👈 <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4466?utm_source=stack-comment-view-in-graphite" target="_blank">(View in Graphite)</a>
* **#4464** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4464?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4463** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4463?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* **#4432** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4432?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>: 1 other dependent PR ([#4447](https://github.com/chroma-core/chroma/pull/4447) <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4447?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>)
* **#4557** <a href="https://app.graphite.dev/github/pr/chroma-core/chroma/4557?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* `main`


<h2></h2>

This stack of pull requests is managed by <a href="https://graphite.dev?utm-source=stack-comment"><b>Graphite</b></a>. Learn more about <a href="https://stacking.dev/?utm_source=stack-comment">stacking</a>.
<!-- Current dependencies on/for this PR: -->
author_association: CONTRIBUTOR
user: login: propel-code-bot[bot]
created_at: 2025-05-13T17:35:45Z
body: <!-- Summary by @propel-code-bot -->

**Add Operator to List All Files Used by a Collection Version (for Garbage Collection)**

This PR introduces a new garbage collection operator that, given a collection version file and version number, enumerates all file paths associated with that collection version, including block files (by traversing sparse indices), HNSW index files, and sparse index files. Additionally, it registers the new operator in the module system and improves RootManager's debug output by implementing a Debug trait.

**Key Changes:**
• Added rust/garbage_collector/src/operators/list_files_at_version.rs: Implements ListFilesAtVersionsOperator to output all files relevant for a specific collection version, including careful traversal of version metadata and asynchronous fetching of block IDs from RootManager.
• Registered the new operator in rust/garbage_collector/src/operators/mod.rs.
• Implemented std::fmt::Debug for RootManager in rust/blockstore/src/arrow/provider.rs, aiding diagnostics.


**Affected Areas:**
• Garbage collector operators
• Blockstore's RootManager (Debug trait implementation)
• Module registration


*This summary was automatically generated by @propel-code-bot*
author_association: CONTRIBUTOR
user: login: codetheweb
created_at: 2025-05-23T02:06:57Z
body: ### Merge activity

* **May 23, 2:06 AM UTC**: @codetheweb merged this pull request with [Graphite](https://app.graphite.dev/github/pr/chroma-core/chroma/4466).

author_association: CONTRIBUTOR