Repository: delta-io/delta-kernel-rs
Pull request #894: feat!: Add in-commit timestamp table feature
Author: login: OussamaSaoudi
Labels: name: breaking-change
description: Public API change that could cause downstream compilation failures. Requires a major version bump.
Created: 2025-04-28T05:03:22Z | Merged: 2025-04-30T17:38:45Z

## What changes are proposed in this pull request?

This PR adds checks to TableConfiguration for in-commit timestamps. The following are introduced: 
- `in_commit_timestamps_supported`: true if ICT is supported
- `in_commit_timestamps_enabled`: true if ICT is enabled
- `in_commit_timestamp_enablement`: Gets enablement version and timestamp

## Breaking changes
This adds a WriterFeature, making it a breaking change.

## How was this change tested?
Check the following cases:
- Supported but not enabled
- Supported and enabled, but is missing the enablement timestamp
- Fully supported and Enabled.


Diff:
diff --git a/kernel/src/table_configuration.rs b/kernel/src/table_configuration.rs
index 524439af66..b947fce1a5 100644
--- a/kernel/src/table_configuration.rs
+++ b/kernel/src/table_configuration.rs
@@ -278,6 +278,63 @@ impl TableConfiguration {
             .has_writer_feature(&WriterFeature::V2Checkpoint);
         read_supported && write_supported
     }
+
+    /// Returns `true` if the table supports writing in-commit timestamps.
+    ///
+    /// To support this feature the table must:
+    /// - Have a min_writer_version of 7
+    /// - Have the [`WriterFeature::InCommitTimestamp`] writer feature.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_supported(&self) -> bool {
+        self.protocol().min_writer_version() == 7
+            && self
+                .protocol()
+                .has_writer_feature(&WriterFeature::InCommitTimestamp)
+    }
+
+    /// Returns `true` if in-commit timestamps is supported and it is enabled. In-commit timestamps
+    /// is enabled when the `delta.enableInCommitTimestamps` configuration is set to `true`.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_enabled(&self) -> bool {
+        self.is_in_commit_timestamps_supported()
+            && self
+                .table_properties()
+                .enable_in_commit_timestamps
+                .unwrap_or(false)
+    }
+
+    /// If in-commit timestamps is enabled, returns a tuple of the in-commit timestamp enablement
+    /// version and timestamp.
+    ///
+    /// If in-commit timestamps is not supported, or not enabled, this returns `None`.
+    /// If in-commit timestams is enabled, but the enablement version or timestamp is not present,
+    /// this returns an error.
+    #[allow(unused)]
+    pub(crate) fn in_commit_timestamp_enablement(&self) -> DeltaResult<Option<(Version, i64)>> {
+        if !self.is_in_commit_timestamps_enabled() {
+            return Ok(None);
+        }
+        let enablement_version = self
+            .table_properties()
+            .in_commit_timestamp_enablement_version;
+        let enablement_timestamp = self
+            .table_properties()
+            .in_commit_timestamp_enablement_timestamp;
+
+        let ict_error = |err: &str| {
+            Error::generic(format!(
+                "In-commit timestamp enabled, but missing Enablement version or timestamp. {err}"
+            ))
+        };
+        match (enablement_version, enablement_timestamp) {
+            (Some(version), Some(timestamp)) => Ok(Some((version, timestamp))),
+            (Some(_), None) => Err(ict_error("Enablement timestamp is not present")),
+            (None, Some(_)) => Err(ict_error("Enablement version is not present")),
+            (None, None) => Err(ict_error(
+                "Enablement version and timestamp are not present.",
+            )),
+        }
+    }
 }
 
 #[cfg(test)]
@@ -289,6 +346,7 @@ mod test {
     use crate::actions::{Metadata, Protocol};
     use crate::table_features::{ReaderFeature, WriterFeature};
     use crate::table_properties::TableProperties;
+    use crate::Error;
 
     use super::TableConfiguration;
 
@@ -341,6 +399,75 @@ mod test {
         assert!(table_config.is_deletion_vector_enabled());
     }
     #[test]
+    fn ict_supported_and_enabled() {
+        let metadata = Metadata {
+            schema_string: r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#.to_string(),
+            configuration: HashMap::from_iter([(
+                "delta.enableInCommitTimestamps".to_string(),
+                "true".to_string(),
+            ),
+                ("delta.inCommitTimestampEnablementVersion".to_string(), "5".to_string()),
+                ("delta.inCommitTimestampEnablementTimestamp".to_string(), "100".to_string())]),
+            ..Default::default()
+        };
+        let protocol = Protocol::try_new(
+            3,
+            7,
+            Some::<Vec<String>>(vec![]),
+            Some([WriterFeature::InCommitTimestamp]),
+        )
+        .unwrap();
+        let table_root = Url::try_from("file:///").unwrap();
+        let table_config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap();
+        assert!(table_config.is_in_commit_timestamps_supported());
+        assert!(table_config.is_in_commit_timestamps_enabled());
+        let enablement = table_config.in_commit_timestamp_enablement().unwrap();
+        assert_eq!(enablement, Some((5, 100)))
+    }
+    #[test]
+    fn ict_supported_and_enabled_without_enablement_info() {
+        let metadata = Metadata {
+            schema_string: r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#.to_string(),
+            configuration: HashMap::from_iter([(
+                "delta.enableInCommitTimestamps".to_string(),
+                "true".to_string(),
+            )]),
+            ..Default::default()
+        };
+        let protocol = Protocol::try_new(
+            3,
+            7,
+            Some::<Vec<String>>(vec![]),
+            Some([WriterFeature::InCommitTimestamp]),
+        )
+        .unwrap();
+        let table_root = Url::try_from("file:///").unwrap();
+        let table_config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap();
+        assert!(table_config.is_in_commit_timestamps_supported());
+        assert!(table_config.is_in_commit_timestamps_enabled());
+        assert!(matches!(
+                table_config.in_commit_timestamp_enablement(),
+                Err(Error::Generic(msg)) if msg.contains("Enablement version and timestamp are not present.")));
+    }
+    #[test]
+    fn ict_supported_and_not_enabled() {
+        let metadata = Metadata {
+            schema_string: r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#.to_string(),
+            ..Default::default()
+        };
+        let protocol = Protocol::try_new(
+            3,
+            7,
+            Some::<Vec<String>>(vec![]),
+            Some([WriterFeature::InCommitTimestamp]),
+        )
+        .unwrap();
+        let table_root = Url::try_from("file:///").unwrap();
+        let table_config = TableConfiguration::try_new(metadata, protocol, table_root, 0).unwrap();
+        assert!(table_config.is_in_commit_timestamps_supported());
+        assert!(!table_config.is_in_commit_timestamps_enabled());
+    }
+    #[test]
     fn fails_on_unsupported_feature() {
         let metadata = Metadata {
             schema_string: r#"{"type":"struct","fields":[{"name":"value","type":"integer","nullable":true,"metadata":{}}]}"#.to_string(),
diff --git a/kernel/src/table_features/mod.rs b/kernel/src/table_features/mod.rs
index 761dc84025..38c909cf99 100644
--- a/kernel/src/table_features/mod.rs
+++ b/kernel/src/table_features/mod.rs
@@ -90,6 +90,8 @@ pub enum WriterFeature {
     ColumnMapping,
     /// ID Columns
     IdentityColumns,
+    /// Monotonically increasing timestamps in the CommitInfo
+    InCommitTimestamp,
     /// Deletion vectors for merge, update, delete
     DeletionVectors,
     /// Row tracking on tables
@@ -244,6 +246,7 @@ mod tests {
             (WriterFeature::GeneratedColumns, "generatedColumns"),
             (WriterFeature::ColumnMapping, "columnMapping"),
             (WriterFeature::IdentityColumns, "identityColumns"),
+            (WriterFeature::InCommitTimestamp, "inCommitTimestamp"),
             (WriterFeature::DeletionVectors, "deletionVectors"),
             (WriterFeature::RowTracking, "rowTracking"),
             (WriterFeature::TimestampWithoutTimezone, "timestampNtz"),


Reviews:
user: login: hntd187
body: LGMT
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-28T13:11:03Z
user: login: OussamaSaoudi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T16:02:59Z
user: login: scovich
body: LGTM
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-28T21:45:43Z
user: login: zachschuermann
body: LGTM just a couple quick things!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-29T16:23:16Z
user: login: OussamaSaoudi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-30T03:55:21Z

Review comments:
diff_hunk: @@ -263,6 +263,63 @@ impl TableConfiguration {
             version => (2..=6).contains(&version),
         }
     }
+
+    /// Returns `true` if the table supports writing in-commit timestamps.
+    ///
+    /// To support this feature the table must:
+    /// - Have a min_writer_version of 7
+    /// - Have the [`WriterFeature::InCommitTimestamps`] writer feature.
+    #[allow(unused)]
path: kernel/src/table_configuration.rs
user: login: hntd187
body: These will no longer be necessary when the stacked PR areas are merged right?
created_at: 2025-04-28T13:10:44Z
author_association: COLLABORATOR
diff_hunk: @@ -263,6 +263,63 @@ impl TableConfiguration {
             version => (2..=6).contains(&version),
         }
     }
+
+    /// Returns `true` if the table supports writing in-commit timestamps.
+    ///
+    /// To support this feature the table must:
+    /// - Have a min_writer_version of 7
+    /// - Have the [`WriterFeature::InCommitTimestamps`] writer feature.
+    #[allow(unused)]
path: kernel/src/table_configuration.rs
user: login: OussamaSaoudi
body: Yeah the allow unused will be removed eventually 👍
created_at: 2025-04-28T16:02:59Z
author_association: COLLABORATOR
diff_hunk: @@ -263,6 +263,63 @@ impl TableConfiguration {
             version => (2..=6).contains(&version),
         }
     }
+
+    /// Returns `true` if the table supports writing in-commit timestamps.
+    ///
+    /// To support this feature the table must:
+    /// - Have a min_writer_version of 7
+    /// - Have the [`WriterFeature::InCommitTimestamps`] writer feature.
+    #[allow(unused)]
path: kernel/src/table_configuration.rs
user: login: zachschuermann
body: aside: I've been thinking about the structure of `TableConfiguration` etc. and I wonder if these actually will all move to `ResolvedTable`/`ResolvedMetadata` in the future
created_at: 2025-04-29T16:16:31Z
author_association: COLLABORATOR
diff_hunk: @@ -263,6 +263,63 @@ impl TableConfiguration {
             version => (2..=6).contains(&version),
         }
     }
+
+    /// Returns `true` if the table supports writing in-commit timestamps.
+    ///
+    /// To support this feature the table must:
+    /// - Have a min_writer_version of 7
+    /// - Have the [`WriterFeature::InCommitTimestamps`] writer feature.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_supported(&self) -> bool {
+        self.protocol().min_writer_version() == 7
+            && self
+                .protocol()
+                .has_writer_feature(&WriterFeature::InCommitTimestamps)
+    }
+
+    /// Returns `true` if in-commit timestamps is supported and it is enabled. In-commit timestamps
+    /// is enabled when the `delta.enableInCommitTimestamps` configuration is set to `true`.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_enabled(&self) -> bool {
+        self.is_in_commit_timestamps_supported()
+            && self
+                .table_properties()
+                .enable_in_commit_timestamps
+                .unwrap_or(false)
+    }
+
+    /// If in-commit timestamps is enabled, this retrieves the in-commit timestamp enablement version
+    /// and timestamps.
path: kernel/src/table_configuration.rs
user: login: zachschuermann
body: nit: 
```suggestion
    /// and timestamp.
```
created_at: 2025-04-29T16:18:17Z
author_association: COLLABORATOR
diff_hunk: @@ -263,6 +263,63 @@ impl TableConfiguration {
             version => (2..=6).contains(&version),
         }
     }
+
+    /// Returns `true` if the table supports writing in-commit timestamps.
+    ///
+    /// To support this feature the table must:
+    /// - Have a min_writer_version of 7
+    /// - Have the [`WriterFeature::InCommitTimestamps`] writer feature.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_supported(&self) -> bool {
+        self.protocol().min_writer_version() == 7
+            && self
+                .protocol()
+                .has_writer_feature(&WriterFeature::InCommitTimestamps)
+    }
+
+    /// Returns `true` if in-commit timestamps is supported and it is enabled. In-commit timestamps
+    /// is enabled when the `delta.enableInCommitTimestamps` configuration is set to `true`.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_enabled(&self) -> bool {
+        self.is_in_commit_timestamps_supported()
+            && self
+                .table_properties()
+                .enable_in_commit_timestamps
+                .unwrap_or(false)
+    }
+
+    /// If in-commit timestamps is enabled, this retrieves the in-commit timestamp enablement version
+    /// and timestamps.
+    ///
+    /// If in-commit timestamps is not supported, or not enabled, this returns `None`.
+    /// If in-commit timestams is enabled, but the enablement version or timestamp is not present,
+    /// this returns an error.
+    #[allow(unused)]
+    pub(crate) fn in_commit_timestamp_enablement(&self) -> DeltaResult<Option<(Version, i64)>> {
path: kernel/src/table_configuration.rs
user: login: zachschuermann
body: nit: can we add a specific "returns a tuple of ..."
created_at: 2025-04-29T16:18:58Z
author_association: COLLABORATOR
diff_hunk: @@ -263,6 +263,63 @@ impl TableConfiguration {
             version => (2..=6).contains(&version),
         }
     }
+
+    /// Returns `true` if the table supports writing in-commit timestamps.
+    ///
+    /// To support this feature the table must:
+    /// - Have a min_writer_version of 7
+    /// - Have the [`WriterFeature::InCommitTimestamps`] writer feature.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_supported(&self) -> bool {
+        self.protocol().min_writer_version() == 7
+            && self
+                .protocol()
+                .has_writer_feature(&WriterFeature::InCommitTimestamps)
+    }
+
+    /// Returns `true` if in-commit timestamps is supported and it is enabled. In-commit timestamps
+    /// is enabled when the `delta.enableInCommitTimestamps` configuration is set to `true`.
+    #[allow(unused)]
+    pub(crate) fn is_in_commit_timestamps_enabled(&self) -> bool {
+        self.is_in_commit_timestamps_supported()
+            && self
+                .table_properties()
+                .enable_in_commit_timestamps
+                .unwrap_or(false)
+    }
+
+    /// If in-commit timestamps is enabled, this retrieves the in-commit timestamp enablement version
+    /// and timestamps.
+    ///
+    /// If in-commit timestamps is not supported, or not enabled, this returns `None`.
+    /// If in-commit timestams is enabled, but the enablement version or timestamp is not present,
+    /// this returns an error.
+    #[allow(unused)]
+    pub(crate) fn in_commit_timestamp_enablement(&self) -> DeltaResult<Option<(Version, i64)>> {
path: kernel/src/table_configuration.rs
user: login: OussamaSaoudi
body: modified the comment to be "returns tuple of ..." instead of "retrieves ..."
created_at: 2025-04-30T03:55:21Z
author_association: COLLABORATOR

PR comments:
user: login: codecov[bot]
created_at: 2025-04-28T05:05:21Z
body: ## [Codecov](https://app.codecov.io/gh/delta-io/delta-kernel-rs/pull/894?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io) Report
Attention: Patch coverage is `97.11538%` with `3 lines` in your changes missing coverage. Please review.
> Project coverage is 85.06%. Comparing base [(`d6c4298`)](https://app.codecov.io/gh/delta-io/delta-kernel-rs/commit/d6c429870044be718a6fb8938141e9e503919b2e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io) to head [(`116a371`)](https://app.codecov.io/gh/delta-io/delta-kernel-rs/commit/116a3718cc3f9a6ebe0ca88b13fb00359ebf2668?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io).
> Report is 1 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/delta-io/delta-kernel-rs/pull/894?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io) | Patch % | Lines |
|---|---|---|
| [kernel/src/table\_configuration.rs](https://app.codecov.io/gh/delta-io/delta-kernel-rs/pull/894?src=pr&el=tree&filepath=kernel%2Fsrc%2Ftable_configuration.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io#diff-a2VybmVsL3NyYy90YWJsZV9jb25maWd1cmF0aW9uLnJz) | 97.08% | [3 Missing :warning: ](https://app.codecov.io/gh/delta-io/delta-kernel-rs/pull/894?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #894      +/-   ##
==========================================
+ Coverage   85.00%   85.06%   +0.05%     
==========================================
  Files          87       87              
  Lines       21451    21555     +104     
  Branches    21451    21555     +104     
==========================================
+ Hits        18234    18335     +101     
- Misses       2268     2271       +3     
  Partials      949      949              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/delta-io/delta-kernel-rs/pull/894?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=delta-io).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: OussamaSaoudi
created_at: 2025-04-30T03:59:56Z
body: Note: I changed the writer feature from `InCommitTimestamps` to `InCommitTimestamp`. Turns out the protocol was inconsistent. [source](https://github.com/delta-io/delta/pull/4471/files)
author_association: COLLABORATOR