Repository: slatedb/slatedb
Pull request #510: Add version bytes for SST and manifest byte formats
Author: login: criccomini
Labels: name: enhancement
description: New feature or request
name: size:L
description: This PR changes 100-499 lines, ignoring generated files.
Created: 2025-03-10T20:11:07Z | Merged: 2025-03-14T18:36:22Z

This PR adds very rudimentary version number handling for SSTs and manifests. My main goal is simply to get the version bytes in the right spot, so we can implement proper version handling going forward. The most important part of this PR are the byte sizes and locations in the byte formats.

Changes:

- Adds a u16 to the _end_ of an SST's on-disk byte format
- Adds a u16 to the _beginning_ of a manifest's on-disk byte format
- Adds version validation logic in `FlatBufferManifestCodec::decode`
- Adds version validation logic in `SsTableFormat::read_info`
- Adds an `InvalidVersion` error
- Adds tests for validation logic

A couple of notes/oddities:

- I opted to put the encode/decode logic inside the manifest's flat buffer codec because we don't currently have an intermediate builder for it the way that we do for SSTs (using `EncodedSsTableBuilder`).
- On the SST side, I opted to validate the version in `read_info` since you can't do anything without reading the SST info for an SST first.
- The SST bytes are at the end while they're at the beginning for the Manifest. This felt most intuitive to me, but if everyone is grossed out by it, I can put the version bytes at the heat of the SST or tail of the manifest.
- The current decoders simply validate the versions are identical. We only have a single version right now, so this is fine. The work to support cross-version decoding is left for a later PR.

The changes are a first step toward #446.


Diff:
diff --git a/src/error.rs b/src/error.rs
index 7a7237d51..56cdecddf 100644
--- a/src/error.rs
+++ b/src/error.rs
@@ -24,6 +24,9 @@ pub enum SlateDBError {
     #[error("Empty block")]
     EmptyBlock,
 
+    #[error("Empty manifest")]
+    EmptyManifest,
+
     #[error("Object store error: {0}")]
     ObjectStoreError(#[from] Arc<object_store::Error>),
 
@@ -112,6 +115,12 @@ pub enum SlateDBError {
 
     #[error("Database already exists: {msg}")]
     DatabaseAlreadyExists { msg: String },
+
+    #[error("Byte format version mismatch: expected {expected_version}, actual {actual_version}")]
+    InvalidVersion {
+        expected_version: u16,
+        actual_version: u16,
+    },
 }
 
 impl From<std::io::Error> for SlateDBError {
diff --git a/src/flatbuffer_types.rs b/src/flatbuffer_types.rs
index e4ed9ccdc..9acb5491e 100644
--- a/src/flatbuffer_types.rs
+++ b/src/flatbuffer_types.rs
@@ -1,7 +1,7 @@
 use std::collections::VecDeque;
 use std::time::{Duration, SystemTime, UNIX_EPOCH};
 
-use bytes::Bytes;
+use bytes::{BufMut, Bytes, BytesMut};
 use flatbuffers::{FlatBufferBuilder, ForwardsUOffset, InvalidFlatbuffer, Vector, WIPOffset};
 use ulid::Ulid;
 
@@ -30,6 +30,8 @@ use crate::flatbuffer_types::manifest_generated::{
 use crate::manifest::{ExternalDb, Manifest, ManifestCodec};
 use crate::utils::clamp_allocated_size_bytes;
 
+pub(crate) const MANIFEST_FORMAT_VERSION: u16 = 1;
+
 /// A wrapper around a `Bytes` buffer containing a FlatBuffer-encoded `SsTableIndex`.
 pub(crate) struct SsTableIndexOwned {
     data: Bytes,
@@ -106,7 +108,18 @@ impl ManifestCodec for FlatBufferManifestCodec {
     }
 
     fn decode(&self, bytes: &Bytes) -> Result<Manifest, SlateDBError> {
-        let manifest = flatbuffers::root::<ManifestV1>(bytes)?;
+        if bytes.len() < 2 {
+            return Err(SlateDBError::EmptyManifest);
+        }
+        let version = u16::from_be_bytes([bytes[0], bytes[1]]);
+        if version != MANIFEST_FORMAT_VERSION {
+            return Err(SlateDBError::InvalidVersion {
+                expected_version: MANIFEST_FORMAT_VERSION,
+                actual_version: version,
+            });
+        }
+        let unversioned_bytes = bytes.slice(2..);
+        let manifest = flatbuffers::root::<ManifestV1>(unversioned_bytes.as_ref())?;
         Ok(Self::manifest(&manifest))
     }
 }
@@ -405,7 +418,10 @@ impl<'b> DbFlatBufferBuilder<'b> {
             },
         );
         self.builder.finish(manifest, None);
-        Bytes::copy_from_slice(self.builder.finished_data())
+        let mut bytes = BytesMut::new();
+        bytes.put_u16(MANIFEST_FORMAT_VERSION);
+        bytes.put_slice(self.builder.finished_data());
+        bytes.into()
     }
 
     fn create_sst_info(&mut self, info: &SsTableInfo) -> Bytes {
@@ -461,18 +477,21 @@ pub(crate) mod test_utils {
 
 #[cfg(test)]
 mod tests {
-    use crate::checkpoint;
     use crate::db_state::{CoreDbState, SsTableId};
     use crate::flatbuffer_types::{FlatBufferManifestCodec, SsTableIndexOwned};
     use crate::manifest::{ExternalDb, Manifest, ManifestCodec};
+    use crate::{checkpoint, SlateDBError};
     use std::time::{Duration, SystemTime};
 
     use crate::flatbuffer_types::test_utils::assert_index_clamped;
     use crate::sst::SsTableFormat;
     use crate::test_utils::build_test_sst;
+    use bytes::{BufMut, BytesMut};
     use ulid::Ulid;
     use uuid::Uuid;
 
+    use super::{manifest_generated, MANIFEST_FORMAT_VERSION};
+
     #[test]
     fn test_should_encode_decode_manifest_checkpoints() {
         // given:
@@ -546,4 +565,66 @@ mod tests {
 
         assert_index_clamped(&clamped, &index);
     }
+
+    #[test]
+    fn test_should_validate_manifest_version() {
+        let codec = FlatBufferManifestCodec {};
+
+        // Create a valid manifest with current version
+        let mut fbb = flatbuffers::FlatBufferBuilder::new();
+
+        // Create minimal required fields for ManifestV1
+        let l0 = fbb.create_vector::<flatbuffers::WIPOffset<_>>(&[]);
+        let compacted = fbb.create_vector::<flatbuffers::WIPOffset<_>>(&[]);
+        let checkpoints = fbb.create_vector::<flatbuffers::WIPOffset<_>>(&[]);
+
+        let manifest = manifest_generated::ManifestV1::create(
+            &mut fbb,
+            &manifest_generated::ManifestV1Args {
+                manifest_id: 0,
+                external_dbs: None,
+                initialized: false,
+                writer_epoch: 0,
+                compactor_epoch: 0,
+                wal_id_last_compacted: 0,
+                wal_id_last_seen: 0,
+                l0_last_compacted: None,
+                l0: Some(l0),
+                compacted: Some(compacted),
+                checkpoints: Some(checkpoints),
+                last_l0_clock_tick: 0,
+                last_l0_seq: 0,
+            },
+        );
+        fbb.finish(manifest, None);
+        let fb_data = fbb.finished_data();
+
+        let mut bytes = BytesMut::with_capacity(2 + fb_data.len());
+        bytes.put_u16(MANIFEST_FORMAT_VERSION);
+        bytes.put_slice(fb_data);
+        let valid_bytes = bytes.freeze();
+
+        // Test valid version
+        match codec.decode(&valid_bytes) {
+            Ok(_) => { /* Expected success with valid flatbuffer data */ }
+            Err(e) => panic!("Should succeed with valid flatbuffer data: {:?}", e),
+        }
+
+        // Test invalid version
+        let mut bytes = BytesMut::with_capacity(2 + fb_data.len());
+        bytes.put_u16(MANIFEST_FORMAT_VERSION + 1);
+        bytes.put_slice(fb_data);
+        let invalid_bytes = bytes.freeze();
+
+        match codec.decode(&invalid_bytes) {
+            Err(SlateDBError::InvalidVersion {
+                expected_version,
+                actual_version,
+            }) => {
+                assert_eq!(expected_version, MANIFEST_FORMAT_VERSION);
+                assert_eq!(actual_version, MANIFEST_FORMAT_VERSION + 1);
+            }
+            _ => panic!("Should fail with version mismatch"),
+        }
+    }
 }
diff --git a/src/sst.rs b/src/sst.rs
index 1496c7452..1c5cda159 100644
--- a/src/sst.rs
+++ b/src/sst.rs
@@ -18,6 +18,12 @@ use crate::types::RowEntry;
 use crate::{blob::ReadOnlyBlob, config::CompressionCodec};
 use crate::{block::BlockBuilder, error::SlateDBError};
 
+pub(crate) const SST_FORMAT_VERSION: u16 = 1;
+
+// 8 bytes for the metadata offset + 2 bytes for the version
+const NUM_FOOTER_BYTES: usize = 10;
+const NUM_FOOTER_BYTES_LONG: u64 = NUM_FOOTER_BYTES as u64;
+
 #[derive(Clone)]
 pub(crate) struct SsTableFormat {
     pub(crate) block_size: usize,
@@ -44,16 +50,31 @@ impl SsTableFormat {
         &self,
         obj: &impl ReadOnlyBlob,
     ) -> Result<SsTableInfo, SlateDBError> {
-        let len = obj.len().await?;
-        if len <= 8 {
+        let obj_len = obj.len().await?;
+        if obj_len <= NUM_FOOTER_BYTES_LONG {
             return Err(SlateDBError::EmptySSTable);
         }
         // Get the size of the metadata
-        let sst_metadata_offset_range = (len - 8)..len;
-        let sst_metadata_offset = obj.read_range(sst_metadata_offset_range).await?.get_u64();
-        // Get the metadata. Last 8 bytes are the offset of SsTableInfo
-        let sst_metadata_range = sst_metadata_offset..len - 8;
-        let sst_metadata_bytes = obj.read_range(sst_metadata_range).await?;
+        let header = obj
+            .read_range((obj_len - NUM_FOOTER_BYTES_LONG)..obj_len)
+            .await?;
+        assert!(header.len() == NUM_FOOTER_BYTES);
+
+        // Last 2 bytes of the header represent the version
+        let version = header.slice(8..NUM_FOOTER_BYTES).get_u16();
+        // TODO: Support older and newer versions
+        if version != SST_FORMAT_VERSION {
+            return Err(SlateDBError::InvalidVersion {
+                expected_version: SST_FORMAT_VERSION,
+                actual_version: version,
+            });
+        }
+
+        // First 8 bytes of the header represent the metadata offset
+        let sst_metadata_offset = header.slice(0..8).get_u64();
+        let sst_metadata_bytes = obj
+            .read_range(sst_metadata_offset..obj_len - NUM_FOOTER_BYTES_LONG)
+            .await?;
         SsTableInfo::decode(sst_metadata_bytes, &*self.sst_codec)
     }
 
@@ -519,6 +540,8 @@ impl EncodedSsTableBuilder<'_> {
     /// +---------------------------------------------------+
     /// |             8-byte Metadata Offset                |
     /// +---------------------------------------------------+
+    /// |                 2-byte Version                    |
+    /// +---------------------------------------------------+
     /// * Only present if num_keys >= min_filter_keys.
     ///
     pub fn build(mut self) -> Result<EncodedSsTable, SlateDBError> {
@@ -574,6 +597,8 @@ impl EncodedSsTableBuilder<'_> {
         // write the metadata offset at the end of the file. FlatBuffer internal
         // representation is not intended to be used directly.
         buf.put_u64(meta_offset);
+        // write the version at the end of the file.
+        buf.put_u16(SST_FORMAT_VERSION);
         self.blocks.push_back(Bytes::from(buf));
         Ok(EncodedSsTable {
             info,
@@ -1022,6 +1047,51 @@ mod tests {
         ));
     }
 
+    #[tokio::test]
+    async fn test_version_checking() {
+        // Create a valid SST
+        let root_path = Path::from("");
+        let object_store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+        let format = SsTableFormat {
+            block_size: 32,
+            ..SsTableFormat::default()
+        };
+
+        let table_store = TableStore::new(object_store, format.clone(), root_path, None);
+        let mut builder = table_store.table_builder();
+        builder.add_value(b"key1", b"value1", gen_attrs(1)).unwrap();
+        builder.add_value(b"key2", b"value2", gen_attrs(2)).unwrap();
+        let encoded = builder.build().unwrap();
+        let mut bytes = BytesMut::new();
+        for block in encoded.unconsumed_blocks {
+            bytes.extend_from_slice(&block);
+        }
+        let bytes = bytes.freeze();
+
+        // Test valid version decodes properly through read_info
+        let valid_blob = BytesBlob {
+            bytes: bytes.clone(),
+        };
+        let result = format.read_info(&valid_blob).await;
+        match result {
+            Ok(_) => {}
+            Err(e) => {
+                panic!("Expected Ok result, but got error: {:?}", e);
+            }
+        }
+
+        let mut invalid_bytes = BytesMut::from(bytes.clone());
+        // Corrupt the version
+        invalid_bytes[bytes.len() - 1] ^= 1;
+        let invalid_blob = BytesBlob {
+            bytes: invalid_bytes.freeze(),
+        };
+        assert!(matches!(
+            format.read_info(&invalid_blob).await,
+            Err(SlateDBError::InvalidVersion { .. })
+        ));
+    }
+
     struct BytesBlob {
         bytes: Bytes,
     }


Reviews:
user: login: dmvk
body: 👍 LGTM overall! I've left few questions and suggestions inline, but the logic is solid.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T08:08:24Z
user: login: rodesai
body: Took a pass through and overall LGTM. One question - the SST info is present in both the manifests and the SSTs (as a footer) - so if we wanted to make a breaking change to that structure then we'd need to bump the manifest version _and_ the sst version?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T19:30:06Z
user: login: criccomini
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T23:49:55Z
user: login: criccomini
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T23:53:35Z
user: login: criccomini
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T23:55:22Z
user: login: criccomini
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T23:56:05Z
user: login: criccomini
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-14T00:02:13Z
user: login: rodesai
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T04:51:41Z

Review comments:
diff_hunk: @@ -45,14 +47,23 @@ impl SsTableFormat {
         obj: &impl ReadOnlyBlob,
     ) -> Result<SsTableInfo, SlateDBError> {
         let len = obj.len().await?;
-        if len <= 8 {
+        if len <= 10 {
+            // 8 bytes for the metadata offset + 2 bytes for the version
             return Err(SlateDBError::EmptySSTable);
         }
+        let version = obj.read_range(len - 2..len).await?.get_u16();
path: src/sst.rs
user: login: dmvk
body: 👍 It would be great if we could avoid doing an extra read against S3.

```diff
diff --git a/src/sst.rs b/src/sst.rs
index 4c0b558..24f060c 100644
--- a/src/sst.rs
+++ b/src/sst.rs
@@ -20,6 +20,9 @@ use crate::{block::BlockBuilder, error::SlateDBError};
 
 pub(crate) const SST_FORMAT_VERSION: u16 = 1;
 
+const NUM_FOOTER_BYTES: usize = 10;
+const NUM_FOOTER_BYTES_LONG: u64 = NUM_FOOTER_BYTES as u64;
+
 #[derive(Clone)]
 pub(crate) struct SsTableFormat {
     pub(crate) block_size: usize,
@@ -46,12 +49,21 @@ impl SsTableFormat {
         &self,
         obj: &impl ReadOnlyBlob,
     ) -> Result<SsTableInfo, SlateDBError> {
-        let len = obj.len().await?;
-        if len <= 10 {
+        let obj_len = obj.len().await?;
+        if obj_len <= NUM_FOOTER_BYTES_LONG {
             // 8 bytes for the metadata offset + 2 bytes for the version
             return Err(SlateDBError::EmptySSTable);
         }
-        let version = obj.read_range(len - 2..len).await?.get_u16();
+        // Get the size of the metadata
+        let header = obj
+            .read_range((obj_len - NUM_FOOTER_BYTES_LONG)..obj_len)
+            .await?;
+        assert!(header.len() == NUM_FOOTER_BYTES);
+
+        // Last 2 bytes of the header represent the version
+        let version = header
+            .slice(8..NUM_FOOTER_BYTES)
+            .get_u16();
         // TODO: Support older and newer versions
         if version != SST_FORMAT_VERSION {
             return Err(SlateDBError::InvalidVersion {
@@ -59,12 +71,12 @@ impl SsTableFormat {
                 actual_version: version,
             });
         }
-        // Get the size of the metadata
-        let sst_metadata_offset_range = (len - 10)..(len - 2);
-        let sst_metadata_offset = obj.read_range(sst_metadata_offset_range).await?.get_u64();
-        // Get the metadata. Last 8 bytes are the offset of SsTableInfo
-        let sst_metadata_range = sst_metadata_offset..len - 10;
-        let sst_metadata_bytes = obj.read_range(sst_metadata_range).await?;
+
+        // First 8 bytes of the header represent the metadata offset
+        let sst_metadata_offset = header.slice(0..8).get_u64();
+        let sst_metadata_bytes = obj
+            .read_range(sst_metadata_offset..obj_len - NUM_FOOTER_BYTES_LONG)
+            .await?;
         SsTableInfo::decode(sst_metadata_bytes, &*self.sst_codec)
     }
```
created_at: 2025-03-11T08:00:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,12 @@ pub enum SlateDBError {
 
     #[error("Database already exists: {msg}")]
     DatabaseAlreadyExists { msg: String },
+
+    #[error("Byte format version mismatch")]
path: src/error.rs
user: login: dmvk
body: 🤔 do we need to include expected_version and actual_version in this string in order to show up in the user facing message?
created_at: 2025-03-11T08:04:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -106,7 +108,15 @@ impl ManifestCodec for FlatBufferManifestCodec {
     }
 
     fn decode(&self, bytes: &Bytes) -> Result<Manifest, SlateDBError> {
-        let manifest = flatbuffers::root::<ManifestV1>(bytes)?;
+        let version = u16::from_be_bytes([bytes[0], bytes[1]]);
path: src/flatbuffer_types.rs
user: login: dmvk
body: Do we need to check for `len >= 2` before parsing version?
created_at: 2025-03-11T08:07:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,14 +47,23 @@ impl SsTableFormat {
         obj: &impl ReadOnlyBlob,
     ) -> Result<SsTableInfo, SlateDBError> {
         let len = obj.len().await?;
-        if len <= 8 {
+        if len <= 10 {
+            // 8 bytes for the metadata offset + 2 bytes for the version
             return Err(SlateDBError::EmptySSTable);
         }
+        let version = obj.read_range(len - 2..len).await?.get_u16();
path: src/sst.rs
user: login: criccomini
body: Good catch! Merged:

e501496f92797e7a526b2497dae831e990e7cdfb
created_at: 2025-03-13T23:49:55Z
author_association: COLLABORATOR
diff_hunk: @@ -112,6 +112,12 @@ pub enum SlateDBError {
 
     #[error("Database already exists: {msg}")]
     DatabaseAlreadyExists { msg: String },
+
+    #[error("Byte format version mismatch")]
path: src/error.rs
user: login: criccomini
body: Good catch. In the tests, I get:

```
Should succeed with valid flatbuffer data: InvalidVersion { expected_version: 1, actual_version: 2 }
```

But that's just dumping the error. I'll update the string.
created_at: 2025-03-13T23:53:35Z
author_association: COLLABORATOR
diff_hunk: @@ -112,6 +112,12 @@ pub enum SlateDBError {
 
     #[error("Database already exists: {msg}")]
     DatabaseAlreadyExists { msg: String },
+
+    #[error("Byte format version mismatch")]
path: src/error.rs
user: login: criccomini
body: Fixed here:

845d912
created_at: 2025-03-13T23:55:22Z
author_association: COLLABORATOR
diff_hunk: @@ -106,7 +108,15 @@ impl ManifestCodec for FlatBufferManifestCodec {
     }
 
     fn decode(&self, bytes: &Bytes) -> Result<Manifest, SlateDBError> {
-        let manifest = flatbuffers::root::<ManifestV1>(bytes)?;
+        let version = u16::from_be_bytes([bytes[0], bytes[1]]);
path: src/flatbuffer_types.rs
user: login: criccomini
body: Hah, yea I debated this. I'll add it just for paranoia.
created_at: 2025-03-13T23:56:05Z
author_association: COLLABORATOR
diff_hunk: @@ -106,7 +108,15 @@ impl ManifestCodec for FlatBufferManifestCodec {
     }
 
     fn decode(&self, bytes: &Bytes) -> Result<Manifest, SlateDBError> {
-        let manifest = flatbuffers::root::<ManifestV1>(bytes)?;
+        let version = u16::from_be_bytes([bytes[0], bytes[1]]);
path: src/flatbuffer_types.rs
user: login: criccomini
body: Fixed here: b323335
created_at: 2025-03-14T00:02:13Z
author_association: COLLABORATOR

PR comments:
user: login: criccomini
created_at: 2025-03-14T00:09:53Z
body: > One question - the SST info is present in both the manifests and the SSTs (as a footer) - so if we wanted to make a breaking change to that structure then we'd need to bump the manifest version and the sst version?

This is a good call out. Yes, both versions would need to be bumped. The versions are independent and represent bytes on disk. Since the `SsTableInfo` ends up in both manifest files and SST files, they'd both need to be bumped.

Do you think we should just go with a single version? It might be mentally simpler to deal with, though it would be less transparent about exactly which files were changing. 🤔 It could also lead to more version numbers.
author_association: COLLABORATOR
user: login: rodesai
created_at: 2025-03-14T04:36:22Z
body: > Do you think we should just go with a single version? It might be mentally simpler to deal with, though it would be less transparent about exactly which files were changing. 🤔 It could also lead to more version numbers.

Nah I like having a separate version number for the manifest vs the ssts. Just confirming my understanding.
author_association: CONTRIBUTOR