Repository: Lightning-AI/litData

Issue:
Mismatch Between index.json and Actual Chunk File Sizes
## 🐛 Bug Report: Mismatch Between index.json and Actual Chunk File Sizes

<!-- A clear and concise description of what the bug is. -->

There seems to be discrepancy between the chunk sizes listed in index.json and the actual sizes of the corresponding chunk files stored on disk. 

### To Reproduce

1. Create a sample [optimized dataset](https://github.com/Lightning-AI/litdata?tab=readme-ov-file#speed-up-model-training)
2. Check the size of chunk files and compare it with present in the index.json file

![Image](https://github.com/user-attachments/assets/b8c904c1-20ab-4634-a1e0-2d7813e89900)
```
ls -l fast_data/
```

### Expected behavior
The actual file sizes should match the chunk_bytes values recorded in index.json.
<!-- A clear and concise description of what you expected to happen. -->

Discussion:

Solution patch:
diff --git a/src/litdata/streaming/config.py b/src/litdata/streaming/config.py
index af96ede..aee0d1f 100644
--- a/src/litdata/streaming/config.py
+++ b/src/litdata/streaming/config.py
@@ -237,9 +237,6 @@ class ChunksConfig:
 
         filesize_bytes = chunk["chunk_bytes"]
 
-        if self._config and self._config.get("encryption") is None and (not local_chunkpath.endswith(".parquet")):
-            filesize_bytes += (1 + chunk["chunk_size"]) * 4
-
         return local_chunkpath, begin, filesize_bytes
 
     def _get_chunk_index_from_filename(self, chunk_filename: str) -> int:
diff --git a/src/litdata/streaming/writer.py b/src/litdata/streaming/writer.py
index 6723e88..2cf6f61 100644
--- a/src/litdata/streaming/writer.py
+++ b/src/litdata/streaming/writer.py
@@ -235,12 +235,11 @@ class BinaryWriter:
         sample_data = b"".join([item.data for item in items])
         data = num_items.tobytes() + offsets.tobytes() + sample_data
 
-        current_chunk_bytes = sum([item.bytes for item in items])
-
         # Whether to encrypt the data at the chunk level
         if self._encryption and self._encryption.level == EncryptionLevel.CHUNK:
             data = self._encryption.encrypt(data)
-            current_chunk_bytes = len(data)
+
+        current_chunk_bytes = len(data)
 
         if self._chunk_bytes and current_chunk_bytes > self._chunk_bytes:
             warnings.warn(


Test patch:
diff --git a/tests/streaming/test_writer.py b/tests/streaming/test_writer.py
index 4ad9785..55366a0 100644
--- a/tests/streaming/test_writer.py
+++ b/tests/streaming/test_writer.py
@@ -251,6 +251,27 @@ def test_writer_unordered_indexes(tmpdir):
     assert data["chunks"][2]["chunk_size"] == 2
 
 
+def test_chunk_bytes_consistency(tmpdir):
+    cache_dir = os.path.join(tmpdir, "chunks")
+    os.makedirs(cache_dir, exist_ok=True)
+
+    binary_writer = BinaryWriter(cache_dir, chunk_size=5)
+
+    for i in range(100):
+        binary_writer[i] = i
+
+    binary_writer.done()
+    binary_writer.merge()
+
+    with open(os.path.join(cache_dir, "index.json")) as f:
+        config_data = json.load(f)
+
+    for chunk in config_data["chunks"]:
+        chunk_file = os.path.join(cache_dir, chunk["filename"])
+        chunk_size = os.path.getsize(chunk_file)
+        assert chunk_size == chunk["chunk_bytes"]
+
+
 def test_writer_save_checkpoint(tmpdir):
     cache_dir = os.path.join(tmpdir, "chunks")
     os.makedirs(cache_dir, exist_ok=True)