Repository: Lightning-AI/litdata

Issue:
Litdata optimise doesn't serialise boolean values
## 🐛 Bug


### To Reproduce

Steps to reproduce the behavior...

- Try to encode a boolean value with litdata.optimize(...)

<details>
  <summary>Code sample</summary>

```python
import litdata as ld

def processor(i):
    return {'index': i, 'col1': "example1", 'col2': 2, 'col3': 3.1, 'col4': True, 'col5': False}

if __name__ == "__main__":
    ld.optimize(
            fn=processor,
            inputs=list(range(3)),
            output_dir="example_output",
            chunk_bytes="1MB",
            mode="overwrite",
            num_workers=1,
        )
    
    # Load and iterate over the optimized data
    dataset = ld.StreamingDataset("example_output")
    
    print("\nIterating over optimized data:")
    for item in dataset:
        print(f"Row: {item}")
```

This results in:
```
Iterating over optimized data:
Row: {'index': np.int64(0), 'col1': 'example1', 'col2': np.int64(2), 'col3': np.float64(3.1), 'col4': np.int64(1), 'col5': np.int64(0)}
Row: {'index': np.int64(1), 'col1': 'example1', 'col2': np.int64(2), 'col3': np.float64(3.1), 'col4': np.int64(1), 'col5': np.int64(0)}
Row: {'index': np.int64(2), 'col1': 'example1', 'col2': np.int64(2), 'col3': np.float64(3.1), 'col4': np.int64(1), 'col5': np.int64(0)}
```

</details>

### Expected behavior

I'd expect this output:

```
Iterating over optimized data:
Row: {'index': np.int64(0), 'col1': 'example1', 'col2': np.int64(2), 'col3': np.float64(3.1), 'col4': True, 'col5': False}
Row: {'index': np.int64(1), 'col1': 'example1', 'col2': np.int64(2), 'col3': np.float64(3.1), 'col4': True, 'col5': False}
Row: {'index': np.int64(2), 'col1': 'example1', 'col2': np.int64(2), 'col3': np.float64(3.1), 'col4': True, 'col5': False}
```

### Additional context

I've implemented a fix and will make a pull request.

<details>
  <summary>Environment detail</summary>

litdata  version 0.2.39

- PyTorch Version (e.g., 1.0):
- OS (e.g., Linux): Linux
- How you installed PyTorch (`conda`, `pip`, source): poetry (pip)
- Build command you used (if compiling from source): No
- Python version: 3.12.9
- CUDA/cuDNN version: None
- GPU models and configuration: None
- Any other relevant information: No

</details>


Discussion:
deependujha: Hi @DominiquePaul can you permit me to write/modify your PR. I am pushing my changes but its not reflecting.

<img width="839" alt="Screenshot 2025-02-21 at 3 42 42 PM" src="https://github.com/user-attachments/assets/945a695e-1161-4420-8040-6939ea791661" />

DominiquePaul: @deependujha Yes, sure. How do I do that. It's my first time contributing to open source.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/Lightning-AI/litdata/pull/481?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Lightning-AI) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 71%. Comparing base [(`0ec5a86`)](https://app.codecov.io/gh/Lightning-AI/litdata/commit/0ec5a86a4e7c2193f2546451a39f0f7df73b203a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Lightning-AI) to head [(`8d13265`)](https://app.codecov.io/gh/Lightning-AI/litdata/commit/8d1326590742e68dcd20f5c588acdb91be13d023?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Lightning-AI).

> :exclamation:  There is a different number of reports uploaded between BASE (0ec5a86) and HEAD (8d13265). Click for more details.
> 
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (0ec5a86) | HEAD (8d13265) |
>|------|------|------|
>|unittests|6|1|
>|windows-2022|2|1|
>|3.10|3|0|
>|3.9|3|1|
>|ubuntu-22.04|2|0|
>|macos-13|2|0|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@         Coverage Diff          @@
##           main   #481    +/-   ##
====================================
- Coverage    79%    71%    -8%     
====================================
  Files        38     38            
  Lines      5606   5597     -9     
====================================
- Hits       4437   3999   -438     
- Misses     1169   1598   +429     
```

</details>

Patch:
diff --git a/src/litdata/streaming/reader.py b/src/litdata/streaming/reader.py
index e3f76b7..dd625a6 100644
--- a/src/litdata/streaming/reader.py
+++ b/src/litdata/streaming/reader.py
@@ -446,7 +446,8 @@ def _get_folder_size(path: str, config: ChunksConfig) -> int:
         if filename in config.filename_to_size_map:
             with contextlib.suppress(FileNotFoundError):
                 size += config.filename_to_size_map[filename]
-        elif not filename.endswith((".cnt", ".lock")):
+        elif not filename.endswith((".cnt", ".lock", ".json", ".zstd.bin")):
+            # ignore .cnt, .lock, .json and .zstd files for warning
             logger.warning(f"File {filename} is not a valid chunk file. It will be ignored.")
     return size
 
diff --git a/src/litdata/streaming/serializers.py b/src/litdata/streaming/serializers.py
index 0d8d504..accc1dc 100644
--- a/src/litdata/streaming/serializers.py
+++ b/src/litdata/streaming/serializers.py
@@ -388,6 +388,43 @@ class FloatSerializer(NumericSerializer, Serializer):
         return isinstance(data, float)
 
 
+class BooleanSerializer(Serializer):
+    """The BooleanSerializer serializes and deserializes boolean values to and from bytes."""
+
+    def serialize(self, item: bool) -> Tuple[bytes, Optional[str]]:
+        """Serialize a boolean value to bytes.
+
+        Args:
+            item: Boolean value to serialize
+
+        Returns:
+            Tuple containing the serialized bytes and None for the format string
+        """
+        return np.bool_(item).tobytes(), None
+
+    def deserialize(self, data: bytes) -> bool:
+        """Deserialize bytes back into a boolean value.
+
+        Args:
+            data: Bytes to deserialize
+
+        Returns:
+            The deserialized boolean value
+        """
+        return bool(np.frombuffer(data, dtype=np.bool_)[0])
+
+    def can_serialize(self, item: Any) -> bool:
+        """Check if the item can be serialized by this serializer.
+
+        Args:
+            item: Item to check
+
+        Returns:
+            True if the item is a boolean, False otherwise
+        """
+        return isinstance(item, bool)
+
+
 class TIFFSerializer(Serializer):
     """Serializer for TIFF files using tifffile."""
 
@@ -411,6 +448,7 @@ class TIFFSerializer(Serializer):
 _SERIALIZERS = OrderedDict(
     **{
         "str": StringSerializer(),
+        "bool": BooleanSerializer(),
         "int": IntegerSerializer(),
         "float": FloatSerializer(),
         "video": VideoSerializer(),


Test patch:
diff --git a/tests/streaming/test_serializer.py b/tests/streaming/test_serializer.py
index 7afd6b4..db02a02 100644
--- a/tests/streaming/test_serializer.py
+++ b/tests/streaming/test_serializer.py
@@ -30,6 +30,7 @@ from litdata.streaming.serializers import (
     _SERIALIZERS,
     _TORCH_DTYPES_MAPPING,
     _TORCH_VISION_AVAILABLE,
+    BooleanSerializer,
     IntegerSerializer,
     JPEGSerializer,
     NoHeaderNumpySerializer,
@@ -57,6 +58,7 @@ def test_serializers():
     keys = list(_SERIALIZERS.keys())
     assert keys == [
         "str",
+        "bool",
         "int",
         "float",
         "video",
@@ -319,3 +321,24 @@ def test_tiff_serializer():
 
     # Clean up
     os.remove(file_path)
+
+
+def test_boolean_serializer():
+    serializer = BooleanSerializer()
+
+    # Test serialization and deserialization of True
+    data, _ = serializer.serialize(True)
+    assert isinstance(data, bytes)
+    assert serializer.deserialize(data) is True
+
+    # Test serialization and deserialization of False
+    data, _ = serializer.serialize(False)
+    assert isinstance(data, bytes)
+    assert serializer.deserialize(data) is False
+
+    # Test can_serialize method
+    assert serializer.can_serialize(True)
+    assert serializer.can_serialize(False)
+    assert not serializer.can_serialize(1)
+    assert not serializer.can_serialize("True")
+    assert not serializer.can_serialize(None)