Repository: activeloopai/deeplake
Pull request: tensor.clear() to delete all samples from tensor
## 🚀 🚀 Pull Request

### Checklist:

- [x]  [My code follows the style guidelines of this project](https://www.notion.so/activeloop/Engineering-Guidelines-d6e502306d0e4133a8ca507516d1baab) and the [Contributing document](https://github.com/activeloopai/Hub/blob/release/2.0/CONTRIBUTING.md)
- [x]  I have commented my code, particularly in hard-to-understand areas
- [x]  I have kept the `coverage-rate` up
- [x]  I have performed a self-review of my own code and resolved any problems
- [x]  I have checked to ensure there aren't any other open [Pull Requests](https://github.com/activeloopai/Hub/pulls) for the same change
- [x]  I have described and made corresponding changes to the relevant documentation
- [x]  New and existing unit tests pass locally with my changes


### Changes
Added method to delete all samples from tensor.
Closes #1165.
<!-- Describe your changes! Describe the scope of this PR's responsibilities. -->

Problem statement:
[FEATURE] Add method to delete all samples from tensor
## 🚨🚨 Feature Request

- [ ] Related to an existing [Issue](../issues) 
- [x] A new implementation (Improvement, Extension) 

### If your feature will improve `HUB`

Add a `tensor.clear()` method that deletes all data from a tensor. The tensor itself should not be deleted from the dataset. 

Difficulty: Medium

Hints from issue discussion:
Hey, can I do this? I am interested but I will need some help since I am new to it.
Hey @pranatimittal, greetings and welcome to Hub! Great to hear that you're interested in contributing. Do you have an initial approach to solve this problem? We can help you out better that way!
@dhiganthrao I was think to take tensor name as input whose data needs to deleted. then use `torch.empty()` to return an empty tensor. is that okay? 
Hey @pranatimittal, apologies for the delayed response! Your method would return an empty tensor, but won't delete the data from the dataset. We need a method to permanently delete the data in a tensor, and retain the tensor itself.

Commit message: update meta + test

Patch to review:
diff --git a/hub/api/tests/test_api.py b/hub/api/tests/test_api.py
index 4cb4b28a31..1229609da9 100644
--- a/hub/api/tests/test_api.py
+++ b/hub/api/tests/test_api.py
@@ -794,3 +794,20 @@ def test_tobytes(memory_ds, compressed_image_paths, audio_paths):
     for i in range(3):
         assert ds.image[i].tobytes() == image_bytes
         assert ds.audio[i].tobytes() == audio_bytes
+
+
+def test_clear(local_ds_generator):
+    ds = local_ds_generator()
+    a = ds.create_tensor("a")
+    a.extend([1, 2, 3, 4])
+    a.clear()
+    assert len(ds) == 0
+    assert len(a) == 0
+
+    image = ds.create_tensor("image", htype="image", sample_compression="png")
+    image.extend(np.ones((4, 224, 224, 3), dtype="uint8"))
+    image.clear()
+    assert len(ds) == 0
+    assert len(image) == 0
+    assert image.htype == "image"
+    assert image.meta.sample_compression == "png"
diff --git a/hub/api/tests/test_chunk_sizes.py b/hub/api/tests/test_chunk_sizes.py
index 9b67f63e56..8c9b57fbfd 100644
--- a/hub/api/tests/test_chunk_sizes.py
+++ b/hub/api/tests/test_chunk_sizes.py
@@ -30,6 +30,11 @@ def _extend_tensors(images, labels):
     labels.extend(np.ones(100, dtype=np.uint32))
 
 
+def _clear_tensors(images, labels):
+    images.clear()
+    labels.clear()
+
+
 def test_append(memory_ds):
     ds = memory_ds
     images, labels = _create_tensors(ds)
@@ -99,3 +104,13 @@ def test_extend_and_append(memory_ds):
     _assert_num_chunks(images, 20)
 
     assert len(ds) == 400
+
+
+def test_clear(memory_ds):
+    ds = memory_ds
+    images, labels = _create_tensors(ds)
+
+    _clear_tensors(images, labels)
+
+    _assert_num_chunks(labels, 0)
+    _assert_num_chunks(images, 0)
diff --git a/hub/core/chunk_engine.py b/hub/core/chunk_engine.py
index 55985b3be2..922d409004 100644
--- a/hub/core/chunk_engine.py
+++ b/hub/core/chunk_engine.py
@@ -418,7 +418,10 @@ def _synchronize_cache(self, chunk_keys: List[str] = None):
 
         # synchronize chunks
         if chunk_keys is None:
-            chunk_keys = [self.last_chunk_key]
+            if self.num_chunks > 0:
+                chunk_keys = [self.last_chunk_key]
+            else:
+                chunk_keys = []
         for chunk_key in chunk_keys:
             chunk = self.get_chunk(chunk_key)
             self.cache.update_used_cache_for_path(chunk_key, chunk.nbytes)  # type: ignore
@@ -543,6 +546,30 @@ def append(self, sample: SampleValue):
         """Formats a single `sample` (compresseses/decompresses if applicable) and feeds it into `_append_bytes`."""
         self.extend([sample])
 
+    def clear(self):
+        """Deletes all samples."""
+        self.cache.check_readonly()
+
+        n_samples = self.num_samples
+        chunk_names = self.get_chunk_names_for_multiple_indexes(0, n_samples, n_samples)
+        commit_id = self.version_state["commit_id"]
+        chunk_keys = [get_chunk_key(self.key, name, commit_id) for name in chunk_names]
+
+        enc_key = get_chunk_id_encoder_key(self.key, commit_id)
+        enc = ChunkIdEncoder()
+        self.cache[enc_key] = enc
+
+        for key in chunk_keys:
+            del self.cache[key]
+
+        self.tensor_meta.length = 0
+        self.tensor_meta.min_shape = []
+        self.tensor_meta.max_shape = []
+        self.tensor_meta.dtype = None
+
+        self._synchronize_cache()
+        self.cache.maybe_flush()
+
     def update(
         self,
         index: Index,
diff --git a/hub/core/tensor.py b/hub/core/tensor.py
index c3c9508dd1..8ec9cca7dd 100644
--- a/hub/core/tensor.py
+++ b/hub/core/tensor.py
@@ -173,6 +173,10 @@ def append(
         """
         self.extend([sample])
 
+    def clear(self):
+        """Deletes all samples from the tensor"""
+        self.chunk_engine.clear()
+
     @property
     def meta(self):
         return self.chunk_engine.tensor_meta


Code review comments:
diff_hunk: @@ -543,6 +546,30 @@ def append(self, sample: SampleValue):
         """Formats a single `sample` (compresseses/decompresses if applicable) and feeds it into `_append_bytes`."""
         self.extend([sample])
 
+    def clear(self):
+        """Deletes all samples."""
+        self.cache.check_readonly()
+
+        n_samples = self.num_samples
+        chunk_names = self.get_chunk_names_for_multiple_indexes(0, n_samples, n_samples)
+        commit_id = self.version_state["commit_id"]
+        chunk_keys = [get_chunk_key(self.key, name, commit_id) for name in chunk_names]
+
+        enc_key = get_chunk_id_encoder_key(self.key, commit_id)
+        enc = ChunkIdEncoder()
+        self.cache[enc_key] = enc
+
+        for key in chunk_keys:
+            del self.cache[key]
+
+        self.tensor_meta.length = 0
+        self.tensor_meta.min_shape = []
+        self.tensor_meta.max_shape = []
+        self.tensor_meta.dtype = None
path: hub/core/chunk_engine.py
text: @user1:
Clearing samples shouldn't reset dtype and htype. (dtype might be provided explicitly by user using create_tensor)

diff_hunk: @@ -794,3 +794,20 @@ def test_tobytes(memory_ds, compressed_image_paths, audio_paths):
     for i in range(3):
         assert ds.image[i].tobytes() == image_bytes
         assert ds.audio[i].tobytes() == audio_bytes
+
+
+def test_clear(local_ds_generator):
+    ds = local_ds_generator()
+    a = ds.create_tensor("a")
+    a.extend([1, 2, 3, 4])
+    a.clear()
+    assert len(ds) == 0
+    assert len(a) == 0
+
+    image = ds.create_tensor("image", htype="image", sample_compression="png")
+    image.extend(np.ones((4, 224, 224, 3), dtype="uint8"))
+    image.clear()
+    assert len(ds) == 0
+    assert len(image) == 0
+    assert image.htype == "image"
+    assert image.meta.sample_compression == "png"
path: hub/api/tests/test_api.py
text: @user1:
Reload the dataset from disk (`ds=local_ds_generator`) and do the assertions again to check for persistence.