Repository: activeloopai/deeplake

Issue:
Dataset.pop() not working as expected.
```
ds = deeplake.load(f'hub://{ACTIVELOOP_ORG_NAME}/{ACTIVELOOP_DATASET}', token=ACTIVELOOP_TOKEN, read_only=False)
    for index in indices_to_be_deleted:
        print("Popping index", index)
        ds.pop(index)
```
I'm not having success deleting all of the rows of my dataset in this manner


Discussion:
@j-beastman thanks for reporting the issue and sorry about the experience. Do you mind sharing the stack trace and the error you are experiencing? 
Hi @davidbuniat! Sorry I didn't include that initially, but what is a stack trace in the context of my program not crashing, but just not behaving as expected? Here's my code:
```
def pull_deeplake_dataset() -> Dataset:
    # either load existing vector store or upload a new one to the hub
    ds = deeplake.load(f'hub://{ACTIVELOOP_ORG_NAME}/{ACTIVELOOP_DATASET}', token=ACTIVELOOP_TOKEN, read_only=False)
    return ds

def clear_dataset():
    try:
        ds = pull_deeplake_dataset()
        len = ds.max_len
        if len != 0:
            print("Deleting data")
            for i in range(0, len): # Apparently this is slow, idk the other way to do it.
                print("Popping index", i)
                ds.text.pop()
                ds.embedding.pop()
                ds.metadata.pop()
                ds.id.pop()
    except (GetChunkError, DatasetHandlerError):
        print("Dataset is already empty")
```
Basically, I'm having to pop off each value from the tensors instead of being able to use ds.pop()
Hey @j-beastman! When you do
```python
for idx in range(length):
    ds.pop(idx)
```
The length of the dataset changes on each iteration, so we end up popping the wrong indices and go beyond the length of the dataset. So we have to do `ds.pop(0)` or `ds.pop()` and it should work. (Did you try this as well and run into some issue?)

About the "slow indexing" warning, you can replace `for i in range(0, len)` with `for i, sample in enumerate(ds.max_view)`:

```python
for i, sample in enumerate(ds.max_view):
    ds.pop()
```

Also, refrain from using `len` as a variable, because it is a built-in function :)
Hey @FayazRahman ! (thanks for the tips) I got rid of using the index. I just use ds.pop() and it doesn't remove from the dataset. When I do ds.tensor.pop() for each tensor, I'm able to clear the dataset. I'm not sure what is going on. 
@j-beastman Interesting, can you share the final state of your code so I can try it out on my end?
Attaching a `ds.summary()` could be useful as well
```
from deeplake.core.dataset import Dataset
from langchain.docstore.document import Document
from langchain.document_loaders import DirectoryLoader
from langchain.vectorstores import DeepLake, VectorStore
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
import deeplake
from deeplake.util.exceptions import GetChunkError, DatasetHandlerError
def pull_deeplake_dataset() -> Dataset:
    # either load existing vector store or upload a new one to the hub
    ds = deeplake.load(f'hub://{ACTIVELOOP_ORG_NAME}/{ACTIVELOOP_DATASET}', token=ACTIVELOOP_TOKEN, read_only=False)
    return ds

def clear_dataset():
    try:
        ds = pull_deeplake_dataset()
        for i, sample in enumerate(ds.max_len): # Try max_view too
            print("Popping index", i)
            ds.text.pop()
            ds.embedding.pop()
            ds.metadata.pop()
            ds.id.pop()
    except (GetChunkError, DatasetHandlerError):
        print("Dataset is already empty")

def get_embeddings():
    return SentenceTransformerEmbeddings(
        model_name=EMBEDDING_MODEL_NAME,
        cache_folder="../streamlit/cache/",
    )

def load_misc(directory):
    loader = DirectoryLoader(f"./{directory}")

    print(f"Loading {directory} directory for any {filter}")
    data = loader.load()
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=10,
    )
    r_docs = splitter.split_documents(data)
    return r_docs
def upload_latin():
    clear_dataset()
    chunked_text = load_misc("data/snippets")
    embedding_function = get_embeddings()
    DeepLake.from_documents(
        chunked_text,
        embedding_function,
        dataset_path=VECTOR_STORE_PATH,
        token=ACTIVELOOP_TOKEN,
    )
```
Try running it twice. I'll provide the file that I use for this snippet too.
[latin.txt](https://github.com/activeloopai/deeplake/files/12304649/latin.txt)

Thanks for the details and the code @j-beastman! I see what is causing the issue, I'll let you know as soon as the fix is released.

Resolving patch:
diff --git a/deeplake/core/dataset/dataset.py b/deeplake/core/dataset/dataset.py
--- a/deeplake/core/dataset/dataset.py
+++ b/deeplake/core/dataset/dataset.py
@@ -4463,8 +4463,6 @@ def pop(self, index: Optional[int] = None):
         Raises:
             IndexError: If the index is out of range.
         """
-        self._initial_autoflush.append(self.storage.autoflush)
-        self.storage.autoflush = False
         max_len = max((t.num_samples for t in self.tensors.values()), default=0)
         if max_len == 0:
             raise IndexError("Can't pop from empty dataset.")
@@ -4479,11 +4477,10 @@ def pop(self, index: Optional[int] = None):
                 f"Index {index} is out of range. The longest tensor has {max_len} samples."
             )
 
-        for tensor in self.tensors.values():
-            if tensor.num_samples > index:
-                tensor.pop(index)
-
-        self.storage.autoflush = self._initial_autoflush.pop()
+        with self:
+            for tensor in self.tensors.values():
+                if tensor.num_samples > index:
+                    tensor.pop(index)
 
     @property
     def is_view(self) -> bool:


Test patch: