Repository: activeloopai/deeplake

Issue:
[BUG] pytorch dataloader index error
## 🐛🐛 Bug Report

I'm trying to understand an issue that is making the PyTorch data loader from deeplake throw an index error for some samples unexpectedly. When I try to fetch the data directly from the data set, the behaviour is not reproducible.


The error first appeared during model training. I was able to reproduce it with the following code:

```python
def deeplake_transform(sample_in, patch_size: int, num_seg_classes: int):
    seg_indices = sample_in["masks/label"]
    partial_mask = sample_in["masks/mask"].astype("float32")
    full_mask = np.zeros((num_seg_classes, patch_size, patch_size), dtype=np.float32)
    for i, idx in enumerate(seg_indices):
        full_mask[idx] = partial_mask[i]

    return dict(
        inputs=dict(image=T.ToTensor()(sample_in["images"])),
        targets=dict(
            segmentations=full_mask,
            classifications=sample_in["labels"].astype("float32"),
        ),
    )
data_loader = ds.pytorch(
        transform=deeplake_transform,
        decode_method={"images": "numpy"},
        batch_size=1,
        num_workers=1,
        transform_kwargs={"num_seg_classes": 67, "patch_size": 512},
    )
iter_loader = iter(data_loader)


while True:
    try:
        sample = next(iter_loader)
    except Exception as e:
        print(e)
        break

    idx += 1
    if idx == len(ds):
        print("finished")
        break
```

The following error is thrown without much context.

```bash
Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 34, in fetch
    data.append(next(self.dataset_iter))
  File "/home/test/.venv/lib/python3.8/site-packages/deeplake/integrations/pytorch/dataset.py", line 472, in __iter__
    for data in stream:
  File ".venv/lib/python3.8/site-packages/deeplake/core/io.py", line 311, in read
    yield from self.stream(block)
  File "/home/test/.venv/lib/python3.8/site-packages/deeplake/core/io.py", line 355, in stream
    data = engine.read_sample_from_chunk(
  File ".venv/lib/python3.8/site-packages/deeplake/core/chunk_engine.py", line 1528, in read_sample_from_chunk
    return chunk.read_sample(
  File ".venv/lib/python3.8/site-packages/deeplake/core/chunk/uncompressed_chunk.py", line 213, in read_sample
    sb, eb = bps[local_index]
  File ".venv/lib/python3.8/site-packages/deeplake/core/meta/encode/base_encoder.py", line 247, in __getitem__
    self._encoded[row_index], row_index, local_sample_index
IndexError: index 7133 is out of bounds for axis 0 with size 7133
```

But the following code produces no errors and exhausts the iterator.

```python
for sample in ds:
    try: # try to read all the data that is used in the 
        sample["images"].data()['value']
        sample["masks/mask"].data()['value']
        sample["masks/label"].data()['value']
        sample["labels"].data()['value']
    except:
        break
    
```

I'm looking for help here since it may be related to the `chunk_engine` behaviour. It could help if the internal exception handler were more explicit about the error.


### ⚙️ Environment

- `Python` version(s): 3.8.10
- `OS`: Ubuntu 18.04
- `IDE`: VS-Code
- `Packages`: [torch==1.13.1, deeplake==3.1.7]






Discussion:
Hey @lspinheiro Thank you for reporting this issue! Were you running this on one of our public datasets? If so, could you pls share the link. It will help us reproduce the issues. If you're running on a private dataset, don't worry about it.  
Hi @istranic. It is a private dataset, sadly I can't share any scripts about the ingestion.

If it helps, I'm ingesting into deeplake by appending samples with the following format:

```python
    dset_entry = {
        "images": image,
        "labels": classes.astype(np.int32),
        "masks/label": seg_classes,
        "masks/mask": masks.astype(np.bool8),
        "metadata": metadata,
    }
```

This is my dataset specification:

<img width="503" alt="image" src="https://user-images.githubusercontent.com/6465613/210360591-0a8b8e58-9821-4a0f-8c3e-1244f683326d.png">

@farizrahman4u @istranic . I'm still debugging this. It looks like the data loader starts failing for all samples after some index. I'm guessing it is something to do with how the `bytes_positions_encoder` is behaving. Can you help me understand how local sample index and global sample index work so that I can investigate further?

The `local_index` variable has the value `9316` which corresponds to `num_samples`. Could it be that one of these variables is not resetting during the chunk lookup process?

<img width="729" alt="image" src="https://user-images.githubusercontent.com/6465613/210461886-78cfc37e-724a-491d-8653-418c1dae2d2b.png">

Hey @lspinheiro This is out of my wheelhouse, but @farizrahman4u will get back to you tomorrow. Thank you for digging in further!
Two updates.

1. It seems the issue is caused by property `num_samples_per_chunk`, in the `chunk_engine` class with the `read_sample` method (can't recall the class name), not being calculated correctly. It looks like the variable is calculated only once (if self._num_samples_per_chunk is None: ...`), for the first chunk, but it can vary slightly between chunks, so the error is thrown whenever the first chunk with more samples than expected is processed.
2. The vary chunk sample size issue seemed to be caused by my `metadata` `json` tensor. I processed the dataset again without adding it and I didn't observe the problem after that.

Maybe you can try to reproduce it by generating a dataset with JSON tensors with attributes of varying types and lengths.

Resolving patch:
diff --git a/deeplake/core/chunk_engine.py b/deeplake/core/chunk_engine.py
--- a/deeplake/core/chunk_engine.py
+++ b/deeplake/core/chunk_engine.py
@@ -1487,7 +1487,7 @@ def read_shape_for_sample(
     @property
     def is_fixed_shape(self):
         tensor_meta = self.tensor_meta
-        return tensor_meta.min_shape == tensor_meta.max_shape
+        return not self.is_text_like and tensor_meta.min_shape == tensor_meta.max_shape
 
     @property
     def num_samples_per_chunk(self):


Test patch: