Repository: mindee/doctr

Issue:
[datasets] Targets are modified inplace
### Bug description

**Targets** are being changed when iterating over some dataset more than one time.
The reason is storing targets in self.data, and changing them in the `__getitem__` ***in place*** using `pre_transforms`, etc.
```python
# _AbstractDataset
def __getitem__(
        self,
        index: int
    ) -> Tuple[Any, Any]:

        # Read image
        img, target = self._read_sample(index)
        # Pre-transforms (format conversion at run-time etc.)
        if self._pre_transforms is not None:
            img, target = self._pre_transforms(img, target)

        if self.img_transforms is not None:
            # typing issue cf. https://github.com/python/mypy/issues/5485
            img = self.img_transforms(img)  # type: ignore[call-arg]

        if self.sample_transforms is not None:
            img, target = self.sample_transforms(img, target)

        return img, target
```

This can be fixed by copying target in the `_read_sample` 
```python
# AbstractDataset
def _read_sample(self, index: int) -> Tuple[tf.Tensor, Any]:
        img_name, target = self.data[index]
        # Read image
        img = read_img_as_tensor(os.path.join(self.root, img_name), dtype=tf.float32)

        return img, target
```

**OR** returning a copy of the target in all transform methods.
```python
def convert_target_to_relative(img: ImageTensor, target: Dict[str, Any]) -> Tuple[ImageTensor, Dict[str, Any]]:

    target['boxes'] = convert_to_relative_coords(target['boxes'], get_img_shape(img))
    return img, target
```


### Code snippet to reproduce the bug

```python
def process_image(train_example):
  img, target = train_example
  img_numpy = img.numpy() * 255
  for example in target['boxes']:
      print(example)
      unnormalized_example = [int(example[0]*img.shape[1]), int(example[1]*img.shape[0]),
                              int(example[2]*img.shape[1]), int(example[3]*img.shape[0])]
      cv2.rectangle(img=img_numpy,
                    pt1=(unnormalized_example[0], unnormalized_example[1]),
                    pt2=(unnormalized_example[2], unnormalized_example[3]),
                    color=(0, 0, 255), thickness=2)
  return img_numpy  


train_set = SROIE(train=True, download=True)

for i in range(2):
  for j, example in enumerate(train_set):
    if j == 0:    
      print(f"{i} ____")
      img_n = process_image(example)
```

P.S. Sorry for not a pretty code style. This snippet is just for an example :) 

### Error traceback

~changed target box coordinates

### Environment

.


Discussion:
Hi @max-kalganov :wave: 

Thanks for reporting this!
Reading your snippets, if you implemented an inplace transform and pass it to the dataset, this will indeed modify the source target. But so far, it's up to the developer. However, you mention a change of target, but your `process_image` is only changing the image. Could you elaborate on your snippet so that it prints / highlight the part where the unwanted behaviour happens please?

And more generally speaking, are you suggesting that we enforce a deepcopy of the target & image in the `_read_sample` method?

Cheers :v: 
Hi @fg-mindee 

Thanks for the fast response! :)
I will try to explain my case in a slightly different way.

1. *Reading your snippets, if you implemented an inplace transform and pass it to the dataset, this will indeed modify the source target.* - `convert_target_to_relative` is ***not*** my code. I have found it in the doctr [sources](https://github.com/mindee/doctr/blob/e5ca2d536864ccff1460ede9bf3da66b13a2ab9a/doctr/datasets/utils.py#L160). 
2. *But so far, it's up to the developer* - well, that's the reason why I haven't created a fork. Leaving the code as it is may lead to some unfortunate behavior as I found. So it would be nice to have some common guideline for samples/targets transformers. Which is generally up to you :) Either making copies in all transform methods or coping samples/targets before all transformations. 
3. *However, you mention a change of target, but your process_image is only changing the image.* - the code in the **Code snippet to reproduce the bug** section is to represent *target changes* (*image bounding boxes in this case*) when iterating twice over `train_set`. But yeah, you are right. My code is quite redundant. Here is a more compact version
```python
def print_example(train_example):
    _, target = train_example
    for target_boxes in target['boxes']:
        print(f"target_boxes - {target_boxes}")

train_set = SROIE(train=True, download=True)

for i in range(3):
    for x_y in train_set:
        print(f"{i} ____")
        print_example(x_y)
        break
```

Results example: 
```
0 ____
target_boxes - [0.08819345 0.09345794 0.5647226  0.11105002]
target_boxes - [0.09672831 0.11050028 0.25320056 0.1275426 ]
target_boxes - [0.09103841 0.12809236 0.5817923  0.14293568]
...
target_boxes - [0.09103841 0.96206707 0.37980086 0.9769104 ]
1 ____
target_boxes - [1.2545299e-04 5.1378749e-05 8.0330385e-04 6.1050043e-05]
target_boxes - [1.3759361e-04 6.0747814e-05 3.6017149e-04 7.0116876e-05]
target_boxes - [1.2949987e-04 7.0419112e-05 8.2758506e-04 7.8579265e-05]
...
target_boxes - [0.0001295  0.0005289  0.00054026 0.00053706]
2 ____
target_boxes - [1.7845376e-07 2.8245601e-08 1.1426797e-06 3.3562422e-08]
target_boxes - [1.9572349e-07 3.3396269e-08 5.1233496e-07 3.8546936e-08]
target_boxes - [1.8421034e-07 3.8713090e-08 1.1772191e-06 4.3199158e-08]
...
target_boxes - [1.8421034e-07 2.9076355e-07 7.6850250e-07 2.9524961e-07]

Process finished with exit code 0
```
As you can see, boxes are changing over iterations.
 

4. *And more generally speaking, are you suggesting that we enforce a deepcopy of the target & image in the _read_sample method?* - well, yes, this could be a solution. But I left it for you to decide. Because as I understand, targets can be `Any` type as was mentioned somewhere in the sources. So using `deepcopy` in this case is not the best solution.


As a conclusion, I just want to point you at this problem. Because of a deep coherency in code structures, changes need to be supported over all lib. So there is only for you to decide which way of solving the problem is more convenient for you. I will accept any of these. :)
Looking forward to see which decision will you make. 

Best regards! 


Hey there :wave: 

I'll try to answer all your points:

- first, I'll have to say: if there is an issue with existing datasets, we wouldn't have spotted it. The reason behind this is that all our training datasets are already in relative coords, and the ones using the relative coord conversion, are datasets we use for test (we only iterate once on it). So it's very possible that there is an issue since it wouldn't have come up on our usage!

>  `convert_target_to_relative` is _**not**_ my code. I have found it in the doctr [sources](https://github.com/mindee/doctr/blob/e5ca2d536864ccff1460ede9bf3da66b13a2ab9a/doctr/datasets/utils.py#L160).

Sure, the function was implemented here, but you pointed out `AbstractDataset` which doesn't use relative conversion. So I guess you're actually using a snippet with the ones that are using it (FUNSD, SROIE, IC13, CORD, SynthText, IIIT5K, SVHN, or Imgur5k)?

> 2. _But so far, it's up to the developer_ - well, that's the reason why I haven't created a fork. Leaving the code as it is may lead to some unfortunate behavior as I found. So it would be nice to have some common guideline for samples/targets transformers. Which is generally up to you :) Either making copies in all transform methods or coping samples/targets before all transformations.

Oh yeah sure, I meant: if you implement custom transforms, it's up to you to check the behaviour you're implementing :) But here if there is an inplace trouble, it's up to us to fix it of course.

> 3. _However, you mention a change of target, but your process_image is only changing the image._ - the code in the **Code snippet to reproduce the bug** section is to represent _target changes_ (_image bounding boxes in this case_) when iterating twice over `train_set`. But yeah, you are right. My code is quite redundant. Here is a more compact version

Thanks, much clearer!

> As a conclusion, I just want to point you at this problem. Because of a deep coherency in code structures, changes need to be supported over all lib. So there is only for you to decide which way of solving the problem is more convenient for you. I will accept any of these. :) Looking forward to see which decision will you make.

I'll run some test, but most likely, I'll just make sure the target is a deepcopy in `_read_sample`. That will sterilize any inplace operations on the target :+1: 

Resolving patch:
diff --git a/doctr/datasets/datasets/pytorch.py b/doctr/datasets/datasets/pytorch.py
--- a/doctr/datasets/datasets/pytorch.py
+++ b/doctr/datasets/datasets/pytorch.py
@@ -4,6 +4,7 @@
 # See LICENSE or go to <https://www.apache.org/licenses/LICENSE-2.0.txt> for full license details.
 
 import os
+from copy import deepcopy
 from typing import Any, List, Tuple
 
 import torch
@@ -22,7 +23,7 @@ def _read_sample(self, index: int) -> Tuple[torch.Tensor, Any]:
         # Read image
         img = read_img_as_tensor(os.path.join(self.root, img_name), dtype=torch.float32)
 
-        return img, target
+        return img, deepcopy(target)
 
     @staticmethod
     def collate_fn(samples: List[Tuple[torch.Tensor, Any]]) -> Tuple[torch.Tensor, List[Any]]:
diff --git a/doctr/datasets/datasets/tensorflow.py b/doctr/datasets/datasets/tensorflow.py
--- a/doctr/datasets/datasets/tensorflow.py
+++ b/doctr/datasets/datasets/tensorflow.py
@@ -4,6 +4,7 @@
 # See LICENSE or go to <https://www.apache.org/licenses/LICENSE-2.0.txt> for full license details.
 
 import os
+from copy import deepcopy
 from typing import Any, List, Tuple
 
 import tensorflow as tf
@@ -22,7 +23,7 @@ def _read_sample(self, index: int) -> Tuple[tf.Tensor, Any]:
         # Read image
         img = read_img_as_tensor(os.path.join(self.root, img_name), dtype=tf.float32)
 
-        return img, target
+        return img, deepcopy(target)
 
     @staticmethod
     def collate_fn(samples: List[Tuple[tf.Tensor, Any]]) -> Tuple[tf.Tensor, List[Any]]:


Test patch:
diff --git a/tests/common/test_datasets.py b/tests/common/test_datasets.py
--- a/tests/common/test_datasets.py
+++ b/tests/common/test_datasets.py
@@ -42,3 +42,14 @@ def test_abstractdataset(mock_image_path):
     ds.sample_transforms = lambda x, y: (x, y + 1)
     img3, target3 = ds[0]
     assert np.all(img3.numpy() == img.numpy()) and (target3 == (target + 1))
+
+    # Check inplace modifications
+    ds.data = [(ds.data[0][0], {"label": "A"})]
+
+    def inplace_transfo(x, target):
+        target["label"] += "B"
+        return x, target
+    ds.sample_transforms = inplace_transfo
+    _, t = ds[0]
+    _, t = ds[0]
+    assert t['label'] == "AB"