Repository: airctic/icevision

Issue:
Rescale transforms and torchvision RCNNs
## 🐛 Bug
torchvisions `FasterRCNN` and `MaskRCNN` internally rescales the images via `GeneralizedRCNNTransform`.

This means that any resizing transform previously applied to the image (probably on the `Dataset` stage) will be unexpectedly  overriden.

It becomes really confusing if we should apply a resize transform via `Dataset` or via the model, ideally we want all transform to be applied at the `Dataset` stage. Going one step further, can this even be said about normalization?

## Solution 1
Changes the default of `min_size` and `max_size` in the model to 2 and 9999 respectively, practically making them non effective.

## Solution 2
Monkey patch `model.transform` to a function that returns the same items it receives (being careful with what are the expected output types)

---
Solution 1 seems like a better idea, it's less magical, we can still have `min/max_size` as parameters, we just change their defaults.

---
Should we do the same with normalize? Changing the mean to 0 and std to 1? It's them clear that if we want to normalize we should do it in the `Dataset` stage.

I'm thinking this because models that we integrate in the future might not have this internal transforms, and it will be very confusing to a new user why sometimes he has to explicity define normalization and sometimes not.



Discussion:
@oke-aditya any thoughts?
Generalized RCNN transform allows user to pass an image of arbitrary size and RCNN takes care of scaling etc. 

But if the user is very specific with his training size images then he may pass images of specific dimension too. And it will work fine.

Yes, future models may not have internal transforms that we think have as of now. So I guess we need to be clear with this issue. I am still unsure of solution.



Resolving patch:
diff --git a/mantisshrimp/models/mantis_rcnn/mantis_faster_rcnn.py b/mantisshrimp/models/mantis_rcnn/mantis_faster_rcnn.py
--- a/mantisshrimp/models/mantis_rcnn/mantis_faster_rcnn.py
+++ b/mantisshrimp/models/mantis_rcnn/mantis_faster_rcnn.py
@@ -4,7 +4,6 @@
 from mantisshrimp.core import *
 from mantisshrimp.models.mantis_rcnn.rcnn_param_groups import *
 from mantisshrimp.models.mantis_rcnn.mantis_rcnn import *
-from mantisshrimp.backbones import *
 
 
 class MantisFasterRCNN(MantisRCNN):
@@ -22,6 +21,7 @@ def __init__(
         backbone: nn.Module = None,
         param_groups: List[nn.Module] = None,
         metrics=None,
+        remove_internal_transforms=True,
         **kwargs,
     ):
         super().__init__(metrics=metrics)
@@ -42,6 +42,9 @@ def __init__(
         self._param_groups = param_groups + [self.model.rpn, self.model.roi_heads]
         check_all_model_params_in_groups(self.model, self._param_groups)
 
+        if remove_internal_transforms:
+            self._remove_transforms_from_model(self.model)
+
     def forward(self, images, targets=None):
         return self.model(images, targets)
 
diff --git a/mantisshrimp/models/mantis_rcnn/mantis_mask_rcnn.py b/mantisshrimp/models/mantis_rcnn/mantis_mask_rcnn.py
--- a/mantisshrimp/models/mantis_rcnn/mantis_mask_rcnn.py
+++ b/mantisshrimp/models/mantis_rcnn/mantis_mask_rcnn.py
@@ -5,7 +5,6 @@
 from mantisshrimp.models.mantis_rcnn.rcnn_param_groups import *
 from mantisshrimp.models.mantis_rcnn.mantis_rcnn import *
 from mantisshrimp.models.mantis_rcnn.mantis_faster_rcnn import *
-from mantisshrimp.backbones import *
 
 
 class MantisMaskRCNN(MantisRCNN):
@@ -15,6 +14,7 @@ def __init__(
         num_classes: int,
         backbone: nn.Module = None,
         param_groups: List[nn.Module] = None,
+        remove_internal_transforms: bool = True,
         **kwargs,
     ):
         super().__init__()
@@ -41,6 +41,9 @@ def __init__(
         self._param_groups = param_groups + [self.model.rpn, self.model.roi_heads]
         check_all_model_params_in_groups(self.model, self.param_groups)
 
+        if remove_internal_transforms:
+            self._remove_transforms_from_model(self.model)
+
     def forward(self, images, targets=None):
         return self.model(images, targets)
 
diff --git a/mantisshrimp/models/mantis_rcnn/mantis_rcnn.py b/mantisshrimp/models/mantis_rcnn/mantis_rcnn.py
--- a/mantisshrimp/models/mantis_rcnn/mantis_rcnn.py
+++ b/mantisshrimp/models/mantis_rcnn/mantis_rcnn.py
@@ -27,6 +27,16 @@ def _predict(
 
         return [convert_raw_prediction(raw_pred) for raw_pred in raw_preds]
 
+    def _remove_transforms_from_model(self, model: GeneralizedRCNN):
+        def noop_normalize(image):
+            return image
+
+        def noop_resize(image, target):
+            return image, target
+
+        model.transform.normalize = noop_normalize
+        model.transform.resize = noop_resize
+
     @staticmethod
     def loss(preds, targs) -> Tensor:
         return sum(preds.values())


Test patch: