Repository: pytorch/vision

Issue:
Avoid view in no_grad mode in GeneralizedRCNNTransform.batch_images
## 🚀 Feature
Thank you very much for this great project! I'm one of the maintainers of the [Adversarial Robustness Toolbox (ART)](https://github.com/Trusted-AI/adversarial-robustness-toolbox) and `torchvision` has been very useful for ART.

## Motivation

ART has a lot of tools for adversarial machine learning that require the gradients of the loss at the model input and in the case of this issue for `torchvision.models.detection.fasterrcnn_resnet50_fpn`. Since `torchvision>=0.8` we are observing the following RunTimeError when calculating loss gradients towards the input tensor, which prevents us from using `torch>=1.7` for the related tools in ART:
```python
        batch_shape = [len(images)] + max_size
        batched_imgs = images[0].new_full(batch_shape, 0)
        for img, pad_img in zip(images, batched_imgs):
>           pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img)
E           RuntimeError: A view was created in no_grad mode and is being modified inplace with grad mode enabled. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.

.../python3.7/site-packages/torchvision/models/detection/transform.py:224: RuntimeError
```

## Pitch

Would it be possible to change https://github.com/pytorch/vision/blob/ef711591a5db69d36f904ab5c39dec13627a58ad/torchvision/models/detection/transform.py#L223-L224
into
```python
for i in range(batched_imgs.shape[0]):
    batched_imgs[i][: images[i].shape[0], : images[i].shape[1], : images[i].shape[2]].copy_(images[i])
```
to avoid the RunTimeError?

If yes, I would be happy to open a pull request with the proposed solution.




Discussion:
For ref, a minimal reproducing example is

```py
import torch
from torchvision.models.detection.transform import GeneralizedRCNNTransform

t = GeneralizedRCNNTransform(min_size=10, max_size=20, image_mean=[0, 0, 0], image_std=[1, 1, 1])
img = torch.rand(10, 3, 4, 5, requires_grad=True)
t(img)[0]
```


I think the proposed changes are reasonable, but it does raise the question of whether torchvision should support `requires_grad` for inputs in general. If this is a desired feature, we might need to introduce dedicated tests for all transforms / ops / models...
Thoughts @fmassa?
Might just be my 2 cents, but I think it's fairly reasonable to expect the inputs to require grad by default - it's IMHO easy enough to wrap the operations that don't need it with `torch.no_grad` or detach them if one doesn't need them or wants to save on performance. 
Hi @NicolasHug and @fmassa Did you have a chance to further consider this issue? We would be happy to help with contributions.
@beat-buesser I'm happy to change the implementation in torchvision to support this, although from the error message maybe something else might need to be changed. Still, if you could send a PR with tests we would be glad to review it and get it merged!
@fmassa WDYT of my comment above?

> I think the proposed changes are reasonable, but it does raise the question of whether torchvision should support requires_grad for inputs in general. If this is a desired feature, we might need to introduce dedicated tests for all transforms / ops / models...
You need this to support adversarial applications.
I have also encountered this issue since `torchvision>=0.8`. If anyone has a workaround for computing the gradient of the RCNN with respect to the input I would be grateful if you could share it.
@fmassa Thank you very much. I'm happy to sign up for this issue and I'll try to open a pull request soon.
@NicolasHug I think supporting requires_grad for inputs is reasonable, as @datumbox mentioned it is useful for some applications to backprop through the input as well.

@beat-buesser just send a PR fixing this and we will be happy to review / merge it

Resolving patch:
diff --git a/torchvision/models/detection/transform.py b/torchvision/models/detection/transform.py
--- a/torchvision/models/detection/transform.py
+++ b/torchvision/models/detection/transform.py
@@ -214,8 +214,9 @@ def batch_images(self, images: List[Tensor], size_divisible: int = 32) -> Tensor
 
         batch_shape = [len(images)] + max_size
         batched_imgs = images[0].new_full(batch_shape, 0)
-        for img, pad_img in zip(images, batched_imgs):
-            pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img)
+        for i in range(batched_imgs.shape[0]):
+            img = images[i]
+            batched_imgs[i, : img.shape[0], : img.shape[1], : img.shape[2]].copy_(img)
 
         return batched_imgs
 


Test patch:
diff --git a/test/test_models.py b/test/test_models.py
--- a/test/test_models.py
+++ b/test/test_models.py
@@ -148,6 +148,35 @@ def _check_fx_compatible(model, inputs):
     torch.testing.assert_close(out, out_fx)
 
 
+def _check_input_backprop(model, inputs):
+    if isinstance(inputs, list):
+        requires_grad = list()
+        for inp in inputs:
+            requires_grad.append(inp.requires_grad)
+            inp.requires_grad_(True)
+    else:
+        requires_grad = inputs.requires_grad
+        inputs.requires_grad_(True)
+
+    out = model(inputs)
+
+    if isinstance(out, dict):
+        out["out"].sum().backward()
+    else:
+        if isinstance(out[0], dict):
+            out[0]["scores"].sum().backward()
+        else:
+            out[0].sum().backward()
+
+    if isinstance(inputs, list):
+        for i, inp in enumerate(inputs):
+            assert inputs[i].grad is not None
+            inp.requires_grad_(requires_grad[i])
+    else:
+        assert inputs.grad is not None
+        inputs.requires_grad_(requires_grad)
+
+
 # If 'unwrapper' is provided it will be called with the script model outputs
 # before they are compared to the eager model outputs. This is useful if the
 # model outputs are different between TorchScript / Eager mode
@@ -263,6 +292,9 @@ def test_memory_efficient_densenet(model_name):
     assert num_params == num_grad
     torch.testing.assert_close(out1, out2, rtol=0.0, atol=1e-5)
 
+    _check_input_backprop(model1, x)
+    _check_input_backprop(model2, x)
+
 
 @pytest.mark.parametrize('dilate_layer_2', (True, False))
 @pytest.mark.parametrize('dilate_layer_3', (True, False))
@@ -312,6 +344,7 @@ def test_inception_v3_eval():
     model = model.eval()
     x = torch.rand(1, 3, 299, 299)
     _check_jit_scriptable(model, (x,), unwrapper=script_model_unwrapper.get(name, None))
+    _check_input_backprop(model, x)
 
 
 def test_fasterrcnn_double():
@@ -327,6 +360,7 @@ def test_fasterrcnn_double():
     assert "boxes" in out[0]
     assert "scores" in out[0]
     assert "labels" in out[0]
+    _check_input_backprop(model, model_input)
 
 
 def test_googlenet_eval():
@@ -343,6 +377,7 @@ def test_googlenet_eval():
     model = model.eval()
     x = torch.rand(1, 3, 224, 224)
     _check_jit_scriptable(model, (x,), unwrapper=script_model_unwrapper.get(name, None))
+    _check_input_backprop(model, x)
 
 
 @needs_cuda
@@ -369,6 +404,8 @@ def checkOut(out):
 
     checkOut(out)
 
+    _check_input_backprop(model, model_input)
+
     # now switch to cpu and make sure it works
     model.cpu()
     x = x.cpu()
@@ -376,6 +413,8 @@ def checkOut(out):
 
     checkOut(out_cpu)
 
+    _check_input_backprop(model, [x])
+
 
 def test_generalizedrcnn_transform_repr():
 
@@ -426,6 +465,8 @@ def test_classification_model(model_name, dev):
                 _assert_expected(out.cpu(), model_name, prec=0.1)
             assert out.shape[-1] == 50
 
+    _check_input_backprop(model, x)
+
 
 @pytest.mark.parametrize('model_name', get_available_segmentation_models())
 @pytest.mark.parametrize('dev', cpu_and_gpu())
@@ -483,6 +524,8 @@ def check_out(out):
         warnings.warn(msg, RuntimeWarning)
         pytest.skip(msg)
 
+    _check_input_backprop(model, x)
+
 
 @pytest.mark.parametrize('model_name', get_available_detection_models())
 @pytest.mark.parametrize('dev', cpu_and_gpu())
@@ -574,6 +617,8 @@ def compute_mean_std(tensor):
         warnings.warn(msg, RuntimeWarning)
         pytest.skip(msg)
 
+    _check_input_backprop(model, model_input)
+
 
 @pytest.mark.parametrize('model_name', get_available_detection_models())
 def test_detection_model_validation(model_name):
@@ -625,6 +670,8 @@ def test_video_model(model_name, dev):
             out = model(x)
             assert out.shape[-1] == 50
 
+    _check_input_backprop(model, x)
+
 
 @pytest.mark.skipif(not ('fbgemm' in torch.backends.quantized.supported_engines and
                          'qnnpack' in torch.backends.quantized.supported_engines),