Repository: open-mmlab/mmaction2

Issue:
RuntimeError: Expected 5-dimensional input for 5-dimensional weight [64, 3, 3, 7, 7], but got 4-dimensional input of size [64, 1, 256, 256] instead
Hi, there, i have succefully converted TSN .pth model to .onnx format in colab, and now i want to convert other models like ircsnto to onnx format, but i came across an error as the title described. The command i run in colab : `!python tools/pytorch2onnx.py ./configs/recognition/csn/ircsn_ig65m_pretrained_bnfrozen_r152_32x2x1_58e_kinetics400_rgb.py ./checkpoints/ircsn_ig65m_pretrained_bnfrozen_r152_32x2x1_58e_kinetics400_rgb_20200812-9037a758.pth --shape 32 2 1 256 256 --verify --show`
Its output logs are : 
```
Traceback (most recent call last):
  File "tools/pytorch2onnx.py", line 163, in <module>
    verify=args.verify)
  File "tools/pytorch2onnx.py", line 74, in pytorch2onnx
    opset_version=opset_version)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 488, in _export
    fixed_batch_size=fixed_batch_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 334, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 291, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, _force_outplace=False, _return_inputs_states=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 278, in _get_trace_graph
    outs = ONNXTracedModule(f, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 361, in forward
    self._force_outplace,
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 348, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/recognizers/recognizer3d.py", line 59, in forward_dummy
    x = self.extract_feat(imgs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
    return old_func(*args, **kwargs)
  File "/content/mmaction2/mmaction/models/recognizers/base.py", line 72, in extract_feat
    x = self.backbone(imgs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/backbones/resnet3d.py", line 795, in forward
    x = self.conv1(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/cnn/bricks/conv_module.py", line 192, in forward
    x = self.conv(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/cnn/bricks/wrappers.py", line 79, in forward
    return super().forward(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 485, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected 5-dimensional input for 5-dimensional weight [64, 3, 3, 7, 7], but got 4-dimensional input of size [64, 1, 256, 256] instead
```
I also tried on i3d and tpn, they all have similiar errors, what is the common reason for this type of error ? Thanks !



Discussion:
The shape is not correct. Check this doc out:

--shape: The shape of input tensor to the model. For 2D recognizer(e.g. TSN), the input should be $batch $clip $channel $height $width(e.g. 1 1 3 224 224); For 3D recognizer(e.g. I3D), the input should be $batch $clip $channel $time $height $width(e.g. 1 1 3 32 224 224); For localizer such as BSN, the input for each module is different, please check the forward function for it. If not specified, it will be set to 1 1 3 224 224.
Ah, I overlooked this point ! The extra parameter `time` means sampling frame interval ？
I have generated following onnx files, i find the `$time $height $width` do not affect file size, is this reasonable ？
![image](https://user-images.githubusercontent.com/43233772/101730622-57d66900-3af5-11eb-82c1-05578b390ab9.png)
By the way, should `$batch $clip $channel`  always be `1 1 3` ?

But this input  tensor shape (1 1 3 32 224 224) is not working with TPN, its output logs are :
```
Traceback (most recent call last):
  File "tools/pytorch2onnx.py", line 163, in <module>
    verify=args.verify)
  File "tools/pytorch2onnx.py", line 74, in pytorch2onnx
    opset_version=opset_version)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 488, in _export
    fixed_batch_size=fixed_batch_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 334, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 291, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, _force_outplace=False, _return_inputs_states=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 278, in _get_trace_graph
    outs = ONNXTracedModule(f, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 361, in forward
    self._force_outplace,
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 348, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/recognizers/recognizer3d.py", line 60, in forward_dummy
    outs = (self.cls_head(x), )
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/mmaction2/mmaction/models/heads/tpn_head.py", line 48, in forward
    kernel_size = (1, x.shape[-2], x.shape[-1])
AttributeError: 'tuple' object has no attribute 'shape'
```

> Ah, I overlooked this point ! The extra parameter `time` means sampling frame interval ？
> I have generated following onnx files, i find the `$time $height $width` do not affect file size, is this reasonable ？
> ![image](https://user-images.githubusercontent.com/43233772/101730622-57d66900-3af5-11eb-82c1-05578b390ab9.png)
> By the way, should `$batch $clip $channel` always be `1 1 3` ?
> 
> But this input tensor shape (1 1 3 32 224 224) is not working with TPN, its output logs are :
> 
> ```
> Traceback (most recent call last):
>   File "tools/pytorch2onnx.py", line 163, in <module>
>     verify=args.verify)
>   File "tools/pytorch2onnx.py", line 74, in pytorch2onnx
>     opset_version=opset_version)
>   File "/usr/local/lib/python3.6/dist-packages/torch/onnx/__init__.py", line 168, in export
>     custom_opsets, enable_onnx_checker, use_external_data_format)
>   File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 69, in export
>     use_external_data_format=use_external_data_format)
>   File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 488, in _export
>     fixed_batch_size=fixed_batch_size)
>   File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 334, in _model_to_graph
>     graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
>   File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 291, in _trace_and_get_graph_from_model
>     torch.jit._get_trace_graph(model, args, _force_outplace=False, _return_inputs_states=True)
>   File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 278, in _get_trace_graph
>     outs = ONNXTracedModule(f, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
>   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 361, in forward
>     self._force_outplace,
>   File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 348, in wrapper
>     outs.append(self.inner(*trace_inputs))
>   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
>     result = self._slow_forward(*input, **kwargs)
>   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
>     result = self.forward(*input, **kwargs)
>   File "/content/mmaction2/mmaction/models/recognizers/recognizer3d.py", line 60, in forward_dummy
>     outs = (self.cls_head(x), )
>   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 548, in __call__
>     result = self._slow_forward(*input, **kwargs)
>   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 534, in _slow_forward
>     result = self.forward(*input, **kwargs)
>   File "/content/mmaction2/mmaction/models/heads/tpn_head.py", line 48, in forward
>     kernel_size = (1, x.shape[-2], x.shape[-1])
> AttributeError: 'tuple' object has no attribute 'shape'
> ```

It seems the `neck` is somehow skipped
> It seems the neck is somehow skipped

@dreamerlin True, `forward_dummy` is used as the inference code. There is no neck in it.
> Ah, I overlooked this point ! The extra parameter time means sampling frame interval ？
I have generated following onnx files, i find the $time $height $width do not affect file size, is this reasonable ？
image
By the way, should $batch $clip $channel always be 1 1 3 ?

1. `time` means the `clip_length`. 2. The size is determined mostly by the number of parameters. Therefore it is not input-sensitive. 3. `1 1 3` would work.

>But this input tensor shape (1 1 3 32 224 224) is not working with TPN, its output logs are :

There is a bug for TPN conversion, which uses a neck module. We will fix it.

Resolving patch:
diff --git a/mmaction/models/recognizers/recognizer2d.py b/mmaction/models/recognizers/recognizer2d.py
--- a/mmaction/models/recognizers/recognizer2d.py
+++ b/mmaction/models/recognizers/recognizer2d.py
@@ -91,6 +91,16 @@ def forward_dummy(self, imgs):
         num_segs = imgs.shape[0] // batches
 
         x = self.extract_feat(imgs)
+        if hasattr(self, 'neck'):
+            x = [
+                each.reshape((-1, num_segs) +
+                             each.shape[1:]).transpose(1, 2).contiguous()
+                for each in x
+            ]
+            x, _ = self.neck(x)
+            x = x.squeeze(2)
+            num_segs = 1
+
         outs = (self.cls_head(x, num_segs), )
         return outs
 
diff --git a/mmaction/models/recognizers/recognizer3d.py b/mmaction/models/recognizers/recognizer3d.py
--- a/mmaction/models/recognizers/recognizer3d.py
+++ b/mmaction/models/recognizers/recognizer3d.py
@@ -55,8 +55,11 @@ def forward_dummy(self, imgs):
             Tensor: Class score.
         """
         imgs = imgs.reshape((-1, ) + imgs.shape[2:])
-
         x = self.extract_feat(imgs)
+
+        if hasattr(self, 'neck'):
+            x, _ = self.neck(x)
+
         outs = (self.cls_head(x), )
         return outs
 


Test patch:
diff --git a/tests/test_models/test_recognizers.py b/tests/test_models/test_recognizers.py
--- a/tests/test_models/test_recognizers.py
+++ b/tests/test_models/test_recognizers.py
@@ -389,6 +389,14 @@ def test_tpn():
         for one_img in img_list:
             recognizer(one_img, None, return_loss=False)
 
+    # Test forward dummy
+    with torch.no_grad():
+        img_list = [img[None, :] for img in imgs]
+        if hasattr(model, 'forward_dummy'):
+            model.forward = model.forward_dummy
+        for one_img in img_list:
+            recognizer(one_img, None, return_loss=False)
+
     # Test forward gradcam
     recognizer(imgs, gradcam=True)
     for one_img in img_list:
@@ -416,6 +424,14 @@ def test_tpn():
         for one_img in img_list:
             recognizer(one_img, None, return_loss=False)
 
+    # Test dummy forward
+    with torch.no_grad():
+        img_list = [img[None, :] for img in imgs]
+        if hasattr(model, 'forward_dummy'):
+            model.forward = model.forward_dummy
+        for one_img in img_list:
+            recognizer(one_img, None, return_loss=False)
+
     # Test forward gradcam
     recognizer(imgs, gradcam=True)
     for one_img in img_list: