Repository: pytorch/TensorRT

Issue:
🐛 [Bug] RuntimeError: Target aten.convolution.default does not support `transposed=True` when using torch_tensorrt as backend in torch.compile
##  Bug Description

When trying to compile a simple PyTorch module with `torch.compile(model, backend="torch_tensorrt")`, I get:

```
RuntimeError: Target aten.convolution.default does not support `transposed=True`
```

I don't get any errors when using `torch.compile(model)` or `torch_tensorrt.compile(model, inputs=inputs)`.
## To Reproduce

Steps to reproduce the behavior:

```python3
import torch
import torch_tensorrt

from torch import nn


class Up(nn.Module):
    def __init__(self, in_channels, out_channels, upsample_stride):
        super().__init__()
        self.up = nn.Sequential(
            nn.ConvTranspose2d(
                in_channels,
                out_channels,
                upsample_stride,
                stride=upsample_stride,
                bias=False,
            ),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
        )

    def forward(self, x):
        return self.up(x)


def main():
    device = torch.device("cuda:0")
    model = Up(64, 128, 2).to(device)
    model.eval()
    print(model)

    x = torch.rand((1, 64, 100, 100)).to(device)
    # Works.
    with torch.no_grad():
        _ = model(x)

    model_opt = torch.compile(model)
    # Works.
    with torch.no_grad():
        _ = model_opt(x)

    torch._dynamo.reset()
    model_opt = torch.compile(model, backend="torch_tensorrt")
    # RuntimeError.
    with torch.no_grad():
        _ = model_opt(x)

    torch._dynamo.reset()
    inputs = [torch_tensorrt.Input(x.shape)]
    # Works.
    trt_ts_module = torch_tensorrt.compile(model, inputs=inputs)


if __name__ == "__main__":
    main()
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

> Build information about Torch-TensorRT can be found by turning on debug messages

 - Torch-TensorRT Version (e.g. 1.0.0): `1.4.0`
 - PyTorch Version (e.g. 1.0): `2.0.1+cu117`
 - CPU Architecture: i7-12800H
 - OS (e.g., Linux): Linux
 - How you installed PyTorch (`conda`, `pip`, `libtorch`, source): `pip`
 - Build command you used (if compiling from source):
 - Are you using local sources or building from archives:
 - Python version: 3.10.10
 - CUDA version: 12.2
 - GPU models and configuration: GeForce RTX 3080 Ti
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->



Discussion:
Hi @airalcorn2  as mentioned in #2506, this error should not come in the recent main of `torch tensorrt.`
At my end I am seeing this error in the above,
```
File "/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch_tensorrt/fx/converters/converter_utils.py", line 400, in broadcast
    a_shape = tuple(a.shape)
torch._dynamo.exc.BackendCompilerFailed: backend='torch_tensorrt' raised:
ValueError: __len__() should return >= 0

```
I suspect this is due to incorrect processing of `tensorrt_bindings.tensorrt.Dims`. I am looking into this further.
Hi @apbose - I am able to reproduce this error, but it seems to be an issue with `convolution` and not with the Batch Norm operator, since I can get it to pass with
```python
"torch_executed_ops": {"torch.ops.aten.convolution.default"}
```
The TRT error appears for the convolution operator, and below is a minimal repro:
```python
import torch
import torch_tensorrt

from torch import nn


class Up(nn.Module):
    def __init__(self, in_channels, out_channels, upsample_stride):
        super().__init__()
        self.up = nn.ConvTranspose2d(
                in_channels,
                out_channels,
                upsample_stride,
                stride=upsample_stride,
                bias=False,
            ),

    def forward(self, x):
        return self.up(x)


def main():
    device = torch.device("cuda:0")
    model = Up(64, 128, 2).to(device)
    model.eval()
    print(model)

    x = torch.rand((1, 64, 100, 100)).to(device)
    model_opt = torch.compile(model, backend="torch_tensorrt",
                              options={"min_block_size": 1,
                                       "debug": True,})
    with torch.no_grad():
        _ = model_opt(x)


if __name__ == "__main__":
    main()
```
@zewenli98 has some experience with the convolution operators - could you take a look?
Hi, I just updated to the latest environment:
```
torch                    2.2.0.dev20231211+cu121
torch-tensorrt           2.2.0.dev0+f0e6d2d29
torchvision              0.17.0.dev20231211+cu121
```
For the code below:
> ```python
> import torch
> import torch_tensorrt
> 
> from torch import nn
> 
> 
> class Up(nn.Module):
>     def __init__(self, in_channels, out_channels, upsample_stride):
>         super().__init__()
>         self.up = nn.Sequential(
>             nn.ConvTranspose2d(
>                 in_channels,
>                 out_channels,
>                 upsample_stride,
>                 stride=upsample_stride,
>                 bias=False,
>             ),
>             nn.BatchNorm2d(out_channels),
>             nn.ReLU(inplace=True),
>         )
> 
>     def forward(self, x):
>         return self.up(x)
> 
> 
> def main():
>     device = torch.device("cuda:0")
>     model = Up(64, 128, 2).to(device)
>     model.eval()
>     print(model)
> 
>     x = torch.rand((1, 64, 100, 100)).to(device)
>     # Works.
>     with torch.no_grad():
>         _ = model(x)
> 
>     model_opt = torch.compile(model)
>     # Works.
>     with torch.no_grad():
>         _ = model_opt(x)
> 
>     torch._dynamo.reset()
>     model_opt = torch.compile(model, backend="torch_tensorrt")
>     # RuntimeError.
>     with torch.no_grad():
>         _ = model_opt(x)
> 
>     torch._dynamo.reset()
>     inputs = [torch_tensorrt.Input(x.shape)]
>     # Works.
>     trt_ts_module = torch_tensorrt.compile(model, inputs=inputs)
> 
> 
> if __name__ == "__main__":
>     main()
> ```

@airalcorn2 I tried this code but there seems to be no error on my side. The output is something like:
```
...
DEBUG: [Torch-TensorRT Plugins Context] - Registered plugin creator - SpecialSlice_TRT, Namespace: 
DEBUG: [Torch-TensorRT Plugins Context] - Registered plugin creator - Split, Namespace: 
DEBUG: [Torch-TensorRT Plugins Context] - Registered plugin creator - VoxelGeneratorPlugin, Namespace: 
DEBUG: [Torch-TensorRT Plugins Context] - Total number of plugins registered: 69
Up(
  (up): Sequential(
    (0): ConvTranspose2d(64, 128, kernel_size=(2, 2), stride=(2, 2), bias=False)
    (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
  )
)
INFO:torch_tensorrt.dynamo.utils:Using Default Torch-TRT Runtime (as requested by user)
INFO:torch_tensorrt.dynamo.utils:Device not specified, using Torch default current device - cuda:0. If this is incorrect, please specify an input device, via the device keyword.
INFO:torch_tensorrt.dynamo.utils:Compilation Settings: CompilationSettings(precision=torch.float32, debug=False, workspace_size=0, min_block_size=5, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_long_and_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, sparse_weights=False, refit=False, engine_capability=<EngineCapability.DEFAULT: 0>, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912)

WARNING:torch_tensorrt.dynamo._compiler:4 supported operations detected in subgraph containing 4 computational nodes. Skipping this subgraph, since min_block_size was detected to be 5
INFO:torch_tensorrt._compile:ir was set to default, using dynamo as ir
INFO:torch_tensorrt.dynamo._compiler:Compilation Settings: CompilationSettings(precision=torch.float32, debug=False, workspace_size=0, min_block_size=5, torch_executed_ops=[], pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_long_and_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, sparse_weights=False, refit=False, engine_capability=<EngineCapability.DEFAULT: 0>, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912)

WARNING:torch_tensorrt.dynamo._compiler:4 supported operations detected in subgraph containing 4 computational nodes. Skipping this subgraph, since min_block_size was detected to be 5
```
Could you try to update your environment and then take a look?
> ```python
> import torch
> import torch_tensorrt
> 
> from torch import nn
> 
> 
> class Up(nn.Module):
>     def __init__(self, in_channels, out_channels, upsample_stride):
>         super().__init__()
>         self.up = nn.ConvTranspose2d(
>                 in_channels,
>                 out_channels,
>                 upsample_stride,
>                 stride=upsample_stride,
>                 bias=False,
>             ),
> 
>     def forward(self, x):
>         return self.up(x)
> 
> 
> def main():
>     device = torch.device("cuda:0")
>     model = Up(64, 128, 2).to(device)
>     model.eval()
>     print(model)
> 
>     x = torch.rand((1, 64, 100, 100)).to(device)
>     model_opt = torch.compile(model, backend="torch_tensorrt",
>                               options={"min_block_size": 1,
>                                        "debug": True,})
>     with torch.no_grad():
>         _ = model_opt(x)
> 
> 
> if __name__ == "__main__":
>     main()
> ```

After updating to the latest environment, running the code above, I got error:
```
DEBUG: [Torch-TensorRT Plugins Context] - Registered plugin creator - Split, Namespace: 
DEBUG: [Torch-TensorRT Plugins Context] - Registered plugin creator - VoxelGeneratorPlugin, Namespace: 
DEBUG: [Torch-TensorRT Plugins Context] - Total number of plugins registered: 69
Up()
Traceback (most recent call last):
  File "../lowering/tmp.py", line 126, in <module>
    main()
  File "../lowering/tmp.py", line 122, in main
    _ = model_opt(x)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 501, in _fn
    return fn(*args, **kwargs)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 667, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 730, in _convert_frame
    result = inner_convert(frame, cache_entry, hooks, frame_state)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 383, in _convert_frame_assert
    compiled_product = _compile(
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 665, in _compile
    raise InternalTorchDynamoError(str(e)).with_traceback(
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 646, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 562, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
    transformations(instructions, code_options)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
    return fn(*args, **kwargs)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 527, in transform
    tracer.run()
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2096, in run
    super().run()
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in run
    and self.step()
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 744, in step
    getattr(self, inst.opname)(inst)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 463, in wrapper
    return inner_fn(self, inst)
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1176, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 639, in call_function
    if hasattr(fn, "value"):
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/variables/lists.py", line 69, in value
    return self.as_python_constant()
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/variables/lists.py", line 72, in as_python_constant
    return self.python_type()([x.as_python_constant() for x in self.items])
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/variables/lists.py", line 72, in <listcomp>
    return self.python_type()([x.as_python_constant() for x in self.items])
  File "/home/zewenl/anaconda3/envs/env1/lib/python3.8/site-packages/torch/_dynamo/variables/base.py", line 237, in as_python_constant
    raise NotImplementedError(f"{self} is not a constant")
torch._dynamo.exc.InternalTorchDynamoError: NNModuleVariable() is not a constant

from user code:
   File "../lowering/tmp.py", line 108, in forward
    return self.up(x)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True
```
@gs-olive Did you get the same error like this?

Hi @zewenli98 - I do see the same error when using your script, but it is due to the extra comma which appears after the `self.up` line. If you remove that comma, I see this error, as before:
```python
[12/11/2023-18:07:03] [TRT] [E] 3: [DECONVOLUTION]-[aten_ops.convolution.default]-[convolution]:kernel weights has count 32768 but 16384 was expected
[12/11/2023-18:07:03] [TRT] [E] 4: [DECONVOLUTION]-[aten_ops.convolution.default]-[convolution]: count of 32768 weights in kernel, but kernel dimensions (2,2) with 64 input channels, 64 output channels and 1 groups were specified. Expected Weights count is 64 * 2*2 * 64 / 1 = 16384
[12/11/2023-18:07:03] [TRT] [E] 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ([DECONVOLUTION]-[aten_ops.convolution.default]-[convolution]: output shape can not be computed)
[12/11/2023-18:07:03] [TRT] [E] 3: [DECONVOLUTION]-[aten_ops.convolution.default]-[convolution]:kernel weights has count 32768 but 16384 was expected
[12/11/2023-18:07:03] [TRT] [E] 4: [DECONVOLUTION]-[aten_ops.convolution.default]-[convolution]: count of 32768 weights in kernel, but kernel dimensions (2,2) with 64 input channels, 64 output channels and 1 groups were specified. Expected Weights count is 64 * 2*2 * 64 / 1 = 16384
[12/11/2023-18:07:03] [TRT] [E] 4: [network.cpp::validate::3121] Error Code 4: Internal Error (Layer [DECONVOLUTION]-[aten_ops.convolution.default]-[convolution] failed validation)
```

Resolving patch:
diff --git a/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py b/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py
--- a/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py
+++ b/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py
@@ -2106,7 +2106,7 @@ def aten_ops_convolution(
             is_conv1d=len(args[3]) == 1,
             input=args[0],
             weight=args[1],
-            bias=args[2],
+            bias=args_bounds_check(args, 2, None),
             stride=args[3],
             padding=args[4],
             dilation=args[5],
@@ -2121,7 +2121,7 @@ def aten_ops_convolution(
             is_deconv1d=len(args[3]) == 1,
             input=args[0],
             weight=args[1],
-            bias=args[2],
+            bias=args_bounds_check(args, 2, None),
             stride=args[3],
             padding=args[4],
             dilation=args[5],
diff --git a/py/torch_tensorrt/dynamo/conversion/impl/conv.py b/py/torch_tensorrt/dynamo/conversion/impl/conv.py
--- a/py/torch_tensorrt/dynamo/conversion/impl/conv.py
+++ b/py/torch_tensorrt/dynamo/conversion/impl/conv.py
@@ -32,10 +32,11 @@ def convNd(
     input: TRTTensor,
     weight: Union[TRTTensor, torch.Tensor, np.ndarray],
     bias: Optional[Union[TRTTensor, torch.Tensor, np.ndarray]],
-    stride: Optional[Union[int, Sequence[int]]],
-    padding: Optional[Union[int, Sequence[int]]],
-    dilation: Optional[Union[int, Sequence[int]]],
+    stride: Union[int, Sequence[int]],
+    padding: Union[int, Sequence[int]],
+    dilation: Union[int, Sequence[int]],
     groups: Optional[int],
+    output_padding: Union[int, Sequence[int]] = 0,
     scale: Optional[Union[torch.Tensor, float]] = None,
     zero_point: Optional[Union[torch.Tensor, float]] = None,
 ) -> TRTTensor:
diff --git a/py/torch_tensorrt/dynamo/conversion/impl/deconv.py b/py/torch_tensorrt/dynamo/conversion/impl/deconv.py
--- a/py/torch_tensorrt/dynamo/conversion/impl/deconv.py
+++ b/py/torch_tensorrt/dynamo/conversion/impl/deconv.py
@@ -32,10 +32,11 @@ def deconvNd(
     input: TRTTensor,
     weight: Union[TRTTensor, torch.Tensor, np.ndarray],
     bias: Optional[Union[TRTTensor, torch.Tensor, np.ndarray]],
-    stride: Optional[Union[int, Sequence[int]]],
-    padding: Optional[Union[int, Sequence[int]]],
+    stride: Union[int, Sequence[int]],
+    padding: Union[int, Sequence[int]],
+    dilation: Union[int, Sequence[int]],
     groups: Optional[int],
-    dilation: Optional[Union[int, Sequence[int]]],
+    output_padding: Union[int, Sequence[int]] = 0,
     scale: Optional[Union[torch.Tensor, float]] = None,
     zero_point: Optional[Union[torch.Tensor, float]] = None,
 ) -> TRTTensor:
@@ -86,7 +87,7 @@ def deconvNd(
     # add deconv layer
     deconv_layer = ctx.net.add_deconvolution_nd(
         input=input,
-        num_output_maps=weight.shape[0],
+        num_output_maps=weight.shape[1] * groups,
         kernel_shape=weight.shape[2:],
         kernel=trt.Weights() if isinstance(weight, TRTTensor) else weight,
         bias=trt.Weights() if isinstance(bias, TRTTensor) else bias,


Test patch:
diff --git a/tests/py/dynamo/backend/test_specialized_models.py b/tests/py/dynamo/backend/test_specialized_models.py
--- a/tests/py/dynamo/backend/test_specialized_models.py
+++ b/tests/py/dynamo/backend/test_specialized_models.py
@@ -365,5 +365,39 @@ def forward(self, x, y):
         torch._dynamo.reset()
 
 
+class TestDeconvolution(TestCase):
+    def test_ConvTranspose2d(self):
+        class Up(torch.nn.Module):
+            def __init__(self, in_channels, out_channels, upsample_stride):
+                super().__init__()
+                self.up = torch.nn.ConvTranspose2d(
+                    in_channels,
+                    out_channels,
+                    upsample_stride,
+                    stride=upsample_stride,
+                    bias=False,
+                )
+
+            def forward(self, x):
+                return self.up(x)
+
+        device = torch.device("cuda:0")
+        model = Up(64, 128, 2).to(device)
+        model.eval()
+        print(model)
+
+        x = torch.rand((1, 64, 100, 100)).to(device)
+        model_opt = torch.compile(
+            model,
+            backend="torch_tensorrt",
+            options={
+                "min_block_size": 1,
+                "debug": True,
+            },
+        )
+        with torch.no_grad():
+            _ = model_opt(x)
+
+
 if __name__ == "__main__":
     run_tests()