Repository: pytorch/TensorRT (language: python)

Issue:
✨[Feature] Dynamo No-Op Repair System
# Context
In TensorRT, there are certain rules and restrictions regarding tensor I/O which are not entirely in line with those in Torch. For instance, outputs from TRT engines cannot be returned more than once, whereas `return (x, y, y)` is a valid expression in Torch. Similarly, inputs in TensorRT cannot be returned immediately as output.

# Current Solution
The current solution is to copy all tensors upon any no-op operations being encountered (`aten._to_copy`, `aten.copy`, `aten.clone`, and others). This solution suffers from a few drawbacks, including a decrease in performance if those unnecessary layers are not optimized away, and a graph which is overly large.

# Proposed Solution
For all operators which are no-ops in TensorRT, omit them entirely from the graph. Then, add a lowering pass which analyzes the graph operators, seeking any graph branches which could be a sequence of no-ops from the inputs to the outputs, or a sequence of no-ops linking _two_ outputs. These are cases which TRT will throw an error for. Insert a single `_to_copy` node with `force_layer=True` for all such scenarios to avoid TRT errors.



Discussion / hints:

Fix patch:
diff --git a/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py b/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py
index fb5db527fb..8d08a0e3cd 100644
--- a/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py
+++ b/py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py
@@ -911,7 +911,7 @@ def aten_ops_clone_copy_dtype(
         name,
         args[0],
         kwargs.get("dtype", args[0].dtype),
-        force_layer=False,
+        force_layer=True,
     )
 
 
@@ -1043,7 +1043,7 @@ def aten_ops_sum(
             name,
             sum_,
             kwargs["output_dtype"],
-            force_layer=False,
+            force_layer=True,
         )
     else:
         return sum_


Test patch:
diff --git a/tests/py/dynamo/conversion/test_casts.py b/tests/py/dynamo/conversion/test_casts.py
index f17eb7b1d4..c067a0b9ad 100644
--- a/tests/py/dynamo/conversion/test_casts.py
+++ b/tests/py/dynamo/conversion/test_casts.py
@@ -75,6 +75,21 @@ def forward(self, x):
                 inputs,
             )
 
+    def test_to_copy_multiple_returns(self):
+        class ToCopyReturns(nn.Module):
+            def forward(self, x):
+                x_1 = x + 1
+                y = torch.ops.aten._to_copy.default(x_1, dtype=torch.float)
+                z = torch.ops.aten._to_copy.default(x_1, dtype=torch.float)
+                return y, z
+
+        inputs = [torch.rand((1, 3, 10))]
+        self.run_test(
+            ToCopyReturns(),
+            inputs,
+            precision=torch.float,
+        )
+
 
 if __name__ == "__main__":
     run_tests()