Repository: Project-MONAI/MONAI

Issue:
Got an error when running bundle inference with TensorRT torchscript
**Describe the bug**
When I run the bundle inference with TensorRT based torchscript models, the below error appeared.

```
  File "/usr/local/lib/python3.8/dist-packages/monai/handlers/stats_handler.py", line 179, in exception_raised
    raise e
  File "/usr/local/lib/python3.8/dist-packages/ignite/engine/engine.py", line 1068, in _run_once_on_dataset_as_gen
    self.state.output = self._process_function(self, self.state.batch)
  File "/usr/local/lib/python3.8/dist-packages/monai/engines/evaluator.py", line 300, in _iteration
    with engine.mode(engine.network):
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/dist-packages/monai/networks/utils.py", line 389, in eval_mode
    training = [n for n in nets if n.training]
  File "/usr/local/lib/python3.8/dist-packages/monai/networks/utils.py", line 389, in <listcomp>
    training = [n for n in nets if n.training]
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_script.py", line 785, in __getattr__
    return super(RecursiveScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_script.py", line 502, in __getattr__
    return super(ScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1587, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'RecursiveScriptModule' object has no attribute 'training'
```

This should relate to the [ with engine.mode(engine.network):](https://github.com/Project-MONAI/MONAI/blob/354c9c2be3139b1654811784340d094f78a5c7d6/monai/engines/evaluator.py#L300) code here, in which the `engine.network.training` method of the torchscript model is directly used without check. I think we should add some checks in the `evaluator.py` file to avoid this issue.

**To Reproduce**
Steps to reproduce the behavior:
1. Start a MONAI docker.
2. Convert a MONAI bundle in model zoo to a TensorRT based torchscript.
3. Add `import torch_tensorrt` to the import part of `inference.json`.
4. Modify the `network_def` to `torch.jit.load(<model_name>)`.
5. Disable the `CheckpointLoader` handler.
6. Run the command `python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf`

**Expected behavior**
Run the bundle and get the inference results.


Discussion:
Hi @yiheng-wang-nv ,

I think this issue is related to the documentation: https://github.com/Project-MONAI/model-zoo/blob/dev/CONTRIBUTING.md#verify-torchscript. Could you please help double confirm it?

Thanks in advance.


Resolving patch:
diff --git a/monai/engines/evaluator.py b/monai/engines/evaluator.py
--- a/monai/engines/evaluator.py
+++ b/monai/engines/evaluator.py
@@ -295,7 +295,6 @@ def _iteration(self, engine: SupervisedEvaluator, batchdata: dict[str, torch.Ten
 
         # put iteration outputs into engine.state
         engine.state.output = {Keys.IMAGE: inputs, Keys.LABEL: targets}
-
         # execute forward computation
         with engine.mode(engine.network):
             if engine.amp:
diff --git a/monai/networks/utils.py b/monai/networks/utils.py
--- a/monai/networks/utils.py
+++ b/monai/networks/utils.py
@@ -375,17 +375,19 @@ def eval_mode(*nets: nn.Module):
             print(p(t).sum().backward())  # will correctly raise an exception as gradients are calculated
     """
 
-    # Get original state of network(s)
-    training = [n for n in nets if n.training]
+    # Get original state of network(s).
+    # Check the training attribute in case it's TensorRT based models which don't have this attribute.
+    training = [n for n in nets if hasattr(n, "training") and n.training]
 
     try:
         # set to eval mode
         with torch.no_grad():
-            yield [n.eval() for n in nets]
+            yield [n.eval() if hasattr(n, "eval") else n for n in nets]
     finally:
         # Return required networks to training
         for n in training:
-            n.train()
+            if hasattr(n, "train"):
+                n.train()
 
 
 @contextmanager
@@ -410,16 +412,18 @@ def train_mode(*nets: nn.Module):
     """
 
     # Get original state of network(s)
-    eval_list = [n for n in nets if not n.training]
+    # Check the training attribute in case it's TensorRT based models which don't have this attribute.
+    eval_list = [n for n in nets if hasattr(n, "training") and (not n.training)]
 
     try:
         # set to train mode
         with torch.set_grad_enabled(True):
-            yield [n.train() for n in nets]
+            yield [n.train() if hasattr(n, "train") else n for n in nets]
     finally:
         # Return required networks to eval_list
         for n in eval_list:
-            n.eval()
+            if hasattr(n, "eval"):
+                n.eval()
 
 
 def get_state_dict(obj: torch.nn.Module | Mapping):


Test patch: