Repository: tenstorrent/tt-forge-fe
Pull request #1952: Use compiled outputs from verify functions for post processing steps in e2e passing Pytorch & ONNX model tests
Author: login: kamalrajkannan78
Created: 2025-05-07T10:28:01Z | Merged: 2025-05-08T07:41:37Z

### Ticket

- This PR fixes https://github.com/tenstorrent/tt-forge-fe/issues/1424

### What's changed

- This PR removes redundant inference and uses compiled output from verify function for post processing steps in e2e passing  Pytorch & ONNX model tests.

### Logs

- [may7_albert.log](https://github.com/user-attachments/files/20081505/may7_albert.log)
- [may7_dpr.log](https://github.com/user-attachments/files/20081554/may7_dpr.log)
- [may7_wideresnet.log](https://github.com/user-attachments/files/20081590/may7_wideresnet.log)
- [may7_mobv2.log](https://github.com/user-attachments/files/20081557/may7_mobv2.log)
- [may7_mobv3.log](https://github.com/user-attachments/files/20081559/may7_mobv3.log)
- [may7_perceiverio.log](https://github.com/user-attachments/files/20081561/may7_perceiverio.log)
- [may7_resnext.log](https://github.com/user-attachments/files/20081562/may7_resnext.log)
- [may7_segformer.log](https://github.com/user-attachments/files/20081564/may7_segformer.log)
- [may7_autoenc.log](https://github.com/user-attachments/files/20081518/may7_autoenc.log)
- [may7_bert.log](https://github.com/user-attachments/files/20081519/may7_bert.log)
- [may7_mobv1.log](https://github.com/user-attachments/files/20081555/may7_mobv1.log)
- [may7_vilt_qa.log](https://github.com/user-attachments/files/20081572/may7_vilt_qa.log)









Diff:
diff --git a/forge/test/models/onnx/vision/segformer/test_segformer.py b/forge/test/models/onnx/vision/segformer/test_segformer.py
index 04205c1bbb..40cdea8981 100644
--- a/forge/test/models/onnx/vision/segformer/test_segformer.py
+++ b/forge/test/models/onnx/vision/segformer/test_segformer.py
@@ -64,14 +64,19 @@ def test_segformer_image_classification_onnx(forge_property_recorder, variant, t
         onnx_model, inputs, forge_property_handler=forge_property_recorder, module_name=module_name
     )
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
         forge_property_handler=forge_property_recorder,
     )
 
+    # Post processing
+    logits = co_out[0]
+    predicted_label = logits.argmax(-1).item()
+    print("Predicted class: ", torch_model.config.id2label[predicted_label])
+
 
 variants_semseg = [
     "nvidia/segformer-b0-finetuned-ade-512-512",
diff --git a/forge/test/models/pytorch/multimodal/vilt/test_vilt.py b/forge/test/models/pytorch/multimodal/vilt/test_vilt.py
index 866717b8c3..9cc5e1eb98 100644
--- a/forge/test/models/pytorch/multimodal/vilt/test_vilt.py
+++ b/forge/test/models/pytorch/multimodal/vilt/test_vilt.py
@@ -74,14 +74,11 @@ def test_vilt_question_answering_hf_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    logits = output[0]
+    logits = co_out[0]
     idx = logits.argmax(-1).item()
     print("Predicted answer:", model.config.id2label[idx])
 
diff --git a/forge/test/models/pytorch/text/albert/test_albert.py b/forge/test/models/pytorch/text/albert/test_albert.py
index 0d3bbca8d2..f3f7720c3f 100644
--- a/forge/test/models/pytorch/text/albert/test_albert.py
+++ b/forge/test/models/pytorch/text/albert/test_albert.py
@@ -75,8 +75,8 @@ def test_albert_masked_lm_pytorch(forge_property_recorder, size, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
@@ -84,11 +84,8 @@ def test_albert_masked_lm_pytorch(forge_property_recorder, size, variant):
         forge_property_handler=forge_property_recorder,
     )
 
-    # Inference
-    output = compiled_model(*inputs)
-
     # post processing
-    logits = output[0]
+    logits = co_out[0]
     mask_token_index = (input_tokens.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
     predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
     print("The predicted token for the [MASK] is: ", tokenizer.decode(predicted_token_id))
@@ -165,8 +162,8 @@ def test_albert_token_classification_pytorch(forge_property_recorder, size, vari
     else:
         pcc = 0.95
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
@@ -174,9 +171,6 @@ def test_albert_token_classification_pytorch(forge_property_recorder, size, vari
         forge_property_handler=forge_property_recorder,
     )
 
-    # Inference
-    co_out = compiled_model(*inputs)
-
     # post processing
     predicted_token_class_ids = co_out[0].argmax(-1)
     predicted_token_class_ids = torch.masked_select(predicted_token_class_ids, (input_tokens["attention_mask"][0] == 1))
@@ -257,11 +251,8 @@ def test_albert_sequence_classification_pytorch(forge_property_recorder, variant
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    co_out = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # post processing
     predicted_class_id = co_out[0].argmax().item()
diff --git a/forge/test/models/pytorch/text/bert/test_bert.py b/forge/test/models/pytorch/text/bert/test_bert.py
index 16f28c831f..99ae2f1f96 100644
--- a/forge/test/models/pytorch/text/bert/test_bert.py
+++ b/forge/test/models/pytorch/text/bert/test_bert.py
@@ -55,19 +55,16 @@ def test_bert_masked_lm_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
         forge_property_handler=forge_property_recorder,
     )
 
-    # Inference
-    output = compiled_model(*inputs)
-
     # post processing
-    logits = output[0]
+    logits = co_out[0]
     mask_token_index = (input_tokens.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
     predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
     print("The predicted token for the [MASK] is: ", tokenizer.decode(predicted_token_id))
@@ -129,20 +126,17 @@ def test_bert_question_answering_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
         forge_property_handler=forge_property_recorder,
     )
 
-    # Inference
-    output = compiled_model(*inputs)
-
     # post processing
-    start_logits = output[0]
-    end_logits = output[1]
+    start_logits = co_out[0]
+    end_logits = co_out[1]
 
     answer_start_index = start_logits.argmax()
     answer_end_index = end_logits.argmax()
@@ -196,10 +190,10 @@ def test_bert_sequence_classification_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
-    co_out = compiled_model(*inputs)
+    # post processing
     predicted_value = co_out[0].argmax(-1).item()
 
     # Answer - "positive"
diff --git a/forge/test/models/pytorch/text/dpr/test_dpr.py b/forge/test/models/pytorch/text/dpr/test_dpr.py
index 137f6aed8a..a6996ebdf5 100644
--- a/forge/test/models/pytorch/text/dpr/test_dpr.py
+++ b/forge/test/models/pytorch/text/dpr/test_dpr.py
@@ -69,19 +69,16 @@ def test_dpr_context_encoder_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
         forge_property_handler=forge_property_recorder,
     )
 
-    # Inference
-    embeddings = compiled_model(*inputs)
-
     # Results
-    print("embeddings", embeddings)
+    print("embeddings", co_out)
 
 
 variants = ["facebook/dpr-question_encoder-single-nq-base", "facebook/dpr-question_encoder-multiset-base"]
diff --git a/forge/test/models/pytorch/text/perceiverio/test_perceiverio.py b/forge/test/models/pytorch/text/perceiverio/test_perceiverio.py
index 40aadf245c..f49c16bca0 100644
--- a/forge/test/models/pytorch/text/perceiverio/test_perceiverio.py
+++ b/forge/test/models/pytorch/text/perceiverio/test_perceiverio.py
@@ -48,13 +48,10 @@ def test_perceiverio_masked_lm_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # post processing
-    logits = output[0]
+    logits = co_out[0]
     masked_tokens_predictions = logits[0, 51:61].argmax(dim=-1)
     print("The predicted token for the [MASK] is: ", tokenizer.decode(masked_tokens_predictions))
diff --git a/forge/test/models/pytorch/vision/autoencoder/test_autoencoder.py b/forge/test/models/pytorch/vision/autoencoder/test_autoencoder.py
index 28d19e5bce..7594133d59 100644
--- a/forge/test/models/pytorch/vision/autoencoder/test_autoencoder.py
+++ b/forge/test/models/pytorch/vision/autoencoder/test_autoencoder.py
@@ -102,14 +102,11 @@ def test_linear_ae_pytorch(forge_property_recorder):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(sample_tensor)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    output_image = output[0].view(1, 28, 28).detach().numpy()
+    output_image = co_out[0].view(1, 28, 28).detach().numpy()
     save_path = "forge/test/models/pytorch/vision/autoencoder/results"
     os.makedirs(save_path, exist_ok=True)
     reconstructed_image_path = f"{save_path}/reconstructed_image.png"
diff --git a/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v1.py b/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v1.py
index 2b9cf86d10..ecaa0622ea 100644
--- a/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v1.py
+++ b/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v1.py
@@ -41,14 +41,11 @@ def test_mobilenetv1_basic(forge_property_recorder):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    #  Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    #  Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    post_processing(output)
+    post_processing(co_out)
 
 
 def generate_model_mobilenetv1_imgcls_hf_pytorch(variant):
diff --git a/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v2.py b/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v2.py
index 50bc842211..aa007c3c6a 100644
--- a/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v2.py
+++ b/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v2.py
@@ -53,14 +53,11 @@ def test_mobilenetv2_basic(forge_property_recorder):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    post_processing(output)
+    post_processing(co_out)
 
 
 def generate_model_mobilenetV2I96_imgcls_hf_pytorch(variant):
diff --git a/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v3.py b/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v3.py
index 5918149af2..732b53fa5b 100644
--- a/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v3.py
+++ b/forge/test/models/pytorch/vision/mobilenet/test_mobilenet_v3.py
@@ -51,14 +51,11 @@ def test_mobilenetv3_basic(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    post_processing(output)
+    post_processing(co_out)
 
 
 def generate_model_mobilenetV3_imgcls_timm_pytorch(variant):
diff --git a/forge/test/models/pytorch/vision/resnext/test_resnext.py b/forge/test/models/pytorch/vision/resnext/test_resnext.py
index 9ae3fde11b..760e999d57 100644
--- a/forge/test/models/pytorch/vision/resnext/test_resnext.py
+++ b/forge/test/models/pytorch/vision/resnext/test_resnext.py
@@ -42,14 +42,11 @@ def test_resnext_50_torchhub_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    post_processing(output)
+    post_processing(co_out)
 
 
 @pytest.mark.push
@@ -76,14 +73,11 @@ def test_resnext_101_torchhub_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    post_processing(output)
+    post_processing(co_out)
 
 
 @pytest.mark.nightly
diff --git a/forge/test/models/pytorch/vision/wideresnet/test_wideresnet.py b/forge/test/models/pytorch/vision/wideresnet/test_wideresnet.py
index ed69b28219..cbc7581a38 100644
--- a/forge/test/models/pytorch/vision/wideresnet/test_wideresnet.py
+++ b/forge/test/models/pytorch/vision/wideresnet/test_wideresnet.py
@@ -51,14 +51,11 @@ def test_wideresnet_pytorch(forge_property_recorder, variant):
         framework_model, sample_inputs=inputs, module_name=module_name, forge_property_handler=forge_property_recorder
     )
 
-    # Model Verification
-    verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
-
-    # Inference
-    output = compiled_model(*inputs)
+    # Model Verification and Inference
+    _, co_out = verify(inputs, framework_model, compiled_model, forge_property_handler=forge_property_recorder)
 
     # Post processing
-    post_processing(output)
+    post_processing(co_out)
 
 
 def generate_model_wideresnet_imgcls_timm(variant):


Reviews:
user: login: saiarthiraguram
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:07:57Z
user: login: vkovinicTT
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T13:48:33Z
user: login: kamalrajkannan78
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T18:55:39Z
user: login: vkovinicTT
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-08T07:29:32Z

Review comments:
diff_hunk: @@ -64,14 +64,19 @@ def test_segformer_image_classification_onnx(forge_property_recorder, variant, t
         onnx_model, inputs, forge_property_handler=forge_property_recorder, module_name=module_name
     )
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
         forge_property_handler=forge_property_recorder,
     )
 
+    # Post processing
+    logits = co_out[0]
+    predicted_label = logits.argmax(-1).item()
+    print("Predicted class: ", torch_model.config.id2label[predicted_label])
+
path: forge/test/models/onnx/vision/segformer/test_segformer.py
user: login: vkovinicTT
body: Why did we add post-processing step here?
created_at: 2025-05-07T13:48:26Z
author_association: MEMBER
diff_hunk: @@ -64,14 +64,19 @@ def test_segformer_image_classification_onnx(forge_property_recorder, variant, t
         onnx_model, inputs, forge_property_handler=forge_property_recorder, module_name=module_name
     )
 
-    # Model Verification
-    verify(
+    # Model Verification and Inference
+    _, co_out = verify(
         inputs,
         framework_model,
         compiled_model,
         forge_property_handler=forge_property_recorder,
     )
 
+    # Post processing
+    logits = co_out[0]
+    predicted_label = logits.argmax(-1).item()
+    print("Predicted class: ", torch_model.config.id2label[predicted_label])
+
path: forge/test/models/onnx/vision/segformer/test_segformer.py
user: login: kamalrajkannan78
body: This model passed e2e & it comes under Priority-1. so I added it
created_at: 2025-05-07T18:55:39Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov-commenter
created_at: 2025-05-07T10:48:06Z
body: ## [Codecov](https://app.codecov.io/gh/tenstorrent/tt-forge-fe/pull/1952?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 44.89%. Comparing base [(`5710001`)](https://app.codecov.io/gh/tenstorrent/tt-forge-fe/commit/5710001647557336f09957922f44b8d3fcaa151c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) to head [(`e880fb9`)](https://app.codecov.io/gh/tenstorrent/tt-forge-fe/commit/e880fb96488d56288975cd24b9147d04df55587a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).
> Report is 5 commits behind head on main.

:white_check_mark: All tests successful. No failed tests found.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #1952   +/-   ##
=======================================
  Coverage   44.89%   44.89%           
=======================================
  Files          56       56           
  Lines        8416     8416           
=======================================
  Hits         3778     3778           
  Misses       4638     4638           
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/tenstorrent/tt-forge-fe/pull/1952?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).

author_association: NONE
user: login: vkovinicTT
created_at: 2025-05-07T13:51:06Z
body: Other than this one minor comment this looks fine :))
author_association: MEMBER