Repository: tenstorrent/tt-torch
Pull request #564: Apr 4 Benchmark promotion + Benchmark Autopromotion Foundation
Author: login: jameszianxuTT
Created: 2025-04-04T21:13:11Z | Merged: 2025-04-05T02:29:20Z

### Ticket
None

### Problem description
The benchmark promotion process when new pytests are added or pytests are renamed is manual and relies on committing a benchmark_durations file to tree.

### What's changed
- Create an artifact-based benchmark matrix auto promotion + quarantining + load balancing workflow (currently not used)
- Manually promote this week's benchmarks due to serdes issues
- Update download_artifacts tool to accept a variable output directory, which allows it to be more easily programmatically called by other tools 

## Todo
- Deal with JSON / YML serdes issues and Github actions syntactical issues

### Checklist
- [x] A dummy benchmark job passed


Diff:
diff --git a/.github/benchmark_durations.csv b/.github/benchmark_durations.csv
deleted file mode 100644
index cac3479cb..000000000
--- a/.github/benchmark_durations.csv
+++ /dev/null
@@ -1,136 +0,0 @@
-full_test_name,achieved_compile_depth,model_name,test_execution_time
-test_beit_image_classification.py::tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-base-patch16-224-eval],TTNN_IR,microsoft/beit-base-patch16-224,25.692774
-test_albert_masked_lm.py::tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-base-v2-eval],EXECUTE,albert/albert-base-v2,53.538065
-test_MobileNetV2_onnx.py::tests/models/MobileNetV2/test_MobileNetV2_onnx.py::test_MobileNetV2[full-eval],FAILED_FE,MobileNetV2,1.662504
-test_qwen2_casual_lm.py::tests/models/Qwen/test_qwen2_casual_lm.py::test_qwen2_casual_lm[full-Qwen/Qwen2.5-1.5B-eval],EXECUTE,Qwen/Qwen2.5-1.5B,1982.969133
-test_RMBG.py::tests/models/RMBG/test_RMBG.py::test_RMBG[full-eval],FAILED_FE,RMBG,136.545079
-test_qwen2_token_classification.py::tests/models/Qwen/test_qwen2_token_classification.py::test_qwen2_token_classification[full-Qwen/Qwen2-7B-eval],TTNN_IR,Qwen/Qwen2-7B,73.865518
-test_beit_image_classification.py::tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-large-patch16-224-eval],TTNN_IR,microsoft/beit-large-patch16-224,48.104982
-test_autoencoder_linear.py::tests/models/autoencoder_linear/test_autoencoder_linear.py::test_autoencoder_linear[full-eval],PASSED,Autoencoder (linear),24.271243
-test_albert_masked_lm.py::tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-large-v2-eval],EXECUTE,albert/albert-large-v2,65.024333
-test_albert_masked_lm.py::tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xxlarge-v2-eval],EXECUTE,albert/albert-xxlarge-v2,89.96741
-test_albert_token_classification.py::tests/models/albert/test_albert_token_classification.py::test_albert_token_classification[full-albert/albert-base-v2-eval],EXECUTE,albert/albert-base-v2,51.301997
-test_albert_masked_lm.py::tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xlarge-v2-eval],EXECUTE,albert/albert-xlarge-v2,77.922481
-test_autoencoder_conv_v2.py::tests/models/autoencoder_conv/test_autoencoder_conv_v2.py::test_autoencoder_conv_v2[full-eval],STABLEHLO,Autoencoder (conv),7.866613
-test_albert_sequence_classification.py::tests/models/albert/test_albert_sequence_classification.py::test_albert_sequence_classification[full-textattack/albert-base-v2-imdb-eval],EXECUTE,textattack/albert-base-v2-imdb,58.167665
-test_albert_question_answering.py::tests/models/albert/test_albert_question_answering.py::test_albert_question_answering[full-twmkn9/albert-base-v2-squad2-eval],EXECUTE,twmkn9/albert-base-v2-squad2,53.489988
-test_MobileNetV2.py::tests/models/MobileNetV2/test_MobileNetV2.py::test_MobileNetV2[full-eval],EXECUTE,MobileNetV2,108.579378
-test_detr.py::tests/models/detr/test_detr.py::test_detr[full-eval],TTNN_IR,DETR,134.863192
-test_distilbert.py::tests/models/distilbert/test_distilbert.py::test_distilbert_multiloop[full-distilbert-base-uncased-eval-64],TTNN_IR,distilbert-base-uncased,93.720394
-test_falcon.py::tests/models/falcon/test_falcon.py::test_falcon[full-eval],STABLEHLO,Falcon,73.713198
-test_glpn_kitti.py::tests/models/glpn_kitti/test_glpn_kitti.py::test_glpn_kitti[full-eval],TTNN_IR,GLPN-KITTI,376.888662
-test_dpr.py::tests/models/dpr/test_dpr.py::test_dpr[full-eval],EXECUTE,DPR,55.131824
-test_clip.py::tests/models/clip/test_clip.py::test_clip[full-eval],STABLEHLO,CLIP,23.284806
-test_deit.py::tests/models/deit/test_deit.py::test_deit[full-facebook/deit-base-patch16-224-eval],TTNN_IR,facebook/deit-base-patch16-224,23.251
-test_gpt2.py::tests/models/gpt2/test_gpt2.py::test_gpt2[full-eval],STABLEHLO,GPT-2,13.244242
-test_hardnet.py::tests/models/hardnet/test_hardnet.py::test_hardnet[full-eval],EXECUTE,HardNet,171.471295
-test_deepseek_qwen.py::tests/models/deepseek/test_deepseek_qwen.py::test_deepseek_qwen[full-deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-eval],TTNN_IR,deepseek-ai/DeepSeek-R1-Distill-Qwen-32B,277.700736
-test_distilbert.py::tests/models/distilbert/test_distilbert.py::test_distilbert[full-distilbert-base-uncased-eval],EXECUTE,distilbert-base-uncased,46.623634
-test_codegen.py::tests/models/codegen/test_codegen.py::test_codegen[full-eval],TTNN_IR,codegen,7.199627
-test_bert.py::tests/models/bert/test_bert.py::test_bert[full-eval],EXECUTE,BERT,80.609636
-test_flan_t5.py::tests/models/flan_t5/test_flan_t5.py::test_flan_t5[full-eval],FAILED_FE,FLAN-T5,169.831463
-test_gpt_neo.py::tests/models/gpt_neo/test_gpt_neo.py::test_gpt_neo[full-eval],FAILED_FE,GPTNeo,5400
-test_bloom.py::tests/models/bloom/test_bloom.py::test_bloom[full-eval],EXECUTE,Bloom,82.250534
-test_mlpmixer.py::tests/models/mlpmixer/test_mlpmixer.py::test_mlpmixer[full-eval],PASSED,MLPMixer,89.786042
-test_perceiver_io.py::tests/models/perceiver_io/test_perceiver_io.py::test_perceiver_io[full-eval],EXECUTE,Perceiver IO,98.815078
-test_opt.py::tests/models/opt/test_opt.py::test_opt[full-eval],EXECUTE,OPT,2454.34629
-test_mnist.py::tests/models/mnist/test_mnist.py::test_mnist_train[full-eval],EXECUTE,Mnist,30.629221
-test_mgp_str_base.py::tests/models/mgp-str-base/test_mgp_str_base.py::test_mgp_str_base[full-eval],TTNN_IR,alibaba-damo/mgp-str-base,67.760057
-test_resnet50.py::tests/models/resnet50/test_resnet50.py::test_resnet[full-eval],EXECUTE,ResNet50,87.821847
-test_mobilenet_ssd.py::tests/models/mobilenet_ssd/test_mobilenet_ssd.py::test_mobilenet_ssd[full-eval],EXECUTE,MobileNetSSD,183.973719
-test_llama_7b.py::tests/models/llama/test_llama_7b.py::test_llama_7b[full-eval],TTNN_IR,Llama,67.183031
-test_llama_3b.py::tests/models/llama/test_llama_3b.py::test_llama_3b[full-meta-llama/Llama-3.2-3B-eval],EXECUTE,meta-llama/Llama-3.2-3B,96.264619
-test_openpose_v2.py::tests/models/openpose/test_openpose_v2.py::test_openpose_v2[full-eval],EXECUTE,OpenPose V2,74.105869
-test_resnet.py::tests/models/resnet/test_resnet.py::test_resnet[full-eval],EXECUTE,ResNet18,57.220953
-test_mamba.py::tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-370m-hf-eval],TTNN_IR,state-spaces/mamba-370m-hf,9343.171456
-test_mamba.py::tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-790m-hf-eval],TTNN_IR,state-spaces/mamba-790m-hf,9601.144558
-test_mamba.py::tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-2.8b-hf-eval],TTNN_IR,state-spaces/mamba-2.8b-hf,18108.696154
-test_mamba.py::tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-1.4b-hf-eval],TTNN_IR,state-spaces/mamba-1.4b-hf,9597.814413
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-tf_efficientnet_lite4.in1k],TTNN_IR,tf_efficientnet_lite4.in1k,36.42662
-test_t5.py::tests/models/t5/test_t5.py::test_t5[full-t5-small-eval],FAILED_FE,t5-small,113.46192
-test_t5.py::tests/models/t5/test_t5.py::test_t5[full-t5-base-eval],FAILED_FE,t5-base,236.264187
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-tf_efficientnet_lite0.in1k],TTNN_IR,tf_efficientnet_lite0.in1k,25.294828
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-ghostnetv2_100.in1k],STABLEHLO,ghostnetv2_100.in1k,23.356908
-test_stable_diffusion_unet.py::tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval],EXECUTE,Stable Diffusion UNET,2219.412391
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-tf_efficientnet_lite1.in1k],TTNN_IR,tf_efficientnet_lite1.in1k,26.936616
-test_t5.py::tests/models/t5/test_t5.py::test_t5[full-t5-large-eval],FAILED_FE,t5-large,574.261194
-test_roberta.py::tests/models/roberta/test_roberta.py::test_roberta[full-eval],EXECUTE,RoBERTa,74.392586
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-ghostnet_100.in1k],EXECUTE,ghostnet_100.in1k,172.400358
-test_squeeze_bert.py::tests/models/squeeze_bert/test_squeeze_bert.py::test_squeeze_bert[full-eval],TTNN_IR,SqueezeBERT,58.742006
-test_stable_diffusion.py::tests/models/stable_diffusion/test_stable_diffusion.py::test_stable_diffusion[full-eval],TTNN_IR,Stable Diffusion,0
-test_segformer.py::tests/models/segformer/test_segformer.py::test_segformer[full-eval],EXECUTE,SegFormer,201.265578
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-tf_efficientnet_lite2.in1k],TTNN_IR,tf_efficientnet_lite2.in1k,26.640506
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-tf_efficientnet_lite3.in1k],TTNN_IR,tf_efficientnet_lite3.in1k,30.162228
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-hrnet_w18.ms_aug_in1k],EXECUTE,hrnet_w18.ms_aug_in1k,175.340584
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-ese_vovnet19b_dw.ra_in1k],FAILED_FE,ese_vovnet19b_dw.ra_in1k,9.464894
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-inception_v4.tf_in1k],STABLEHLO,inception_v4.tf_in1k,35.639052
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-xception71.tf_in1k],EXECUTE,xception71.tf_in1k,164.46882
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-mobilenet_v3_large],EXECUTE,mobilenet_v3_large,143.55997
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-mixer_b16_224.goog_in21k],TTNN_IR,mixer_b16_224.goog_in21k,20.251623
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-mobilenetv1_100.ra4_e3600_r224_in1k],EXECUTE,mobilenetv1_100.ra4_e3600_r224_in1k,74.128904
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-googlenet],FAILED_FE,googlenet,10.824093
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-mobilenet_v3_small],EXECUTE,mobilenet_v3_small,131.170933
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnet34],EXECUTE,resnet34,62.244929
-test_timm_image_classification.py::tests/models/timm/test_timm_image_classification.py::test_timm_image_classification[full-eval-dla34.in1k],EXECUTE,dla34.in1k,88.40829
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-densenet121],STABLEHLO,densenet121,20.827117
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-densenet169],STABLEHLO,densenet169,29.327243
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnet18],EXECUTE,resnet18,57.171152
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-densenet201],STABLEHLO,densenet201,35.060152
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-densenet161],STABLEHLO,densenet161,32.065644
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-mobilenet_v2],EXECUTE,mobilenet_v2,104.678179
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnet50],EXECUTE,resnet50,87.982209
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg11],EXECUTE,vgg11,74.591877
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg19_bn],EXECUTE,vgg19_bn,103.179442
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnet152],EXECUTE,resnet152,109.963306
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vit_l_16],TTNN_IR,vit_l_16,41.019936
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg19],EXECUTE,vgg19,98.161253
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnext101_64x4d],EXECUTE,resnext101_64x4d,117.595401
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnext50_32x4d],EXECUTE,resnext50_32x4d,84.614171
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnet101],EXECUTE,resnet101,99.445675
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg13],EXECUTE,vgg13,84.099058
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg13_bn],EXECUTE,vgg13_bn,90.217115
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg11_bn],EXECUTE,vgg11_bn,80.541257
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg16_bn],EXECUTE,vgg16_bn,97.195806
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vit_b_32],TTNN_IR,vit_b_32,18.104361
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-resnext101_32x8d],EXECUTE,resnext101_32x8d,125.855668
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vgg16],EXECUTE,vgg16,90.482082
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vit_b_16],TTNN_IR,vit_b_16,21.102956
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_128gf],TTNN_IR,regnet_y_128gf,586.163387
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_x_1_6gf],EXECUTE,regnet_x_1_6gf,82.557953
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_x_400mf],EXECUTE,regnet_x_400mf,73.01385
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vit_l_32],TTNN_IR,vit_l_32,33.453093
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-wide_resnet101_2],EXECUTE,wide_resnet101_2,111.496607
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-wide_resnet50_2],EXECUTE,wide_resnet50_2,91.307342
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_x_800mf],EXECUTE,regnet_x_800mf,75.690341
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-vit_h_14],TTNN_IR,vit_h_14,217.447248
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_800mf],EXECUTE,regnet_y_800mf,114.176216
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_x_8gf],EXECUTE,regnet_x_8gf,99.117349
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_32gf],EXECUTE,regnet_y_32gf,197.532025
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_1_6gf],EXECUTE,regnet_y_1_6gf,125.406878
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_16gf],EXECUTE,regnet_y_16gf,170.048485
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_3_2gf],EXECUTE,regnet_y_3_2gf,129.306948
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_400mf],EXECUTE,regnet_y_400mf,107.304057
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_y_8gf],EXECUTE,regnet_y_8gf,132.690434
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_x_3_2gf],EXECUTE,regnet_x_3_2gf,88.091651
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-swin_v2_t],STABLEHLO,swin_v2_t,19.59003
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-swin_b],STABLEHLO,swin_b,33.52885
-test_unet_brain.py::tests/models/unet_brain/test_unet_brain.py::test_unet_brain[full-eval],STABLEHLO,Unet-brain,60.299789
-test_vilt.py::tests/models/vilt/test_vilt.py::test_vilt[full-eval],TTNN_IR,ViLT,23.150183
-test_torchvision_object_detection.py::tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn_v2-eval],STABLEHLO,retinanet_resnet50_fpn_v2,17.277918
-test_unet_carvana.py::tests/models/unet_carvana/test_unet_carvana.py::test_unet_carvana[full-eval],STABLEHLO,Unet-carvana,83.051722
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-swin_v2_s],STABLEHLO,swin_v2_s,38.783457
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-swin_v2_b],STABLEHLO,swin_v2_b,40.672199
-test_unet.py::tests/models/unet/test_unet.py::test_unet[full-eval],STABLEHLO,U-Net,68.117641
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_x_16gf],EXECUTE,regnet_x_16gf,107.48523
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-swin_t],STABLEHLO,swin_t,16.585437
-test_whisper.py::tests/models/whisper/test_whisper.py::test_whisper[full-eval],TTNN_IR,Whisper,94.290768
-test_torchvision_object_detection.py::tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssdlite320_mobilenet_v3_large-eval],EXECUTE,ssdlite320_mobilenet_v3_large,196.555126
-test_torchvision_object_detection.py::tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn-eval],STABLEHLO,retinanet_resnet50_fpn,16.921105
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-swin_s],STABLEHLO,swin_s,35.971901
-test_torchvision_image_classification.py::tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification[full-eval-regnet_x_32gf],EXECUTE,regnet_x_32gf,150.632132
-test_torchvision_object_detection.py::tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssd300_vgg16-eval],FAILED_FE,ssd300_vgg16,8.194743
-test_yolov3.py::tests/models/yolov3/test_yolov3.py::test_yolov3[full-eval],EXECUTE,YOLOv3,264.84962
-test_yolos.py::tests/models/yolos/test_yolos.py::test_yolos[full-eval],TTNN_IR,YOLOS,19.292479
-test_xglm.py::tests/models/xglm/test_xglm.py::test_xglm[full-eval],STABLEHLO,XGLM,28.05034
-test_yolov4.py::tests/models/yolov4/test_yolov4.py::test_yolov4[full-eval],PASSED,YOLOv4,199.567578
-test_yolov5.py::tests/models/yolov5/test_yolov5.py::test_yolov5[full-eval],EXECUTE,YOLOv5,122.601326
diff --git a/.github/workflows/depth-benchmarks.yml b/.github/workflows/depth-benchmarks.yml
index 487f75096..09b542055 100644
--- a/.github/workflows/depth-benchmarks.yml
+++ b/.github/workflows/depth-benchmarks.yml
@@ -24,12 +24,17 @@ jobs:
     secrets: inherit
     with:
       docker-image: ${{ needs.docker-build.outputs.docker-image }}
+  generate-matrix-dynamic:
+    needs: [build]
+    uses: ./.github/workflows/generate-benchmark-matrix-dynamic.yml
+    secrets: inherit
   run-benchmark:
-    needs: [docker-build, build]
+    needs: [docker-build, build, generate-matrix-dynamic]
     uses: ./.github/workflows/run-depth-benchmark-tests.yml
     secrets: inherit
     with:
       docker-image: ${{ needs.docker-build.outputs.docker-image }}
+      matrix-json: ${{ needs.generate-matrix-dynamic.outputs.matrix-json }} # Pass the dynamically generated matrix to the benchmark job
   generate-report:
     needs: [run-benchmark]
     uses: ./.github/workflows/generate-benchmark-report.yml
diff --git a/.github/workflows/generate-benchmark-matrix-dynamic.yml b/.github/workflows/generate-benchmark-matrix-dynamic.yml
new file mode 100644
index 000000000..a01e17e2a
--- /dev/null
+++ b/.github/workflows/generate-benchmark-matrix-dynamic.yml
@@ -0,0 +1,89 @@
+name: Generate Benchmark Test Matrix Dynamically
+
+on:
+  workflow_call:
+    outputs:
+        matrix-json:
+            description: "Generated test matrix JSON string"
+            value: ${{ jobs.tests.outputs.matrix-json }} # Expose job output as workflow output
+  workflow_run:
+    workflows: [Build]
+    types: [completed]
+
+jobs:
+  tests:
+    outputs:
+        matrix-json: ${{ steps.generate_matrix.outputs.matrix-json }}
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+
+    runs-on:
+      - wormhole_b0
+
+    container:
+      image: ghcr.io/tenstorrent/tt-torch/tt-torch-ci-ubuntu-22-04:latest
+      options: --user root --device /dev/tenstorrent/0
+      volumes:
+        - /dev/hugepages:/dev/hugepages
+        - /dev/hugepages-1G:/dev/hugepages-1G
+        - /etc/udev/rules.d:/etc/udev/rules.d
+        - /lib/modules:/lib/modules
+        - /opt/tt_metal_infra/provisioning/provisioning_env:/opt/tt_metal_infra/provisioning/provisioning_env
+
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        submodules: recursive
+        lfs: true
+
+    - name: Set reusable strings
+      id: strings
+      shell: bash
+      run: |
+        echo "work-dir=$(pwd)" >> "$GITHUB_OUTPUT"
+        echo "install-dir=$(pwd)/install" >> "$GITHUB_OUTPUT"
+        echo "dist-dir=$(pwd)/dist" >> "$GITHUB_OUTPUT"
+        echo "test-output-dir=$(pwd)/results/models/tests/" >> "$GITHUB_OUTPUT"
+
+    - name: Git safe dir
+      run: git config --global --add safe.directory ${{ steps.strings.outputs.work-dir }}
+
+    - name: Use build artifacts
+      uses: actions/download-artifact@v4
+      with:
+        name: install-artifacts
+        path: ${{ steps.strings.outputs.install-dir }}
+
+    - name: 'Untar install directory'
+      shell: bash
+      working-directory: ${{ steps.strings.outputs.install-dir }}
+      run: |
+        tar xvf artifact.tar
+        mkdir -p ${{ steps.strings.outputs.dist-dir }}
+        mv wheels/* ${{ steps.strings.outputs.dist-dir }}
+
+    - name: install tt-torch
+      shell: bash
+      run: |
+        source env/activate
+        pip install ${{ steps.strings.outputs.dist-dir }}/*.whl
+
+
+    - name: Generate test matrix
+      id: generate_matrix
+      shell: bash
+      env:
+        GH_TOKEN: ${{ github.token }} # needed to download artifacts
+      working-directory: ${{ steps.strings.outputs.work-dir }}
+      run: |
+        source env/activate
+
+        # Using the actual built docker image seems to cause git corruption
+        apt install -y libgl1 libglx-mesa0
+
+        python tt_torch/tools/benchmark_promotion.py --gen-matrix
+
+        # hardcoded filename to avoid cluttering cliargs for benchmark promotion
+        # this does not work for multilined output..
+        echo "matrix-json=$(cat benchmark_test_matrix.json)" >> "$GITHUB_OUTPUT"
diff --git a/.github/workflows/run-depth-benchmark-tests.yml b/.github/workflows/run-depth-benchmark-tests.yml
index 716f73bde..f815abbec 100644
--- a/.github/workflows/run-depth-benchmark-tests.yml
+++ b/.github/workflows/run-depth-benchmark-tests.yml
@@ -8,9 +8,14 @@ on:
         description: 'Docker image to use for build'
         required: true
         type: string
+      matrix-json:
+        description: 'Matrix of tests to run as a JSON string'
+        required: true
+        type: string
   workflow_run:
     workflows: [Build] # backref to run-build as dependency
     types: [completed]
+
 jobs:
   tests:
     timeout-minutes: 500
@@ -18,25 +23,27 @@ jobs:
       fail-fast: false
       matrix:
         build: [
-          {runs-on: 'wormhole_b0', name: 'benchmark_1', tests: ['tests/models/MobileNetV2/test_MobileNetV2.py::test_MobileNetV2[full-eval]', 'tests/models/MobileNetV2/test_MobileNetV2_onnx.py::test_MobileNetV2[full-eval]', 'tests/models/Qwen/test_qwen2_casual_lm.py::test_qwen2_casual_lm[full-Qwen/Qwen2.5-1.5B-eval]', 'tests/models/Qwen/test_qwen2_token_classification.py::test_qwen2_token_classification[full-Qwen/Qwen2-7B-eval]', 'tests/models/RMBG/test_RMBG.py::test_RMBG[full-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-base-v2-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-large-v2-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xlarge-v2-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xxlarge-v2-eval]', 'tests/models/albert/test_albert_question_answering.py::test_albert_question_answering[full-twmkn9/albert-base-v2-squad2-eval]', 'tests/models/albert/test_albert_sequence_classification.py::test_albert_sequence_classification[full-textattack/albert-base-v2-imdb-eval]', 'tests/models/albert/test_albert_token_classification.py::test_albert_token_classification[full-albert/albert-base-v2-eval]', 'tests/models/autoencoder_conv/test_autoencoder_conv.py::test_autoencoder_conv[full-eval]', 'tests/models/autoencoder_conv/test_autoencoder_conv_v2.py::test_autoencoder_conv_v2[full-eval]', 'tests/models/autoencoder_linear/test_autoencoder_linear.py::test_autoencoder_linear[full-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-base-patch16-224-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-large-patch16-224-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_2', tests: ['tests/models/bert/test_bert.py::test_bert[full-eval]', 'tests/models/bloom/test_bloom.py::test_bloom[full-eval]', 'tests/models/clip/test_clip.py::test_clip[full-eval]', 'tests/models/codegen/test_codegen.py::test_codegen[full-eval]', 'tests/models/deepseek/test_deepseek_qwen.py::test_deepseek_qwen[full-deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-eval]', 'tests/models/deit/test_deit.py::test_deit[full-facebook/deit-base-patch16-224-eval]', 'tests/models/detr/test_detr.py::test_detr[full-eval]', 'tests/models/distilbert/test_distilbert.py::test_distilbert[full-distilbert-base-uncased-eval]', 'tests/models/distilbert/test_distilbert.py::test_distilbert_multiloop[full-distilbert-base-uncased-eval-64]', 'tests/models/dpr/test_dpr.py::test_dpr[full-eval]', 'tests/models/falcon/test_falcon.py::test_falcon[full-eval]', 'tests/models/flan_t5/test_flan_t5.py::test_flan_t5[full-eval]', 'tests/models/glpn_kitti/test_glpn_kitti.py::test_glpn_kitti[full-eval]', 'tests/models/gpt2/test_gpt2.py::test_gpt2[full-eval]', 'tests/models/gpt_neo/test_gpt_neo.py::test_gpt_neo[full-eval]', 'tests/models/hand_landmark/test_hand_landmark.py::test_hand_landmark[full-eval]', 'tests/models/hardnet/test_hardnet.py::test_hardnet[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3a', tests: ['tests/models/llama/test_llama_3b.py::test_llama_3b[full-meta-llama/Llama-3.2-3B-eval]', 'tests/models/llama/test_llama_7b.py::test_llama_7b[full-eval]', 'tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-370m-hf-eval]', 'tests/models/mgp-str-base/test_mgp_str_base.py::test_mgp_str_base[full-eval]', 'tests/models/mlpmixer/test_mlpmixer.py::test_mlpmixer[full-eval]', 'tests/models/mnist/test_mnist.py::test_mnist_train[full-eval]', 'tests/models/mobilenet_ssd/test_mobilenet_ssd.py::test_mobilenet_ssd[full-eval]','tests/models/openpose/test_openpose.py::test_openpose[full-eval]', 'tests/models/openpose/test_openpose_v2.py::test_openpose_v2[full-eval]', 'tests/models/opt/test_opt.py::test_opt[full-eval]', 'tests/models/perceiver_io/test_perceiver_io.py::test_perceiver_io[full-eval]', 'tests/models/resnet/test_resnet.py::test_resnet[full-eval]', 'tests/models/resnet50/test_resnet50.py::test_resnet[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3b', tests: [ 'tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-790m-hf-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3c', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-2.8b-hf-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3d', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-1.4b-hf-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_4', tests: ['tests/models/roberta/test_roberta.py::test_roberta[full-eval]', 'tests/models/segformer/test_segformer.py::test_segformer[full-eval]', 'tests/models/segment_anything/test_segment_anything.py::test_segment_anything[full-eval]', 'tests/models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[full-eval]', 'tests/models/squeeze_bert/test_squeeze_bert.py::test_squeeze_bert[full-eval]', 'tests/models/stable_diffusion/test_stable_diffusion.py::test_stable_diffusion[full-eval]', 'tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-small-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-base-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-large-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite0.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite1.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite2.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite3.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite4.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnet_100.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnetv2_100.in1k]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_5', tests: ['tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-inception_v4.tf_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mixer_b16_224.goog_in21k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mobilenetv1_100.ra4_e3600_r224_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_red[full-eval-ese_vovnet19b_dw.ra_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-xception71.tf_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-dla34.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-hrnet_w18.ms_aug_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-googlenet]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet121]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet161]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet169]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet201]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-mobilenet_v2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_small]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_large]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet18]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet34]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_6', tests: ['tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet50]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet101]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet152]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext50_32x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_32x8d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_64x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_32]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_16]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_7', tests: ['tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_32]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_h_14]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet50_2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet101_2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_400mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_800mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_1_6gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_8gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_32gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_128gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_400mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_800mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_1_6gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_8gf]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_8', tests: ['tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_32gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_t]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_s]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_b]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_t]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_s]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_b]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssd300_vgg16-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssdlite320_mobilenet_v3_large-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn_v2-eval]', 'tests/models/unet/test_unet.py::test_unet[full-eval]', 'tests/models/unet_brain/test_unet_brain.py::test_unet_brain[full-eval]', 'tests/models/unet_carvana/test_unet_carvana.py::test_unet_carvana[full-eval]', 'tests/models/vilt/test_vilt.py::test_vilt[full-eval]', 'tests/models/whisper/test_whisper.py::test_whisper[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_9', tests: ['tests/models/xglm/test_xglm.py::test_xglm[full-eval]', 'tests/models/yolos/test_yolos.py::test_yolos[full-eval]', 'tests/models/yolov3/test_yolov3.py::test_yolov3[full-eval]', 'tests/models/yolov4/test_yolov4.py::test_yolov4[full-eval]', 'tests/models/yolov5/test_yolov5.py::test_yolov5[full-eval]']},
-          # quarantined new tests
-          {runs-on: 'wormhole_b0', name: 'benchmark_q1', tests: ['tests/models/mistral/test_mistral_7b.py::test_mistral_7b[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q2', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-2-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q3', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1.5-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q4', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q5', tests: ['tests/models/deepseek/test_deepseek.py::test_deepseek[full-deepseek-ai/DeepSeek-V3-eval]']},
-          ]
+          {runs-on: 'wormhole_b0', name: 'bmk_1_qtn_test_mamba[full-state-spaces/mamba-2.8b-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-2.8b-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_2_qtn_test_mamba[full-state-spaces/mamba-370m-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-370m-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_3_qtn_test_mamba[full-state-spaces/mamba-1.4b-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-1.4b-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_4_qtn_test_mamba[full-state-spaces/mamba-790m-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-790m-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_5', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_32gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_800mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_8gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet101]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-dla34.in1k]', 'tests/models/unet_carvana/test_unet_carvana.py::test_unet_carvana[full-eval]', 'tests/models/bloom/test_bloom.py::test_bloom[full-eval]', 'tests/models/roberta/test_roberta.py::test_roberta[full-eval]', 'tests/models/unet_brain/test_unet_brain.py::test_unet_brain[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet18]', 'tests/models/albert/test_albert_question_answering.py::test_albert_question_answering[full-twmkn9/albert-base-v2-squad2-eval]', 'tests/models/clip/test_clip.py::test_clip[full-eval]', 'tests/models/xglm/test_xglm.py::test_xglm[full-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-base-patch16-224-eval]', 'tests/models/yolos/test_yolos.py::test_yolos[full-eval]', 'tests/models/gpt2/test_gpt2.py::test_gpt2[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_6', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1.5-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-base-eval]', 'tests/models/mobilenet_ssd/test_mobilenet_ssd.py::test_mobilenet_ssd[full-eval]', 'tests/models/flan_t5/test_flan_t5.py::test_flan_t5[full-eval]', 'tests/models/RMBG/test_RMBG.py::test_RMBG[full-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite0.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet101_2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19]', 'tests/models/llama/test_llama_3b.py::test_llama_3b[full-meta-llama/Llama-3.2-3B-eval]', 'tests/models/resnet50/test_resnet50.py::test_resnet[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext50_32x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_800mf]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xlarge-v2-eval]', 'tests/models/falcon/test_falcon.py::test_falcon[full-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-base-v2-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_s]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_s]', 'tests/models/vilt/test_vilt.py::test_vilt[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_t]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssd300_vgg16-eval]', 'tests/models/autoencoder_conv/test_autoencoder_conv_v2.py::test_autoencoder_conv_v2[full-eval]', 'tests/models/stable_diffusion/test_stable_diffusion.py::test_stable_diffusion[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_7', tests: ['tests/models/gpt_neo/test_gpt_neo.py::test_gpt_neo[full-eval]', 'tests/models/deepseek/test_deepseek_qwen.py::test_deepseek_qwen[full-deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_32gf]', 'tests/models/hardnet/test_hardnet.py::test_hardnet[full-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite2.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite1.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_64x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_400mf]', 'tests/models/perceiver_io/test_perceiver_io.py::test_perceiver_io[full-eval]', 'tests/models/whisper/test_whisper.py::test_whisper[full-eval]', 'tests/models/distilbert/test_distilbert.py::test_distilbert_multiloop[full-distilbert-base-uncased-eval-64]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11]', 'tests/models/openpose/test_openpose_v2.py::test_openpose_v2[full-eval]', 'tests/models/Qwen/test_qwen2_token_classification.py::test_qwen2_token_classification[full-Qwen/Qwen2-7B-eval]', 'tests/models/dpr/test_dpr.py::test_dpr[full-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-large-patch16-224-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-inception_v4.tf_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet169]', 'tests/models/deit/test_deit.py::test_deit[full-facebook/deit-base-patch16-224-eval]', 'tests/models/autoencoder_linear/test_autoencoder_linear.py::test_autoencoder_linear[full-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn_v2-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_8', tests: ['tests/models/opt/test_opt.py::test_opt[full-eval]', 'tests/models/glpn_kitti/test_glpn_kitti.py::test_glpn_kitti[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_h_14]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite4.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnet_100.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_large]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_small]', 'tests/models/t5/test_t5.py::test_t5[full-t5-small-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13_bn]', 'tests/models/mlpmixer/test_mlpmixer.py::test_mlpmixer[full-eval]', 'tests/models/llama/test_llama_7b.py::test_llama_7b[full-eval]', 'tests/models/mgp-str-base/test_mgp_str_base.py::test_mgp_str_base[full-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-large-v2-eval]', 'tests/models/squeeze_bert/test_squeeze_bert.py::test_squeeze_bert[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_b]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet201]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet161]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnetv2_100.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_16]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_red[full-eval-ese_vovnet19b_dw.ra_in1k]', 'tests/models/codegen/test_codegen.py::test_codegen[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_9', tests: ['tests/models/Qwen/test_qwen2_casual_lm.py::test_qwen2_casual_lm[full-Qwen/Qwen2.5-1.5B-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-large-eval]', 'tests/models/yolov3/test_yolov3.py::test_yolov3[full-eval]', 'tests/models/yolov4/test_yolov4.py::test_yolov4[full-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssdlite320_mobilenet_v3_large-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite3.in1k]', 'tests/models/detr/test_detr.py::test_detr[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_1_6gf]', 'tests/models/MobileNetV2/test_MobileNetV2.py::test_MobileNetV2[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet50_2]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xxlarge-v2-eval]', 'tests/models/bert/test_bert.py::test_bert[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11_bn]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mobilenetv1_100.ra4_e3600_r224_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet34]', 'tests/models/distilbert/test_distilbert.py::test_distilbert[full-distilbert-base-uncased-eval]', 'tests/models/albert/test_albert_sequence_classification.py::test_albert_sequence_classification[full-textattack/albert-base-v2-imdb-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_32]', 'tests/models/mnist/test_mnist.py::test_mnist_train[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet121]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_32]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_t]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_10', tests: ['tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_128gf]', 'tests/models/deepseek/test_deepseek.py::test_deepseek[full-deepseek-ai/DeepSeek-V3-eval]', 'tests/models/segformer/test_segformer.py::test_segformer[full-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-hrnet_w18.ms_aug_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-xception71.tf_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_8gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_32x8d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet152]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-mobilenet_v2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet50]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_1_6gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_400mf]', 'tests/models/unet/test_unet.py::test_unet[full-eval]', 'tests/models/resnet/test_resnet.py::test_resnet[full-eval]', 'tests/models/albert/test_albert_token_classification.py::test_albert_token_classification[full-albert/albert-base-v2-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_b]', 'tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-2-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mixer_b16_224.goog_in21k]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-googlenet]', 'tests/models/MobileNetV2/test_MobileNetV2_onnx.py::test_MobileNetV2[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_11_qtn_test_autoencoder_conv[full-eval]', tests: ['tests/models/autoencoder_conv/test_autoencoder_conv.py::test_autoencoder_conv[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_12_qtn_test_hand_landmark[full-eval]', tests: ['tests/models/hand_landmark/test_hand_landmark.py::test_hand_landmark[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_13_qtn_test_mistral_7b[full-eval]', tests: ['tests/models/mistral/test_mistral_7b.py::test_mistral_7b[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_14_qtn_test_musicgen_small[full-eval]', tests: ['tests/models/musicgen_small/test_musicgen_small.py::test_musicgen_small[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_15_qtn_test_openpose[full-eval]', tests: ['tests/models/openpose/test_openpose.py::test_openpose[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_16_qtn_test_segment_anything[full-eval]', tests: ['tests/models/segment_anything/test_segment_anything.py::test_segment_anything[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_17_qtn_test_speecht5_tts[full-eval]', tests: ['tests/models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_18_qtn_test_vit[full-eval]', tests: ['tests/models/vit/test_vit.py::test_vit[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_19_qtn_test_yolov5[full-eval]', tests: ['tests/models/yolov5/test_yolov5.py::test_yolov5[full-eval]']},
+        ]
+
     runs-on:
       - ${{ matrix.build.runs-on }}
 
diff --git a/results/download_artifacts.py b/results/download_artifacts.py
index d6e26fdcd..48189e613 100755
--- a/results/download_artifacts.py
+++ b/results/download_artifacts.py
@@ -32,6 +32,11 @@ def get_token(cli_token):
     if env_token:
         return env_token
 
+    # Priority 3.5: Environment variable (as commonly used in our CI)
+    env_token = os.environ.get("GH_TOKEN")
+    if env_token:
+        return env_token
+
     # Priority 4: Fallback to GitHub CLI (gh)
     try:
         token = subprocess.check_output(["gh", "auth", "token"], text=True).strip()
@@ -288,6 +293,11 @@ def main():
         default=4,
         help="Number of concurrent download threads (default: 4)",
     )
+    parser.add_argument(
+        "--output-folder",
+        "-o",
+        help="Output directory for artifacts",
+    )
 
     args = parser.parse_args()
 
@@ -309,6 +319,8 @@ def main():
     # Retrieve run details (either specific run-id or the latest run)
     run_id, date_str, folder_name = get_run_details(args, headers, workflow_name)
 
+    folder_name = args.output_folder if args.output_folder else folder_name
+
     if args.list:
         print(f"Listing artifacts for run {run_id}.")
     else:
diff --git a/tt_torch/tools/benchmark_promotion.py b/tt_torch/tools/benchmark_promotion.py
index a7818dacc..22ee4ede8 100644
--- a/tt_torch/tools/benchmark_promotion.py
+++ b/tt_torch/tools/benchmark_promotion.py
@@ -6,6 +6,11 @@
 from tt_torch.tools.generate_benchmark_report import parse_tests_from_matrix
 from collections import Counter
 import csv
+import argparse
+import pandas as pd
+import os
+import pprint
+import json
 
 
 def enumerate_all_tests(filter_full_eval=True):
@@ -82,12 +87,13 @@ def discover_tests():
         find_duplicates(in_tree_tests) == []
     ), "There are duplicate test cases in the in-tree tests."
 
-    print("Identifying changed pytest test set.")
     removed_tests = set(benchmark_tests) - set(in_tree_tests)
     added_tests = set(in_tree_tests) - set(benchmark_tests)
 
     if removed_tests != set():
         print(f"Warning {len(removed_tests)} pytests have been removed from tree.")
+    else:
+        print("No pytests have been removed from the benchmark tests.")
 
     # Added tests should be quarantined into their own runners and run in isolation
     # so unexpected/unrecoverable breakage doesn't cause other benchmarks to fail
@@ -98,58 +104,213 @@ def discover_tests():
         for i, t in enumerate(added_tests):
             print(f"{i+1}\t{t}")
 
+    return added_tests
 
-def load_balance_tests_greedy(csv_file_path, n_partitions):
+
+def load_balance_tests_greedy(test_durations, n_partitions=10, print_summary=True):
     """
-    Load balances test names from a CSV file into N partitions based on test execution time.
+    Load balances test names into N partitions based on test execution time.
 
     Args:
-        csv_file_path (str): Path to the CSV file containing test data.
+        test_durations (dict): A dictionary where keys are test names and values are their execution times.
         n_partitions (int): Number of partitions to split the test names into.
 
     Returns:
         list of lists: A list containing N partitions, each with a subset of test names.
     """
-    # Read the CSV file and extract test names and their execution times
-    tests = []
-    with open(csv_file_path, mode="r") as csv_file:
-        reader = csv.DictReader(csv_file)
-        for row in reader:
-            test_name = row["full_test_name"]
-            test_time = float(row["test_execution_time"])
-            tests.append((test_name, test_time))
-
-    # Sort tests by execution time in descending order
-    tests.sort(key=lambda x: x[1], reverse=True)
-    print(f"Found {len(tests)} tests in {csv_file_path}")
+    # Separate tests with known durations and unknown durations (-1)
+    known_tests = [
+        (test_name, duration)
+        for test_name, duration in test_durations.items()
+        if duration != -1
+    ]
+    unknown_tests = [
+        test_name for test_name, duration in test_durations.items() if duration == -1
+    ]
+
+    # Sort known tests by execution time in descending order
+    known_tests.sort(key=lambda x: x[1], reverse=True)
 
     # Initialize partitions and their total times
     partitions = [[] for _ in range(n_partitions)]
     partition_times = [0] * n_partitions
 
-    # Distribute tests greedily to minimize the maximum partition time
-    for test_name, test_time in tests:
+    # Distribute known tests greedily to minimize the maximum partition time
+    for test_name, test_time in known_tests:
         # Find the partition with the smallest total time
         min_index = partition_times.index(min(partition_times))
-        partitions[min_index].append((test_name, test_time))
+        partitions[min_index].append(test_name)
         partition_times[min_index] += test_time
 
+    # Add unknown tests (-1 duration) to their own partitions
+    for test_name in unknown_tests:
+        partitions.append([test_name])
+
+    if print_summary:
+        print("\nPartition Summary:")
+        for i, partition in enumerate(partitions):
+            partition_duration = sum(test_durations.get(test, 0) for test in partition)
+            print(f"Partition {i + 1}:")
+            print(f"  Tests: {partition}")
+            print(f"  Estimated Duration: {partition_duration:.2f} seconds\n")
+
     return partitions
 
 
-if __name__ == "__main__":
-    # Example usage
-    csv_file_path = ".github/benchmark_durations.csv"
-    n_partitions = 4
+def parse_benchmark_results_xlsx(file_path):
+    """
+    Parse the benchmark results from an Excel (.xlsx) file using pandas.
 
-    partitions = load_balance_tests_greedy(csv_file_path, n_partitions)
-    for i, partition in enumerate(partitions):
-        print(
-            f"Partition {i + 1}: {len(partition)} tests, Total Time: {sum(float(row[1])/3600 for row in partition):.2f}h"
+    Args:
+        file_path (str): Path to the Excel file.
+
+    Returns:
+        dict: A dictionary where keys are test case names and values are execution times.
+    """
+    try:
+        # Read the Excel file into a pandas DataFrame
+        df = pd.read_excel(file_path)
+
+        # Ensure required columns exist
+        if "Testcase Name" not in df.columns or "Execution Time (s)" not in df.columns:
+            raise ValueError(
+                "The Excel file must contain 'Testcase Name' and 'Execution Time (s)' columns."
+            )
+
+        # Convert the DataFrame to a dictionary
+        results = df.set_index("Testcase Name")["Execution Time (s)"].to_dict()
+
+        return results
+
+    except Exception as e:
+        print(f"Error parsing benchmark results: {e}")
+        return {}
+
+
+def generate_formatted_test_matrix_from_partitions(
+    partitions, base_name="bmk", runs_on="wormhole_b0"
+):
+    """
+    Generate a formatted test matrix for GitHub Actions based on test partitions.
+
+    Args:
+        partitions (list of lists): A list of partitions, where each partition is a list of test names.
+        base_name (str): Base name for the benchmark jobs (default: "benchmark").
+        runs_on (str): The runner type for the jobs (default: "wormhole_b0").
+
+    Returns:
+        str: A formatted test matrix as a string.
+    """
+    matrix = []
+    for i, partition in enumerate(partitions, start=1):
+        job_name = f"{base_name}_{i}"
+        # Append the test name to the job name for quarantined tests
+        if len(partition) == 1:
+            job_name += "_qtn_" + partition[0].split("::")[-1]
+        matrix.append(
+            {
+                "runs-on": runs_on,
+                "name": job_name,
+                "tests": partition,
+            }
         )
-        print(
-            partition[:5]
-        )  # Print the first 5 test names in each partition for brevity
 
-    formatted_partitions = [[pair[0] for pair in p] for p in partitions]
-    print(formatted_partitions)
+    # Format the matrix as a string
+    formatted_matrix = "build: ["
+    for job in matrix:
+        formatted_matrix += f"  {{runs-on: '{job['runs-on']}', name: '{job['name']}', tests: {job['tests']}}},"
+    formatted_matrix += "]"
+
+    return formatted_matrix
+
+
+def generate_dynamic_benchmark_test_matrix():
+
+    output_file = "benchmark_test_matrix.json"  # hardcoded into CI
+    report_dir = "benchmark_report"
+
+    # download previous report from run of depth-benchmarks workflow on main
+    subprocess.run(
+        [
+            "python",
+            "results/download_artifacts.py",
+            "--workflow",
+            "depth-benchmarks.yml",
+            "--filter",
+            "xlsx",
+            "-o",
+            report_dir,
+        ],
+        check=True,
+        shell=False,
+    )
+
+    # parse the xlsx file to get known test execution times
+    reports = os.listdir(report_dir)
+    assert (
+        len(reports) == 1
+    ), f"Expected exactly one xlsx file in {report_dir}, found: {reports}"
+    previous_run_results = parse_benchmark_results_xlsx(report_dir + "/" + reports[0])
+
+    in_tree_tests = enumerate_all_tests()
+    actual_test_durations_list = {}
+
+    quarantined_tests = []
+
+    for test in in_tree_tests:
+        modified_test_name = test.replace("_red", "").replace("_generality", "")
+
+        if test in previous_run_results.keys():
+            actual_test_durations_list[test] = previous_run_results[test]
+        if modified_test_name in previous_run_results.keys():
+            # Handle cases where the test name in the report may be slightly different
+            actual_test_durations_list[test] = previous_run_results[modified_test_name]
+        else:
+            # quarantined testss
+            actual_test_durations_list[test] = -1
+            quarantined_tests.append(test)
+
+    print(f"Quarantined test list (ct: {len(quarantined_tests)})")
+    # pprint.pprint(quarantined_tests)
+
+    print(f"Actual test list (ct: {len(actual_test_durations_list)})")
+    # pprint.pprint(actual_test_durations_list)
+
+    # Load balance the tests into a dynamic test matrix
+    test_splits = load_balance_tests_greedy(actual_test_durations_list)
+    fmt_matrix = generate_formatted_test_matrix_from_partitions(test_splits)
+    # print(fmt_matrix)
+
+    # Write the matrix to a file
+    with open(output_file, "w") as f:
+        f.write(fmt_matrix)
+    print(f"Test matrix written to {output_file}")
+
+
+if __name__ == "__main__":
+
+    parser = argparse.ArgumentParser(
+        description="Utilities for new pytest promotion to benchmarks"
+    )
+    parser.add_argument(
+        "--discover-tests",
+        action="store_true",
+        help="Discover tests in tree and compare with benchmark tests.",
+    )
+
+    parser.add_argument(
+        "--gen-matrix",
+        action="store_true",
+        help="Dynamically generate the test matrix with quarantined tests",
+    )
+
+    args = parser.parse_args()
+    if args.discover_tests:
+        """
+        Discover tests in the tree and compare with benchmark tests.
+        """
+        discover_tests()
+
+    if args.gen_matrix:
+        # new_tests = discover_tests()
+        generate_dynamic_benchmark_test_matrix()


Reviews:
user: login: jameszianxuTT
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T21:14:57Z
user: login: jameszianxuTT
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T21:50:07Z
user: login: jameszianxuTT
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T21:52:30Z
user: login: kmabeeTT
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-05T02:19:59Z

Review comments:
diff_hunk: @@ -0,0 +1,89 @@
+name: Generate Benchmark Test Matrix Dynamically
+
+on:
+  workflow_call:
+    outputs:
+        matrix-json:
+            description: "Generated test matrix JSON string"
+            value: ${{ jobs.tests.outputs.matrix-json }} # Expose job output as workflow output
+  workflow_run:
+    workflows: [Build]
+    types: [completed]
+
+jobs:
+  tests:
+    outputs:
+        matrix-json: ${{ steps.generate_matrix.outputs.matrix-json }}
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+
+    runs-on:
+      - wormhole_b0
+
+    container:
+      image: ghcr.io/tenstorrent/tt-torch/tt-torch-ci-ubuntu-22-04:latest
+      options: --user root --device /dev/tenstorrent/0
+      volumes:
+        - /dev/hugepages:/dev/hugepages
+        - /dev/hugepages-1G:/dev/hugepages-1G
+        - /etc/udev/rules.d:/etc/udev/rules.d
+        - /lib/modules:/lib/modules
+        - /opt/tt_metal_infra/provisioning/provisioning_env:/opt/tt_metal_infra/provisioning/provisioning_env
+
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        submodules: recursive
+        lfs: true
+
+    - name: Set reusable strings
+      id: strings
+      shell: bash
+      run: |
+        echo "work-dir=$(pwd)" >> "$GITHUB_OUTPUT"
+        echo "install-dir=$(pwd)/install" >> "$GITHUB_OUTPUT"
+        echo "dist-dir=$(pwd)/dist" >> "$GITHUB_OUTPUT"
+        echo "test-output-dir=$(pwd)/results/models/tests/" >> "$GITHUB_OUTPUT"
+
+    - name: Git safe dir
+      run: git config --global --add safe.directory ${{ steps.strings.outputs.work-dir }}
+
+    - name: Use build artifacts
+      uses: actions/download-artifact@v4
+      with:
+        name: install-artifacts
+        path: ${{ steps.strings.outputs.install-dir }}
+
+    - name: 'Untar install directory'
+      shell: bash
+      working-directory: ${{ steps.strings.outputs.install-dir }}
+      run: |
+        tar xvf artifact.tar
+        mkdir -p ${{ steps.strings.outputs.dist-dir }}
+        mv wheels/* ${{ steps.strings.outputs.dist-dir }}
+
+    - name: install tt-torch
+      shell: bash
+      run: |
+        source env/activate
+        pip install ${{ steps.strings.outputs.dist-dir }}/*.whl
+
+
+    - name: Generate test matrix
+      id: generate_matrix
+      shell: bash
+      env:
+        GH_TOKEN: ${{ github.token }} # needed to download artifacts
+      working-directory: ${{ steps.strings.outputs.work-dir }}
+      run: |
+        source env/activate
+
+        # Using the actual built docker image seems to cause git corruption
+        apt install -y libgl1 libglx-mesa0
path: .github/workflows/generate-benchmark-matrix-dynamic.yml
user: login: jameszianxuTT
body: I will amend this if I can figure out the git corruption due to incorrect string passing between jobs, but it is a workaround for now. Ideally we can use the CI docker but that requires some more testing
created_at: 2025-04-04T21:14:57Z
author_association: COLLABORATOR
diff_hunk: @@ -24,12 +24,17 @@ jobs:
     secrets: inherit
     with:
       docker-image: ${{ needs.docker-build.outputs.docker-image }}
+  generate-matrix-dynamic:
+    needs: [build]
+    uses: ./.github/workflows/generate-benchmark-matrix-dynamic.yml
+    secrets: inherit
   run-benchmark:
-    needs: [docker-build, build]
+    needs: [docker-build, build, generate-matrix-dynamic]
     uses: ./.github/workflows/run-depth-benchmark-tests.yml
     secrets: inherit
     with:
       docker-image: ${{ needs.docker-build.outputs.docker-image }}
+      matrix-json: ${{ needs.generate-matrix-dynamic.outputs.matrix-json }} # Pass the dynamically generated matrix to the benchmark job
path: .github/workflows/depth-benchmarks.yml
user: login: jameszianxuTT
body: Note - this output is not used in the run-benchmark job, so the preceding job currently has no effect. 
created_at: 2025-04-04T21:50:07Z
author_association: COLLABORATOR
diff_hunk: @@ -8,35 +8,42 @@ on:
         description: 'Docker image to use for build'
         required: true
         type: string
+      matrix-json:
+        description: 'Matrix of tests to run as a JSON string'
+        required: true
+        type: string
   workflow_run:
     workflows: [Build] # backref to run-build as dependency
     types: [completed]
+
 jobs:
   tests:
     timeout-minutes: 500
     strategy:
       fail-fast: false
       matrix:
         build: [
-          {runs-on: 'wormhole_b0', name: 'benchmark_1', tests: ['tests/models/MobileNetV2/test_MobileNetV2.py::test_MobileNetV2[full-eval]', 'tests/models/MobileNetV2/test_MobileNetV2_onnx.py::test_MobileNetV2[full-eval]', 'tests/models/Qwen/test_qwen2_casual_lm.py::test_qwen2_casual_lm[full-Qwen/Qwen2.5-1.5B-eval]', 'tests/models/Qwen/test_qwen2_token_classification.py::test_qwen2_token_classification[full-Qwen/Qwen2-7B-eval]', 'tests/models/RMBG/test_RMBG.py::test_RMBG[full-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-base-v2-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-large-v2-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xlarge-v2-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xxlarge-v2-eval]', 'tests/models/albert/test_albert_question_answering.py::test_albert_question_answering[full-twmkn9/albert-base-v2-squad2-eval]', 'tests/models/albert/test_albert_sequence_classification.py::test_albert_sequence_classification[full-textattack/albert-base-v2-imdb-eval]', 'tests/models/albert/test_albert_token_classification.py::test_albert_token_classification[full-albert/albert-base-v2-eval]', 'tests/models/autoencoder_conv/test_autoencoder_conv.py::test_autoencoder_conv[full-eval]', 'tests/models/autoencoder_conv/test_autoencoder_conv_v2.py::test_autoencoder_conv_v2[full-eval]', 'tests/models/autoencoder_linear/test_autoencoder_linear.py::test_autoencoder_linear[full-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-base-patch16-224-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-large-patch16-224-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_2', tests: ['tests/models/bert/test_bert.py::test_bert[full-eval]', 'tests/models/bloom/test_bloom.py::test_bloom[full-eval]', 'tests/models/clip/test_clip.py::test_clip[full-eval]', 'tests/models/codegen/test_codegen.py::test_codegen[full-eval]', 'tests/models/deepseek/test_deepseek_qwen.py::test_deepseek_qwen[full-deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-eval]', 'tests/models/deit/test_deit.py::test_deit[full-facebook/deit-base-patch16-224-eval]', 'tests/models/detr/test_detr.py::test_detr[full-eval]', 'tests/models/distilbert/test_distilbert.py::test_distilbert[full-distilbert-base-uncased-eval]', 'tests/models/distilbert/test_distilbert.py::test_distilbert_multiloop[full-distilbert-base-uncased-eval-64]', 'tests/models/dpr/test_dpr.py::test_dpr[full-eval]', 'tests/models/falcon/test_falcon.py::test_falcon[full-eval]', 'tests/models/flan_t5/test_flan_t5.py::test_flan_t5[full-eval]', 'tests/models/glpn_kitti/test_glpn_kitti.py::test_glpn_kitti[full-eval]', 'tests/models/gpt2/test_gpt2.py::test_gpt2[full-eval]', 'tests/models/gpt_neo/test_gpt_neo.py::test_gpt_neo[full-eval]', 'tests/models/hand_landmark/test_hand_landmark.py::test_hand_landmark[full-eval]', 'tests/models/hardnet/test_hardnet.py::test_hardnet[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3a', tests: ['tests/models/llama/test_llama_3b.py::test_llama_3b[full-meta-llama/Llama-3.2-3B-eval]', 'tests/models/llama/test_llama_7b.py::test_llama_7b[full-eval]', 'tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-370m-hf-eval]', 'tests/models/mgp-str-base/test_mgp_str_base.py::test_mgp_str_base[full-eval]', 'tests/models/mlpmixer/test_mlpmixer.py::test_mlpmixer[full-eval]', 'tests/models/mnist/test_mnist.py::test_mnist_train[full-eval]', 'tests/models/mobilenet_ssd/test_mobilenet_ssd.py::test_mobilenet_ssd[full-eval]','tests/models/openpose/test_openpose.py::test_openpose[full-eval]', 'tests/models/openpose/test_openpose_v2.py::test_openpose_v2[full-eval]', 'tests/models/opt/test_opt.py::test_opt[full-eval]', 'tests/models/perceiver_io/test_perceiver_io.py::test_perceiver_io[full-eval]', 'tests/models/resnet/test_resnet.py::test_resnet[full-eval]', 'tests/models/resnet50/test_resnet50.py::test_resnet[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3b', tests: [ 'tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-790m-hf-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3c', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-2.8b-hf-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_3d', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-1.4b-hf-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_4', tests: ['tests/models/roberta/test_roberta.py::test_roberta[full-eval]', 'tests/models/segformer/test_segformer.py::test_segformer[full-eval]', 'tests/models/segment_anything/test_segment_anything.py::test_segment_anything[full-eval]', 'tests/models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[full-eval]', 'tests/models/squeeze_bert/test_squeeze_bert.py::test_squeeze_bert[full-eval]', 'tests/models/stable_diffusion/test_stable_diffusion.py::test_stable_diffusion[full-eval]', 'tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-small-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-base-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-large-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite0.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite1.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite2.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite3.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite4.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnet_100.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnetv2_100.in1k]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_5', tests: ['tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-inception_v4.tf_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mixer_b16_224.goog_in21k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mobilenetv1_100.ra4_e3600_r224_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_red[full-eval-ese_vovnet19b_dw.ra_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-xception71.tf_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-dla34.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-hrnet_w18.ms_aug_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-googlenet]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet121]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet161]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet169]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet201]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-mobilenet_v2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_small]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_large]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet18]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet34]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_6', tests: ['tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet50]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet101]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet152]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext50_32x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_32x8d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_64x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_32]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_16]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_7', tests: ['tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_32]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_h_14]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet50_2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet101_2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_400mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_800mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_1_6gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_8gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_32gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_128gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_400mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_800mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_1_6gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_8gf]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_8', tests: ['tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_32gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_t]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_s]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_b]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_t]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_s]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_b]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssd300_vgg16-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssdlite320_mobilenet_v3_large-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn_v2-eval]', 'tests/models/unet/test_unet.py::test_unet[full-eval]', 'tests/models/unet_brain/test_unet_brain.py::test_unet_brain[full-eval]', 'tests/models/unet_carvana/test_unet_carvana.py::test_unet_carvana[full-eval]', 'tests/models/vilt/test_vilt.py::test_vilt[full-eval]', 'tests/models/whisper/test_whisper.py::test_whisper[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_9', tests: ['tests/models/xglm/test_xglm.py::test_xglm[full-eval]', 'tests/models/yolos/test_yolos.py::test_yolos[full-eval]', 'tests/models/yolov3/test_yolov3.py::test_yolov3[full-eval]', 'tests/models/yolov4/test_yolov4.py::test_yolov4[full-eval]', 'tests/models/yolov5/test_yolov5.py::test_yolov5[full-eval]']},
-          # quarantined new tests
-          {runs-on: 'wormhole_b0', name: 'benchmark_q1', tests: ['tests/models/mistral/test_mistral_7b.py::test_mistral_7b[full-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q2', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-2-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q3', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1.5-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q4', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1-eval]']},
-          {runs-on: 'wormhole_b0', name: 'benchmark_q5', tests: ['tests/models/deepseek/test_deepseek.py::test_deepseek[full-deepseek-ai/DeepSeek-V3-eval]']},
-          ]
+          {runs-on: 'wormhole_b0', name: 'bmk_1_qtn_test_mamba[full-state-spaces/mamba-2.8b-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-2.8b-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_2_qtn_test_mamba[full-state-spaces/mamba-370m-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-370m-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_3_qtn_test_mamba[full-state-spaces/mamba-1.4b-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-1.4b-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_4_qtn_test_mamba[full-state-spaces/mamba-790m-hf-eval]', tests: ['tests/models/mamba/test_mamba.py::test_mamba[full-state-spaces/mamba-790m-hf-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_5', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_32gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_800mf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_8gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet101]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-dla34.in1k]', 'tests/models/unet_carvana/test_unet_carvana.py::test_unet_carvana[full-eval]', 'tests/models/bloom/test_bloom.py::test_bloom[full-eval]', 'tests/models/roberta/test_roberta.py::test_roberta[full-eval]', 'tests/models/unet_brain/test_unet_brain.py::test_unet_brain[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet18]', 'tests/models/albert/test_albert_question_answering.py::test_albert_question_answering[full-twmkn9/albert-base-v2-squad2-eval]', 'tests/models/clip/test_clip.py::test_clip[full-eval]', 'tests/models/xglm/test_xglm.py::test_xglm[full-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-base-patch16-224-eval]', 'tests/models/yolos/test_yolos.py::test_yolos[full-eval]', 'tests/models/gpt2/test_gpt2.py::test_gpt2[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_6', tests: ['tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-1.5-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-base-eval]', 'tests/models/mobilenet_ssd/test_mobilenet_ssd.py::test_mobilenet_ssd[full-eval]', 'tests/models/flan_t5/test_flan_t5.py::test_flan_t5[full-eval]', 'tests/models/RMBG/test_RMBG.py::test_RMBG[full-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite0.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet101_2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19]', 'tests/models/llama/test_llama_3b.py::test_llama_3b[full-meta-llama/Llama-3.2-3B-eval]', 'tests/models/resnet50/test_resnet50.py::test_resnet[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext50_32x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_800mf]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xlarge-v2-eval]', 'tests/models/falcon/test_falcon.py::test_falcon[full-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-base-v2-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_s]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_s]', 'tests/models/vilt/test_vilt.py::test_vilt[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_t]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssd300_vgg16-eval]', 'tests/models/autoencoder_conv/test_autoencoder_conv_v2.py::test_autoencoder_conv_v2[full-eval]', 'tests/models/stable_diffusion/test_stable_diffusion.py::test_stable_diffusion[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_7', tests: ['tests/models/gpt_neo/test_gpt_neo.py::test_gpt_neo[full-eval]', 'tests/models/deepseek/test_deepseek_qwen.py::test_deepseek_qwen[full-deepseek-ai/DeepSeek-R1-Distill-Qwen-32B-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_32gf]', 'tests/models/hardnet/test_hardnet.py::test_hardnet[full-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite2.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite1.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_64x4d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_400mf]', 'tests/models/perceiver_io/test_perceiver_io.py::test_perceiver_io[full-eval]', 'tests/models/whisper/test_whisper.py::test_whisper[full-eval]', 'tests/models/distilbert/test_distilbert.py::test_distilbert_multiloop[full-distilbert-base-uncased-eval-64]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_3_2gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11]', 'tests/models/openpose/test_openpose_v2.py::test_openpose_v2[full-eval]', 'tests/models/Qwen/test_qwen2_token_classification.py::test_qwen2_token_classification[full-Qwen/Qwen2-7B-eval]', 'tests/models/dpr/test_dpr.py::test_dpr[full-eval]', 'tests/models/beit/test_beit_image_classification.py::test_beit_image_classification[full-microsoft/beit-large-patch16-224-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-inception_v4.tf_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet169]', 'tests/models/deit/test_deit.py::test_deit[full-facebook/deit-base-patch16-224-eval]', 'tests/models/autoencoder_linear/test_autoencoder_linear.py::test_autoencoder_linear[full-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn_v2-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_8', tests: ['tests/models/opt/test_opt.py::test_opt[full-eval]', 'tests/models/glpn_kitti/test_glpn_kitti.py::test_glpn_kitti[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_h_14]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite4.in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnet_100.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_large]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-mobilenet_v3_small]', 'tests/models/t5/test_t5.py::test_t5[full-t5-small-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg19_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13_bn]', 'tests/models/mlpmixer/test_mlpmixer.py::test_mlpmixer[full-eval]', 'tests/models/llama/test_llama_7b.py::test_llama_7b[full-eval]', 'tests/models/mgp-str-base/test_mgp_str_base.py::test_mgp_str_base[full-eval]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-large-v2-eval]', 'tests/models/squeeze_bert/test_squeeze_bert.py::test_squeeze_bert[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_v2_b]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet201]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet161]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-ghostnetv2_100.in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_16]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_red[full-eval-ese_vovnet19b_dw.ra_in1k]', 'tests/models/codegen/test_codegen.py::test_codegen[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_9', tests: ['tests/models/Qwen/test_qwen2_casual_lm.py::test_qwen2_casual_lm[full-Qwen/Qwen2.5-1.5B-eval]', 'tests/models/t5/test_t5.py::test_t5[full-t5-large-eval]', 'tests/models/yolov3/test_yolov3.py::test_yolov3[full-eval]', 'tests/models/yolov4/test_yolov4.py::test_yolov4[full-eval]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-ssdlite320_mobilenet_v3_large-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-tf_efficientnet_lite3.in1k]', 'tests/models/detr/test_detr.py::test_detr[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_1_6gf]', 'tests/models/MobileNetV2/test_MobileNetV2.py::test_MobileNetV2[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_16gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-wide_resnet50_2]', 'tests/models/albert/test_albert_masked_lm.py::test_albert_masked_lm[full-albert/albert-xxlarge-v2-eval]', 'tests/models/bert/test_bert.py::test_bert[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg11_bn]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mobilenetv1_100.ra4_e3600_r224_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet34]', 'tests/models/distilbert/test_distilbert.py::test_distilbert[full-distilbert-base-uncased-eval]', 'tests/models/albert/test_albert_sequence_classification.py::test_albert_sequence_classification[full-textattack/albert-base-v2-imdb-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_l_32]', 'tests/models/mnist/test_mnist.py::test_mnist_train[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-densenet121]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-vit_b_32]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_t]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_10', tests: ['tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_128gf]', 'tests/models/deepseek/test_deepseek.py::test_deepseek[full-deepseek-ai/DeepSeek-V3-eval]', 'tests/models/segformer/test_segformer.py::test_segformer[full-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-hrnet_w18.ms_aug_in1k]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-xception71.tf_in1k]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_y_8gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnext101_32x8d]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet152]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-mobilenet_v2]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg16_bn]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-resnet50]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-vgg13]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_1_6gf]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-regnet_x_400mf]', 'tests/models/unet/test_unet.py::test_unet[full-eval]', 'tests/models/resnet/test_resnet.py::test_resnet[full-eval]', 'tests/models/albert/test_albert_token_classification.py::test_albert_token_classification[full-albert/albert-base-v2-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_red[full-eval-swin_b]', 'tests/models/phi/test_phi_1_1p5_2.py::test_phi[full-microsoft/phi-2-eval]', 'tests/models/timm/test_timm_image_classification.py::test_timm_image_classification_generality[full-eval-mixer_b16_224.goog_in21k]', 'tests/models/torchvision/test_torchvision_object_detection.py::test_torchvision_object_detection[full-retinanet_resnet50_fpn-eval]', 'tests/models/torchvision/test_torchvision_image_classification.py::test_torchvision_image_classification_generality[full-eval-googlenet]', 'tests/models/MobileNetV2/test_MobileNetV2_onnx.py::test_MobileNetV2[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_11_qtn_test_autoencoder_conv[full-eval]', tests: ['tests/models/autoencoder_conv/test_autoencoder_conv.py::test_autoencoder_conv[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_12_qtn_test_hand_landmark[full-eval]', tests: ['tests/models/hand_landmark/test_hand_landmark.py::test_hand_landmark[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_13_qtn_test_mistral_7b[full-eval]', tests: ['tests/models/mistral/test_mistral_7b.py::test_mistral_7b[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_14_qtn_test_musicgen_small[full-eval]', tests: ['tests/models/musicgen_small/test_musicgen_small.py::test_musicgen_small[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_15_qtn_test_openpose[full-eval]', tests: ['tests/models/openpose/test_openpose.py::test_openpose[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_16_qtn_test_segment_anything[full-eval]', tests: ['tests/models/segment_anything/test_segment_anything.py::test_segment_anything[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_17_qtn_test_speecht5_tts[full-eval]', tests: ['tests/models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_18_qtn_test_vit[full-eval]', tests: ['tests/models/vit/test_vit.py::test_vit[full-eval]']},
+          {runs-on: 'wormhole_b0', name: 'bmk_19_qtn_test_yolov5[full-eval]', tests: ['tests/models/yolov5/test_yolov5.py::test_yolov5[full-eval]']},
+        ]
+
path: .github/workflows/run-depth-benchmark-tests.yml
user: login: jameszianxuTT
body: The next step here is to replace all these testlists between `build [ * ]` with a single `${{ inputs.matrix-json}}` but there are a bunch of github-actions-specific issues with passing it between the previous job and this one. I have just copy-pasted the result of the previous job here, but this should be fully automated by next week.
created_at: 2025-04-04T21:52:30Z
author_association: COLLABORATOR

PR comments:
user: login: jameszianxuTT
created_at: 2025-04-04T21:23:02Z
body: This autopromotion + load balancer + quarantiner is neat because it doesn't require us to commit the table of results to tree. 

Instead, we use @kmabeeTT 's artifact downloader tool to pull the output report artifact from last week's benchmark (run on main) and parse it to extract timings and a testlist. 

From there, we can find the set of tests we ran last week, and therefore the set of tests this week that should be quarantined (because they were not run last week). We can also load balance the tests because we know their test durations.

Note: This fails at the very last step, because of some complexity in how to pass a dynamically generated test matrix between jobs. I cannot pass a file to specify the test matrix, I must make a properly formatted matrix as a JSON string, and properly escape/unescape everything per Github Actions spec. 

Github actions also has issues passing around a multiline string between jobs, and I'm sure I'll run into more quirks as I try to fix this
author_association: COLLABORATOR
user: login: github-actions[bot]
created_at: 2025-04-04T21:35:12Z
body: <table><tr><th><th>Tests</th><th>Passed ✅</th><th>Skipped ⚠️</th><th>Failed</th></tr><tr><td>TT-Torch Tests</td><td>468 ran</td><td>461 passed</td><td>7 skipped</td><td>0 failed</td></tr></table>

<table><tr><th>Test</th><th>Result</th></tr><tr><td colspan="2">No test annotations available</td></tr></table>

<!-- Summary comment for ["TT-Torch Tests"] by mikepenz/action-junit-report -->
author_association: NONE
user: login: codecov-commenter
created_at: 2025-04-04T21:35:48Z
body: ## [Codecov](https://app.codecov.io/gh/tenstorrent/tt-torch/pull/564?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 71.56%. Comparing base [(`d5bd54f`)](https://app.codecov.io/gh/tenstorrent/tt-torch/commit/d5bd54fb0676f2343905a504d6461eb8b934a381?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) to head [(`c196634`)](https://app.codecov.io/gh/tenstorrent/tt-torch/commit/c1966342c2323a96597a2d2141e9542d9f843c5a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).

:white_check_mark: All tests successful. No failed tests found.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #564      +/-   ##
==========================================
- Coverage   74.64%   71.56%   -3.09%     
==========================================
  Files          16       16              
  Lines        2138     2138              
==========================================
- Hits         1596     1530      -66     
- Misses        542      608      +66     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/tenstorrent/tt-torch/pull/564?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).

author_association: NONE