Repository: NVIDIA/TensorRT-LLM
Pull request #4242: [TRTLLM-4932] Add QA accuracy tests for NIM-prioritized models
Author: login: moraxu
Created: 2025-05-12T20:11:08Z | Merged: 2025-05-24T11:17:21Z

New tests added:
- Llama-3.2-1B: ~~added mmlu benchmark~~ - removed due to low accuracy
- Llama-3.1-Nemotron-Nano-8B-v1: added GSM8K, GPQADiamond benchmarks
- Llama-3_1-Nemotron-Ultra-253B-v1: added the entire model (FP8 variant is being added to `ftp/llm-models`)
- Phi-4-mini-instruct: added the model to the tests; skipped the test as the model likely has to be added to Torch models first (given the current error)


Diff:
diff --git a/tests/integration/defs/accuracy/references/cnn_dailymail.yaml b/tests/integration/defs/accuracy/references/cnn_dailymail.yaml
index df5f1d7f3d29..afe2c7a485b4 100644
--- a/tests/integration/defs/accuracy/references/cnn_dailymail.yaml
+++ b/tests/integration/defs/accuracy/references/cnn_dailymail.yaml
@@ -137,6 +137,8 @@ meta-llama/Llama-3.2-1B:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 27.029
+  - quant_algo: FP8
+    accuracy: 27.029
   - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
     accuracy: 27.257
   - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
@@ -310,5 +312,3 @@ Qwen3/Qwen3-8B:
     accuracy: 30
 nvidia/Llama-3_3-Nemotron-Super-49B-v1:
   - accuracy: 34.003
-nvidia/Llama-3.1-Nemotron-Nano-8B-v1:
-  - accuracy: 27.810
diff --git a/tests/integration/defs/accuracy/references/gpqa_diamond.yaml b/tests/integration/defs/accuracy/references/gpqa_diamond.yaml
index 9e4e988f335d..d5ac97761e10 100644
--- a/tests/integration/defs/accuracy/references/gpqa_diamond.yaml
+++ b/tests/integration/defs/accuracy/references/gpqa_diamond.yaml
@@ -18,3 +18,10 @@ nvidia/Llama-3_3-Nemotron-Super-49B-v1:
   - accuracy: 44.95
   - quant_algo: FP8
     accuracy: 49.49
+nvidia/Llama-3.1-Nemotron-Nano-8B-v1:
+  - accuracy: 40.40
+nvidia/Llama-3_1-Nemotron-Ultra-253B-v1:
+  - accuracy: 58.08
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 57.07
diff --git a/tests/integration/defs/accuracy/references/gsm8k.yaml b/tests/integration/defs/accuracy/references/gsm8k.yaml
index 9748b122bcd1..c39c0ce2466e 100644
--- a/tests/integration/defs/accuracy/references/gsm8k.yaml
+++ b/tests/integration/defs/accuracy/references/gsm8k.yaml
@@ -69,3 +69,10 @@ nvidia/Llama-3_3-Nemotron-Super-49B-v1:
     accuracy: 92.42
 nvidia/Nemotron-H-8B-Base-8K:
   - accuracy: 46.20
+nvidia/Llama-3.1-Nemotron-Nano-8B-v1:
+  - accuracy: 37.15
+nvidia/Llama-3_1-Nemotron-Ultra-253B-v1:
+  - accuracy: 94.43
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 94.16
diff --git a/tests/integration/defs/accuracy/references/mmlu.yaml b/tests/integration/defs/accuracy/references/mmlu.yaml
index 4380b799764b..eb227058d266 100644
--- a/tests/integration/defs/accuracy/references/mmlu.yaml
+++ b/tests/integration/defs/accuracy/references/mmlu.yaml
@@ -28,6 +28,26 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
+  - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN
+    accuracy: 32.72
+  - quant_algo: W8A8_SQ_PER_CHANNEL
+    accuracy: 32.07
+  - quant_algo: W4A16_AWQ
+    accuracy: 30.56
+  - quant_algo: W4A16_AWQ
+    kv_cache_quant_algo: INT8
+    accuracy: 31.29
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 31.02
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    accuracy: 33.97
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    extra_acc_spec: meta_recipe
+    accuracy: 33.87
+  - extra_acc_spec: max_attention_window_size=960
+    accuracy: 32.82
 meta-llama/Llama-3.3-70B-Instruct:
   - accuracy: 81.31
   - quant_algo: NVFP4
@@ -127,3 +147,8 @@ nvidia/Llama-3.1-Nemotron-Nano-8B-v1:
   - accuracy: 57.97
 nvidia/Nemotron-H-8B-Base-8K:
   - accuracy: 69.590
+nvidia/Llama-3_1-Nemotron-Ultra-253B-v1:
+  - accuracy: 83.70
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 83.36
diff --git a/tests/integration/defs/accuracy/test_cli_flow.py b/tests/integration/defs/accuracy/test_cli_flow.py
index 286477e7813b..0dab161a8b82 100644
--- a/tests/integration/defs/accuracy/test_cli_flow.py
+++ b/tests/integration/defs/accuracy/test_cli_flow.py
@@ -224,6 +224,73 @@ def test_fp8_prequantized_tp2(self, mocker):
                  quant_algo=QuantAlgo.FP8)
 
 
+class TestNemotronNano(CliFlowAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3.1-Nemotron-Nano-8B-v1"
+    MODEL_PATH = f"{llm_models_root()}/Llama-3.1-Nemotron-Nano-8B-v1"
+    EXAMPLE_FOLDER = "models/core/llama"
+
+    def test_auto_dtype(self):
+        self.run(tasks=[MMLU(self.MODEL_NAME)], dtype='auto')
+
+
+class TestNemotronUltra(CliFlowAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+    EXAMPLE_FOLDER = "models/core/nemotron_nas"
+
+    @skip_pre_hopper
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_auto_dtype(self, cuda_graph, tp_size, pp_size, ep_size):
+        extra_summarize_args = []
+        if cuda_graph:
+            extra_summarize_args.append("--cuda_graph_mode")
+
+        self.run(tasks=[MMLU(self.MODEL_NAME)],
+                 tp_size=tp_size,
+                 pp_size=pp_size,
+                 extra_convert_args=[
+                     f"--moe_tp_size={tp_size // ep_size}",
+                     f"--moe_ep_size={ep_size}", f"--moe_renorm_mode={0}"
+                 ],
+                 extra_build_args=["--gemm_plugin=auto", "--moe_plugin=auto"],
+                 extra_summarize_args=extra_summarize_args)
+
+    @skip_pre_hopper
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_fp8_prequantized(self, cuda_graph, tp_size, pp_size, ep_size,
+                              mocker):
+        mocker.patch.object(
+            self.__class__, "MODEL_PATH",
+            f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1-FP8"
+        )
+
+        extra_summarize_args = []
+        if cuda_graph:
+            extra_summarize_args.append("--cuda_graph_mode")
+
+        self.run(tasks=[MMLU(self.MODEL_NAME)],
+                 quant_algo=QuantAlgo.FP8,
+                 kv_cache_quant_algo=QuantAlgo.FP8,
+                 tp_size=tp_size,
+                 pp_size=pp_size,
+                 extra_convert_args=[
+                     f"--moe_tp_size={tp_size // ep_size}",
+                     f"--moe_ep_size={ep_size}", f"--moe_renorm_mode={0}"
+                 ],
+                 extra_build_args=["--gemm_plugin=auto", "--moe_plugin=auto"],
+                 extra_summarize_args=extra_summarize_args)
+
+
 class TestPhi2(CliFlowAccuracyTestHarness):
     MODEL_NAME = "microsoft/phi-2"
     MODEL_PATH = f"{llm_models_root()}/phi-2"
diff --git a/tests/integration/defs/accuracy/test_llm_api_pytorch.py b/tests/integration/defs/accuracy/test_llm_api_pytorch.py
index 3156782e977c..4b09d0d94d54 100644
--- a/tests/integration/defs/accuracy/test_llm_api_pytorch.py
+++ b/tests/integration/defs/accuracy/test_llm_api_pytorch.py
@@ -20,8 +20,9 @@
 from tensorrt_llm.models.modeling_utils import QuantConfig
 from tensorrt_llm.quantization import QuantAlgo
 
-from ..conftest import (llm_models_root, parametrize_with_ids, skip_pre_ada,
-                        skip_pre_blackwell, skip_pre_hopper)
+from ..conftest import (llm_models_root, parametrize_with_ids,
+                        skip_post_blackwell, skip_pre_ada, skip_pre_blackwell,
+                        skip_pre_hopper)
 from .accuracy_core import (GSM8K, MMLU, CnnDailymail, GPQADiamond,
                             LlmapiAccuracyTestHarness)
 
@@ -207,6 +208,79 @@ def test_fp8_llm_sampler(self):
                           extra_acc_spec="temperature=0.8,top_p=0.95")
 
 
+class TestLlama3_2_1B(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "meta-llama/Llama-3.2-1B"
+    MODEL_PATH = f"{llm_models_root()}/llama-3.2-models/Llama-3.2-1B"
+    EXAMPLE_FOLDER = "models/core/llama"
+
+    def test_auto_dtype(self):
+        with LLM(self.MODEL_PATH) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+    @skip_post_blackwell
+    def test_smooth_quant(self):
+        quant_config = QuantConfig(
+            QuantAlgo.W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN)
+        with LLM(self.MODEL_PATH, quant_config=quant_config) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+    @skip_post_blackwell
+    def test_smooth_quant_ootb(self):
+        quant_config = QuantConfig(QuantAlgo.W8A8_SQ_PER_CHANNEL)
+        with LLM(self.MODEL_PATH, quant_config=quant_config) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+    @skip_post_blackwell
+    def test_int4_awq(self):
+        quant_config = QuantConfig(QuantAlgo.W4A16_AWQ)
+        with LLM(self.MODEL_PATH, quant_config=quant_config) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+    @skip_post_blackwell
+    def test_int4_awq_int8_kv_cache(self):
+        quant_config = QuantConfig(QuantAlgo.W4A16_AWQ)
+        kv_cache_config = KvCacheConfig(quant_algo=QuantAlgo.INT8)
+        with LLM(self.MODEL_PATH,
+                 quant_config=quant_config,
+                 kv_cache_config=kv_cache_config) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+    @skip_pre_ada
+    def test_fp8(self):
+        quant_config = QuantConfig(QuantAlgo.FP8)
+        kv_cache_config = KvCacheConfig(quant_algo=QuantAlgo.FP8)
+        with LLM(self.MODEL_PATH,
+                 quant_config=quant_config,
+                 kv_cache_config=kv_cache_config) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+    @skip_pre_ada
+    @pytest.mark.skip_less_device(2)
+    def test_fp8_pp2(self):
+        quant_config = QuantConfig(QuantAlgo.FP8)
+        kv_cache_config = KvCacheConfig(quant_algo=QuantAlgo.FP8)
+        with LLM(self.MODEL_PATH,
+                 pipeline_parallel_size=2,
+                 quant_config=quant_config,
+                 kv_cache_config=kv_cache_config) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+    @skip_pre_ada
+    @skip_post_blackwell
+    def test_fp8_rowwise(self):
+        quant_config = QuantConfig(QuantAlgo.FP8_PER_CHANNEL_PER_TOKEN)
+        with LLM(self.MODEL_PATH, quant_config=quant_config) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+
+
 class TestLlama3_3_70BInstruct(LlmapiAccuracyTestHarness):
     MODEL_NAME = "meta-llama/Llama-3.3-70B-Instruct"
 
@@ -927,10 +1001,61 @@ class TestNemotronNano(LlmapiAccuracyTestHarness):
 
     def test_auto_dtype(self):
         with LLM(self.MODEL_PATH) as llm:
-            task = CnnDailymail(self.MODEL_NAME)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+
+class TestNemotronUltra(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_auto_dtype(self, cuda_graph, tp_size, pp_size, ep_size):
+        with LLM(self.MODEL_PATH,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_fp8_prequantized(self, cuda_graph, tp_size, pp_size, ep_size):
+        model_path = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1-FP8"
+        with LLM(model_path,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
+            assert llm.args.quant_config.quant_algo == QuantAlgo.FP8
+            assert llm.args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
 
 
 class TestNemotronH(LlmapiAccuracyTestHarness):
@@ -1093,3 +1218,24 @@ def test_fp8_block_scales(self, tp_size, pp_size, ep_size, attention_dp,
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+
+
+class TestPhi4MiniInstruct(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "microsoft/Phi-4-mini-instruct"
+    MODEL_PATH = f"{llm_models_root()}/Phi-4-mini-instruct"
+
+    @pytest.mark.skip(
+        reason=
+        "Temporarily skipping test_auto_dtype while resolving Phi-4's architecture issue."
+    )
+    def test_auto_dtype(self):
+        with LLM(self.MODEL_PATH) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
diff --git a/tests/integration/test_lists/qa/examples_test_list.txt b/tests/integration/test_lists/qa/examples_test_list.txt
index b6f31c083cd5..42294d027e9d 100644
--- a/tests/integration/test_lists/qa/examples_test_list.txt
+++ b/tests/integration/test_lists/qa/examples_test_list.txt
@@ -375,6 +375,14 @@ accuracy/test_cli_flow.py::TestLlama3_2_1B::test_fp8_rowwise
 accuracy/test_cli_flow.py::TestLlama3_2_1B::test_weight_streaming[1.0]
 accuracy/test_cli_flow.py::TestLlama3_2_1B::test_cyclic_kv_cache
 accuracy/test_cli_flow.py::TestLlama3_2_1B::test_cyclic_kv_cache_beam_search
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_smooth_quant
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_smooth_quant_ootb
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_int4_awq
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_int4_awq_int8_kv_cache
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_fp8
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_fp8_pp2
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_fp8_rowwise
 accuracy/test_cli_flow.py::TestMistral7B::test_beam_search
 accuracy/test_cli_flow.py::TestMistral7B::test_fp8_tp4pp2
 accuracy/test_cli_flow.py::TestMistral7B::test_smooth_quant_tp4pp1
@@ -450,6 +458,11 @@ accuracy/test_llm_api_pytorch.py::TestLlama3_3NemotronSuper49Bv1::test_fp8_prequ
 accuracy/test_cli_flow.py::TestLlama3_3NemotronSuper49Bv1::test_auto_dtype_tp2
 accuracy/test_cli_flow.py::TestLlama3_3NemotronSuper49Bv1::test_fp8_prequantized_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_cli_flow.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8ep4-cuda_graph=True]
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_fp8_prequantized[tp8ep4-cuda_graph=True]
+accuracy/test_cli_flow.py::TestNemotronUltra::test_auto_dtype[tp8ep4-cuda_graph=True]
+accuracy/test_cli_flow.py::TestNemotronUltra::test_fp8_prequantized[tp8ep4-cuda_graph=True]
 accuracy/test_llm_api_pytorch.py::TestNemotronH::test_auto_dtype
 accuracy/test_llm_api_pytorch.py::TestQwen2_7BInstruct::test_auto_dtype
 accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_8gpus[latency]
diff --git a/tests/integration/test_lists/qa/llm_sanity_test.txt b/tests/integration/test_lists/qa/llm_sanity_test.txt
index e79a38e73ed5..fd6776a5fc07 100644
--- a/tests/integration/test_lists/qa/llm_sanity_test.txt
+++ b/tests/integration/test_lists/qa/llm_sanity_test.txt
@@ -113,6 +113,9 @@ accuracy/test_cli_flow.py::TestLlama3_1_8B::test_fp8_rowwise_tp4[disable_gemm_al
 accuracy/test_cli_flow.py::TestLlama3_1_8B::test_autoq
 accuracy/test_cli_flow.py::TestLlama3_1_8BInstruct::test_medusa_fp8_prequantized
 accuracy/test_cli_flow.py::TestLlama3_2_1B::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestLlama3_2_1B::test_auto_dtype
+accuracy/test_cli_flow.py::TestLlama3_3_70BInstruct::test_fp8_prequantized_tp4
+accuracy/test_cli_flow.py::TestLlama3_3_70BInstruct::test_nvfp4_prequantized_tp4
 accuracy/test_cli_flow.py::TestMistral7B::test_fp8_tp4pp2
 accuracy/test_cli_flow.py::TestMistral7B::test_beam_search
 accuracy/test_cli_flow.py::TestMixtral8x7B::test_fp8_tp2pp2
@@ -139,9 +142,15 @@ accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestLlama3_3NemotronSuper49Bv1::test_auto_dtype_tp2
 accuracy/test_cli_flow.py::TestLlama3_3NemotronSuper49Bv1::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_cli_flow.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8-cuda_graph=False]
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_fp8_prequantized[tp8-cuda_graph=False]
+accuracy/test_cli_flow.py::TestNemotronUltra::test_auto_dtype[tp8-cuda_graph=False]
+accuracy/test_cli_flow.py::TestNemotronUltra::test_fp8_prequantized[tp8-cuda_graph=False]
 accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False]
 accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp_nextn=0-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False]
 accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4[fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False]
+accuracy/test_llm_api_pytorch.py::TestPhi4MiniInstruct::test_auto_dtype
 
 # Pivot to Pytorch test cases.
 test_e2e.py::test_ptp_quickstart_advanced[Llama3.1-8B-BF16-llama-3.1-model/Meta-Llama-3.1-8B]


Reviews:
user: login: tijyojwad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T20:49:00Z
user: login: moraxu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T20:54:33Z
user: login: moraxu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T06:26:28Z
user: login: syuoni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T06:38:33Z
user: login: crazydemo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T06:52:07Z
user: login: moraxu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T07:17:12Z
user: login: crazydemo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T07:23:56Z
user: login: moraxu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T00:50:52Z
user: login: crazydemo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T05:30:48Z
user: login: crazydemo
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-15T05:33:29Z
user: login: Tracin
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T08:28:45Z
user: login: Tracin
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T08:32:32Z
user: login: Tracin
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T08:42:11Z
user: login: moraxu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T17:24:03Z
user: login: syuoni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T01:37:24Z
user: login: chang-l
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T05:21:19Z
user: login: chang-l
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T05:31:44Z
user: login: moraxu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T15:35:35Z
user: login: LarryXFly
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-21T01:15:02Z
user: login: chang-l
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-21T16:29:58Z
user: login: schetlur-nv
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-23T19:17:48Z

Review comments:
diff_hunk: @@ -12,3 +12,7 @@ deepseek-ai/DeepSeek-R1:
     accuracy: 70.06
 nvidia/Llama-3_3-Nemotron-Super-49B-v1:
   - accuracy: 44.95
+nvidia/Llama-3.1-Nemotron-Nano-8B-v1:
+  - accuracy: 40.40
+# nvidia/Llama-3_1-Nemotron-Ultra-253B-v1:
+#   - accuracy: 34.003
path: tests/integration/defs/accuracy/references/gpqa_diamond.yaml
user: login: tijyojwad
body: can you add a comment that the accuracy is placeholder and should be updated? alternatively, if you have that value already, let's add it in
created_at: 2025-05-12T20:40:26Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
+  - accuracy: 32.82
+  - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN
+    accuracy: 32.72
+  - quant_algo: W8A8_SQ_PER_CHANNEL
+    accuracy: 32.07
+  - quant_algo: W4A16_AWQ
+    accuracy: 30.56
+  - quant_algo: W4A16_AWQ
+    kv_cache_quant_algo: INT8
+    accuracy: 31.29
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 31.02
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    accuracy: 33.97
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    extra_acc_spec: meta_recipe
+    accuracy: 33.87
+  - extra_acc_spec: max_attention_window_size=960
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: tijyojwad
body: what is this use case covering? some sort of sliding window?
created_at: 2025-05-12T20:41:07Z
author_association: COLLABORATOR
diff_hunk: @@ -698,7 +728,42 @@ def test_auto_dtype(self):
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+
+class TestNemotronUltra(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    def test_auto_dtype(self):
+        with LLM(self.MODEL_PATH) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
 
+    # @pytest.mark.skip_less_device(8)
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: tijyojwad
body: instead of comment this skip, can we mark this test as skipped for now with a reason?
created_at: 2025-05-12T20:48:53Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
+  - accuracy: 32.82
+  - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN
+    accuracy: 32.72
+  - quant_algo: W8A8_SQ_PER_CHANNEL
+    accuracy: 32.07
+  - quant_algo: W4A16_AWQ
+    accuracy: 30.56
+  - quant_algo: W4A16_AWQ
+    kv_cache_quant_algo: INT8
+    accuracy: 31.29
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 31.02
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    accuracy: 33.97
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    extra_acc_spec: meta_recipe
+    accuracy: 33.87
+  - extra_acc_spec: max_attention_window_size=960
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: moraxu
body: I believe it defines the maximum number of tokens that can be attended to in the attention at any given time. However, the corresponding test `test_cyclic_kv_cache` is the only one that has been failing for me locally. I will shortly paste the trace, I've temporarily lost it.
created_at: 2025-05-12T20:53:53Z
author_association: COLLABORATOR
diff_hunk: @@ -12,3 +12,7 @@ deepseek-ai/DeepSeek-R1:
     accuracy: 70.06
 nvidia/Llama-3_3-Nemotron-Super-49B-v1:
   - accuracy: 44.95
+nvidia/Llama-3.1-Nemotron-Nano-8B-v1:
+  - accuracy: 40.40
+# nvidia/Llama-3_1-Nemotron-Ultra-253B-v1:
+#   - accuracy: 34.003
path: tests/integration/defs/accuracy/references/gpqa_diamond.yaml
user: login: moraxu
body: I'll update it shortly, just got access to 8 GPUs required for the tests.
created_at: 2025-05-12T20:54:17Z
author_association: COLLABORATOR
diff_hunk: @@ -698,7 +728,42 @@ def test_auto_dtype(self):
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+
+class TestNemotronUltra(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    def test_auto_dtype(self):
+        with LLM(self.MODEL_PATH) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
 
+    # @pytest.mark.skip_less_device(8)
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: moraxu
body: same as above, I'll update it shortly, just got access to 8 GPUs required for the tests

EDIT: Actually, can skip for now, as I'm still waiting for being able to add some models to the scratch
created_at: 2025-05-12T20:54:31Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
+  - accuracy: 32.82
+  - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN
+    accuracy: 32.72
+  - quant_algo: W8A8_SQ_PER_CHANNEL
+    accuracy: 32.07
+  - quant_algo: W4A16_AWQ
+    accuracy: 30.56
+  - quant_algo: W4A16_AWQ
+    kv_cache_quant_algo: INT8
+    accuracy: 31.29
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 31.02
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    accuracy: 33.97
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    extra_acc_spec: meta_recipe
+    accuracy: 33.87
+  - extra_acc_spec: max_attention_window_size=960
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: moraxu
body: @syuoni , the `tests/integration/defs/accuracy/test_cli_flow.py::TestLlama3_2_1B::test_cyclic_kv_cache` has been failing for me for the existing `CnnDailymail`:
```
[TensorRT-LLM][INFO] Refreshed the MPI local session
[05/13/2025-05:39:27] [TRT-LLM] [I] TensorRT-LLM (total latency: 5.606834173202515 sec)
[05/13/2025-05:39:27] [TRT-LLM] [I] TensorRT-LLM (total output tokens: 43895)
[05/13/2025-05:39:27] [TRT-LLM] [I] TensorRT-LLM (tokens per second: 7828.838635855005)
[05/13/2025-05:39:27] [TRT-LLM] [I] TensorRT-LLM beam 0 result
[05/13/2025-05:39:29] [TRT-LLM] [I]   rouge1: 24.430759025787125
[05/13/2025-05:39:29] [TRT-LLM] [I]   rouge2: 8.86259392441887
[05/13/2025-05:39:29] [TRT-LLM] [I]   rougeL: 16.81456938896439
[05/13/2025-05:39:29] [TRT-LLM] [I]   rougeLsum: 21.009406685676627
Traceback (most recent call last):
  File "/code/tensorrt_llm/examples/summarize.py", line 889, in <module>
    main(args)
  File "/code/tensorrt_llm/examples/summarize.py", line 795, in main
    assert rouge1 > args.tensorrt_llm_rouge1_threshold, f"[FAILED] rouge1 ({rouge1}) is smaller than threshold ({args.tensorrt_llm_rouge1_threshold})."
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: [FAILED] rouge1 (24.430759025787125) is smaller than threshold (25.26947069785025).
```
Given it's on the QA list, how can I check if this failure is already tracked?
created_at: 2025-05-13T05:44:09Z
author_association: COLLABORATOR
diff_hunk: @@ -805,3 +861,24 @@ def test_fp8_block_scales(self, tp_size, pp_size, ep_size, attention_dp,
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+
+
+class TestPhi4MiniInstruct(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "microsoft/Phi-4-mini-instruct"
+    MODEL_PATH = f"{llm_models_root()}/Phi-4-mini-instruct"
+
+    @pytest.mark.skip(
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: moraxu
body: Given the error I got:
```
[05/13/2025-05:45:54] [TRT-LLM] [I] Fallback to regular model init: Traceback (most recent call last):
  File "/code/tensorrt_llm/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 867, in _load_model
    model = AutoModelForCausalLM.from_config(config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Unknown architecture for AutoModelForCausalLM: Phi3ForCausalLM


[05/13/2025-05:45:54] [TRT-LLM] [E] Failed to initialize executor on rank 0: Unknown architecture for AutoModelForCausalLM: Phi3ForCausalLM
[05/13/2025-05:45:54] [TRT-LLM] [E] Traceback (most recent call last):
  File "/code/tensorrt_llm/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 867, in _load_model
    model = AutoModelForCausalLM.from_config(config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_auto.py", line 24, in from_config
    raise ValueError(
ValueError: Unknown architecture for AutoModelForCausalLM: Phi3ForCausalLM
```
I'd have to follow https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/torch/adding_new_model.md to add Phi, correct?
created_at: 2025-05-13T05:59:50Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
+  - accuracy: 32.82
+  - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN
+    accuracy: 32.72
+  - quant_algo: W8A8_SQ_PER_CHANNEL
+    accuracy: 32.07
+  - quant_algo: W4A16_AWQ
+    accuracy: 30.56
+  - quant_algo: W4A16_AWQ
+    kv_cache_quant_algo: INT8
+    accuracy: 31.29
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 31.02
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    accuracy: 33.97
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    extra_acc_spec: meta_recipe
+    accuracy: 33.87
+  - extra_acc_spec: max_attention_window_size=960
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: syuoni
body: The test is waived: https://github.com/NVIDIA/TensorRT-LLM/blob/main/tests/integration/test_lists/waives.txt#L443
created_at: 2025-05-13T06:38:33Z
author_association: COLLABORATOR
diff_hunk: @@ -438,6 +438,8 @@ accuracy/test_llm_api_pytorch.py::TestMinitron4BBaseInstruct::test_fp8_prequanti
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype
path: tests/integration/test_lists/qa/examples_test_list.txt
user: login: crazydemo
body: Could you please notify the parallelism strategy on this case? 
BTW, how long does this test case take to run?
created_at: 2025-05-13T06:52:07Z
author_association: COLLABORATOR
diff_hunk: @@ -438,6 +438,8 @@ accuracy/test_llm_api_pytorch.py::TestMinitron4BBaseInstruct::test_fp8_prequanti
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype
path: tests/integration/test_lists/qa/examples_test_list.txt
user: login: moraxu
body: 1. Ah, right, I forgot to ask why for other models you chose only specific combinations, like:
```
accuracy/test_llm_api_pytorch.py::TestLlama4MaverickInstruct::test_auto_dtype[tp8-cuda_graph=False]
accuracy/test_llm_api_pytorch.py::TestLlama4MaverickInstruct::test_auto_dtype[tp8ep4-cuda_graph=True]
accuracy/test_llm_api_pytorch.py::TestLlama4MaverickInstruct::test_auto_dtype[tp8ep8-cuda_graph=True]
```
Was it on purpose based on some known performance characteristics?

2. Definitely between 0.5 to 1h due to GPQADiamond alone. I can compare with the second largest model Llama-3.3-70B-Instruct that also uses this bench. What time would be prohibitive?
created_at: 2025-05-13T07:17:12Z
author_association: COLLABORATOR
diff_hunk: @@ -438,6 +438,8 @@ accuracy/test_llm_api_pytorch.py::TestMinitron4BBaseInstruct::test_fp8_prequanti
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype
path: tests/integration/test_lists/qa/examples_test_list.txt
user: login: crazydemo
body: > Ah, right, I forgot to ask why for other models you chose only specific combinations, like:

I choose these combinations to align with the [L0 test cases](https://github.com/NVIDIA/TensorRT-LLM/commit/906cddffb0e5f20df98ccb7484d1e93f00170a61).

> What time would be prohibitive?

Currently, QA has 4 hours time limitation for one test case. So this case is acceptable.
created_at: 2025-05-13T07:23:56Z
author_association: COLLABORATOR
diff_hunk: @@ -438,6 +438,8 @@ accuracy/test_llm_api_pytorch.py::TestMinitron4BBaseInstruct::test_fp8_prequanti
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype
path: tests/integration/test_lists/qa/examples_test_list.txt
user: login: moraxu
body: @crazydemo , `accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8ep8-cuda_graph=True]` ran for 77 mins on 8 H100s.
created_at: 2025-05-15T00:50:51Z
author_association: COLLABORATOR
diff_hunk: @@ -438,6 +438,8 @@ accuracy/test_llm_api_pytorch.py::TestMinitron4BBaseInstruct::test_fp8_prequanti
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype
path: tests/integration/test_lists/qa/examples_test_list.txt
user: login: crazydemo
body: Thanks for letting me know, it is acceptable.
created_at: 2025-05-15T05:30:47Z
author_association: COLLABORATOR
diff_hunk: @@ -805,3 +861,24 @@ def test_fp8_block_scales(self, tp_size, pp_size, ep_size, attention_dp,
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+
+
+class TestPhi4MiniInstruct(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "microsoft/Phi-4-mini-instruct"
+    MODEL_PATH = f"{llm_models_root()}/Phi-4-mini-instruct"
+
+    @pytest.mark.skip(
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: Tracin
body: I think we already support Phi-4-mini-instruct. You are supposed to run it directly via llm_api.
created_at: 2025-05-16T08:28:44Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: Tracin
body: Why Llama3.2-1B test cases are not adding into QA list?
created_at: 2025-05-16T08:32:31Z
author_association: COLLABORATOR
diff_hunk: @@ -442,6 +442,12 @@ accuracy/test_llm_api_pytorch.py::TestMinitron4BBaseInstruct::test_fp8_prequanti
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8-cuda_graph=False]
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8ep4-cuda_graph=True]
path: tests/integration/test_lists/qa/examples_test_list.txt
user: login: Tracin
body: tp8ep4(tp2) can also cover tp8tp8 and tp8ep8. So I would suggest remove two others.
created_at: 2025-05-16T08:42:11Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: moraxu
body: I see one TestLlama3_2_1B in `llm_sanity_test.txt` and all of them in `examples_test_list.txt`?
created_at: 2025-05-16T17:11:25Z
author_association: COLLABORATOR
diff_hunk: @@ -442,6 +442,12 @@ accuracy/test_llm_api_pytorch.py::TestMinitron4BBaseInstruct::test_fp8_prequanti
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8-cuda_graph=False]
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8ep4-cuda_graph=True]
path: tests/integration/test_lists/qa/examples_test_list.txt
user: login: moraxu
body: Ah I see, I removed the two others. I assume that by tp8tp8 you actually meant tp8?
created_at: 2025-05-16T17:22:15Z
author_association: COLLABORATOR
diff_hunk: @@ -805,3 +861,24 @@ def test_fp8_block_scales(self, tp_size, pp_size, ep_size, attention_dp,
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+
+
+class TestPhi4MiniInstruct(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "microsoft/Phi-4-mini-instruct"
+    MODEL_PATH = f"{llm_models_root()}/Phi-4-mini-instruct"
+
+    @pytest.mark.skip(
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: moraxu
body: But that's how I'm running those tests in this file, right? I don't think Phi exists in `tensorrt_llm/_torch/models`, hence the error?
created_at: 2025-05-16T17:24:00Z
author_association: COLLABORATOR
diff_hunk: @@ -704,6 +704,62 @@ def test_auto_dtype(self):
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+
+class TestNemotronUltra(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_auto_dtype(self, cuda_graph, tp_size, pp_size, ep_size):
+        with LLM(self.MODEL_PATH,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: syuoni
body: If we can afford GSM8K, CnnDailymail is not necessary. GSM8K is more difficult than CnnDailymail.
created_at: 2025-05-19T01:33:21Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
+  - accuracy: 32.82
+  - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN
+    accuracy: 32.72
+  - quant_algo: W8A8_SQ_PER_CHANNEL
+    accuracy: 32.07
+  - quant_algo: W4A16_AWQ
+    accuracy: 30.56
+  - quant_algo: W4A16_AWQ
+    kv_cache_quant_algo: INT8
+    accuracy: 31.29
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 31.02
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    accuracy: 33.97
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    extra_acc_spec: meta_recipe
+    accuracy: 33.87
+  - extra_acc_spec: max_attention_window_size=960
+    accuracy: 32.82
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: syuoni
body: The MMLU scores seem too low -- a random guess can give 25 on MMLU.

Hence, I think MMLU is too difficult for this model.
created_at: 2025-05-19T01:36:05Z
author_association: COLLABORATOR
diff_hunk: @@ -704,6 +704,62 @@ def test_auto_dtype(self):
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+
+class TestNemotronUltra(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_auto_dtype(self, cuda_graph, tp_size, pp_size, ep_size):
+        with LLM(self.MODEL_PATH,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+    @pytest.mark.skip(
+        reason=
+        "Temporarily skipping test_fp8_tp8 due to not having FP8 model on scratch."
+    )
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_fp8_tp8(self, cuda_graph, tp_size, pp_size, ep_size):
+        model_path = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1-FP8"
+        with LLM(model_path,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
+            assert llm.args.quant_config.quant_algo == QuantAlgo.FP8
+            assert llm.args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: chang-l
body: For FP8 accuracy, can we add some test cases involving reasoning, such as `gsm8k` or `gpqa` for the 253B model?
created_at: 2025-05-19T05:21:19Z
author_association: COLLABORATOR
diff_hunk: @@ -136,9 +136,12 @@ accuracy/test_llm_api_pytorch.py::TestMixtral8x7B::test_nvfp4_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8-cuda_graph=False]
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_fp8_tp8[tp8-cuda_graph=False]
path: tests/integration/test_lists/qa/llm_sanity_test.txt
user: login: chang-l
body: Super nit:  can we rename this test to `test_fp8_<QUANT_ALG>` instead of `..._tp8`, since tp8 is already reflected in the following parameterized ID?
created_at: 2025-05-19T05:31:44Z
author_association: COLLABORATOR
diff_hunk: @@ -25,6 +25,27 @@ meta-llama/Llama-3.1-8B-Instruct:
   - quant_algo: FP8
     kv_cache_quant_algo: FP8
     accuracy: 67.87
+meta-llama/Llama-3.2-1B:
+  - accuracy: 32.82
+  - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN
+    accuracy: 32.72
+  - quant_algo: W8A8_SQ_PER_CHANNEL
+    accuracy: 32.07
+  - quant_algo: W4A16_AWQ
+    accuracy: 30.56
+  - quant_algo: W4A16_AWQ
+    kv_cache_quant_algo: INT8
+    accuracy: 31.29
+  - quant_algo: FP8
+    kv_cache_quant_algo: FP8
+    accuracy: 31.02
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    accuracy: 33.97
+  - quant_algo: FP8_PER_CHANNEL_PER_TOKEN
+    extra_acc_spec: meta_recipe
+    accuracy: 33.87
+  - extra_acc_spec: max_attention_window_size=960
+    accuracy: 32.82
path: tests/integration/defs/accuracy/references/mmlu.yaml
user: login: moraxu
body: Thanks for clarifying, reverted.
created_at: 2025-05-19T07:20:35Z
author_association: COLLABORATOR
diff_hunk: @@ -704,6 +704,62 @@ def test_auto_dtype(self):
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+
+class TestNemotronUltra(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_auto_dtype(self, cuda_graph, tp_size, pp_size, ep_size):
+        with LLM(self.MODEL_PATH,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: moraxu
body: Fixed
created_at: 2025-05-19T07:23:10Z
author_association: COLLABORATOR
diff_hunk: @@ -704,6 +704,62 @@ def test_auto_dtype(self):
             task.evaluate(llm)
             task = MMLU(self.MODEL_NAME)
             task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+
+class TestNemotronUltra(LlmapiAccuracyTestHarness):
+    MODEL_NAME = "nvidia/Llama-3_1-Nemotron-Ultra-253B-v1"
+    MODEL_PATH = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1"
+
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_auto_dtype(self, cuda_graph, tp_size, pp_size, ep_size):
+        with LLM(self.MODEL_PATH,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GSM8K(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = GPQADiamond(self.MODEL_NAME)
+            task.evaluate(llm,
+                          extra_evaluator_kwargs=dict(apply_chat_template=True))
+
+    @pytest.mark.skip(
+        reason=
+        "Temporarily skipping test_fp8_tp8 due to not having FP8 model on scratch."
+    )
+    @pytest.mark.skip_less_device(8)
+    @pytest.mark.skip_device_not_contain(["H100", "B200"])
+    @parametrize_with_ids("cuda_graph", [False, True])
+    @pytest.mark.parametrize("tp_size,pp_size,ep_size", [(8, 1, 1), (8, 1, 4),
+                                                         (8, 1, 8)],
+                             ids=["tp8", "tp8ep4", "tp8ep8"])
+    def test_fp8_tp8(self, cuda_graph, tp_size, pp_size, ep_size):
+        model_path = f"{llm_models_root()}/nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1-FP8"
+        with LLM(model_path,
+                 tensor_parallel_size=tp_size,
+                 pipeline_parallel_size=pp_size,
+                 moe_expert_parallel_size=ep_size,
+                 use_cuda_graph=cuda_graph) as llm:
+            assert llm.args.quant_config.quant_algo == QuantAlgo.FP8
+            assert llm.args.quant_config.kv_cache_quant_algo == QuantAlgo.FP8
+            task = CnnDailymail(self.MODEL_NAME)
+            task.evaluate(llm)
+            task = MMLU(self.MODEL_NAME)
path: tests/integration/defs/accuracy/test_llm_api_pytorch.py
user: login: moraxu
body: Done
created_at: 2025-05-19T09:38:17Z
author_association: COLLABORATOR
diff_hunk: @@ -136,9 +136,12 @@ accuracy/test_llm_api_pytorch.py::TestMixtral8x7B::test_nvfp4_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8
 accuracy/test_llm_api_pytorch.py::TestNemotronSuper::test_auto_dtype_tp2
 accuracy/test_llm_api_pytorch.py::TestNemotronNano::test_auto_dtype
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_auto_dtype[tp8-cuda_graph=False]
+accuracy/test_llm_api_pytorch.py::TestNemotronUltra::test_fp8_tp8[tp8-cuda_graph=False]
path: tests/integration/test_lists/qa/llm_sanity_test.txt
user: login: moraxu
body: Done
created_at: 2025-05-19T09:38:23Z
author_association: COLLABORATOR

PR comments:
user: login: moraxu
created_at: 2025-05-13T06:24:53Z
body: /bot run
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-13T06:29:19Z
body: @syuoni @crazydemo @LarryXFly - can you review this PR? Feel free to unassign yourself and tag someone else instead
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-13T18:38:18Z
body: /bot run
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-14T06:37:48Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T07:20:41Z
body: [PR_Github #5121](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5121/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T09:45:49Z
body: [PR_Github #5121](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5121/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3735](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3735/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-14T19:15:02Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T19:21:39Z
body: [PR_Github #5205](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5205/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T20:43:22Z
body: [PR_Github #5205](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5205/) [ run ] completed with state `FAILURE`
[/LLM/main/L0_MergeRequest_PR pipeline #3798](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3798/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-14T21:24:03Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T21:35:13Z
body: [PR_Github #5211](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5211/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-15T01:35:10Z
body: [PR_Github #5211](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5211/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3803](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3803/) completed with status: 'SUCCESS'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-18T20:14:11Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-18T20:21:08Z
body: [PR_Github #5627](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5627/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T04:03:37Z
body: [PR_Github #5627](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5627/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4111](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4111/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-19T15:50:40Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T16:01:21Z
body: [PR_Github #5747](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5747/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T17:48:33Z
body: [PR_Github #5747](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5747/) [ run ] completed with state `FAILURE`
[/LLM/main/L0_MergeRequest_PR pipeline #4203](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4203/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-19T18:27:27Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T18:33:42Z
body: [PR_Github #5764](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5764/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T22:10:08Z
body: [PR_Github #5764](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5764/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4218](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4218/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-19T22:22:47Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T22:31:21Z
body: [PR_Github #5780](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5780/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-19T22:38:48Z
body: @syuoni @Tracin @chang-l @tijyojwad Could you review a stacked PR (can't tag you there directly) that fills the remaining gaps? https://github.com/moraxu/TensorRT-LLM/pull/1

I figured it would be easier to merge it to this branch, given the existing acc references here.
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-19T22:39:00Z
body: /bot kill
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T22:44:35Z
body: [PR_Github #5781](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5781/) [ kill ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T22:44:37Z
body: [PR_Github #5780](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5780/) [ run ] completed with state `ABORTED`

author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-19T22:45:07Z
body: [PR_Github #5781](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5781/) [ kill ] completed with state `SUCCESS`
Successfully killed previous jobs for commit `2b33c83`

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-21T04:01:12Z
body: /bot run
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-21T06:49:40Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-21T06:54:54Z
body: [PR_Github #5975](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5975/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-21T10:26:19Z
body: [PR_Github #5975](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5975/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4369](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4369/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-21T15:43:19Z
body: /bot run
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-21T20:18:23Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-21T20:28:51Z
body: [PR_Github #6059](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6059/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-21T22:52:19Z
body: [PR_Github #6059](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6059/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4431](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4431/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-22T05:12:40Z
body: /bot run
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-23T00:35:54Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T00:41:26Z
body: [PR_Github #6190](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6190/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T03:53:25Z
body: [PR_Github #6190](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6190/) [ run ] completed with state `SUCCESS`
[/LLM/release-0.20/L0_MergeRequest_PR pipeline #47](https://nv/trt-llm-cicd/job/release-0.20/job/L0_MergeRequest_PR/47/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-23T05:57:34Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T06:02:58Z
body: [PR_Github #6248](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6248/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T12:37:07Z
body: [PR_Github #6248](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6248/) [ run ] completed with state `SUCCESS`
[/LLM/release-0.20/L0_MergeRequest_PR pipeline #53](https://nv/trt-llm-cicd/job/release-0.20/job/L0_MergeRequest_PR/53/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-23T16:33:31Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T16:38:57Z
body: [PR_Github #6323](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6323/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T18:18:46Z
body: [PR_Github #6323](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6323/) [ run ] completed with state `SUCCESS`
[/LLM/release-0.20/L0_MergeRequest_PR pipeline #61](https://nv/trt-llm-cicd/job/release-0.20/job/L0_MergeRequest_PR/61/) completed with status: 'SUCCESS'

author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-23T22:56:26Z
body: /bot reuse-pipeline
author_association: COLLABORATOR
user: login: moraxu
created_at: 2025-05-23T23:11:52Z
body: /bot reuse-pipeline
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T23:17:20Z
body: [PR_Github #6339](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6339/) [ reuse-pipeline ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T23:25:22Z
body: [PR_Github #6339](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6339/) [ reuse-pipeline ] completed with state `SUCCESS`
Reusing [PR_Github #6323](https://prod.blsm.nvidia.com/sw-tensorrt-top-1/job/LLM/job/helpers/job/PR_Github/6323/) for commit `17bdbcd`

author_association: COLLABORATOR