Repository: NVIDIA/TensorRT-LLM
Pull request #4574: feat: estimate GPU mem. usage w/ minimal KV cache
Author: login: ixlmar
Created: 2025-05-22T11:49:36Z | Merged: 2025-05-30T08:40:45Z

# feat: estimate GPU mem. usage w/ minimal KV cache

## Description

This improves the accuracy of the estimation of the GPU memory available for KV cache pages (cf. `free_gpu_memory_fraction` option) by performing the estimation with the smallest size possible initial KV cache.

~~**Note:** Commit b824e44394813ca17192c9ff3dfe8a52920f54ae is from pending PR #4497 and should no longer be part on this PR by the time it is merged.~~

## Test Coverage

<!--
Please list clearly what are the relevant test(s) that can safeguard the changes in the PR. This helps us to ensure we have sufficient test coverage for the PR.
-->

## GitHub Bot Help

`/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...`

Provide a user friendly way for developers to interact with a Jenkins server.

Run `/bot [-h|--help]` to print this help message.

See details below for each supported subcommand.

<details>

`run  [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]`

Launch build/test pipelines. All previously running jobs will be killed.

`--disable-fail-fast ` *(OPTIONAL)* : Disable fail fast on build/tests/infra failures.

`--skip-test ` *(OPTIONAL)* : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does **NOT** update GitHub check status.

`--stage-list "A10-1, xxx"` *(OPTIONAL)* : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does **NOT** update GitHub check status.

`--gpu-type "A30, H100_PCIe"` *(OPTIONAL)* : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does **NOT** update GitHub check status.

`--only-multi-gpu-test ` *(OPTIONAL)* : Only run the multi-GPU tests. Note: Does **NOT** update GitHub check status.

`--disable-multi-gpu-test ` *(OPTIONAL)* : Disable the multi-GPU tests. Note: Does **NOT** update GitHub check status.

`--add-multi-gpu-test ` *(OPTIONAL)* : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

`--extra-stage "H100_PCIe-[Post-Merge]-1, xxx"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

### kill

`kill  `

Kill all running builds associated with pull request.

### skip

`skip --comment COMMENT `

Skip testing for latest commit on pull request. `--comment "Reason for skipping build/test"` is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

### reuse-pipeline

`reuse-pipeline `

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

</details>



Diff:
diff --git a/tensorrt_llm/_torch/pyexecutor/_util.py b/tensorrt_llm/_torch/pyexecutor/_util.py
index 92b155df6399..dfb3c6eeb4bf 100644
--- a/tensorrt_llm/_torch/pyexecutor/_util.py
+++ b/tensorrt_llm/_torch/pyexecutor/_util.py
@@ -17,6 +17,7 @@
                                        load_torch_hf_lora)
 from tensorrt_llm.mapping import Mapping
 
+from ..model_config import ModelConfig
 from ..speculative import get_spec_decoder
 from .config_utils import is_mla, is_nemotron_hybrid
 from .kv_cache_transceiver import AttentionTypeCpp, create_kv_cache_transceiver
@@ -98,6 +99,8 @@ def cal_max_tokens(peak_memory, total_gpu_memory, fraction, model_config,
 
 def create_dummy_context_requests(max_num_tokens: int, max_seq_len: int,
                                   vocab_size: int):
+    # NB: The requests constructed here need to be compatible with
+    #     get_token_num_for_estimation.
     requests = []
     max_seq_len = min(max_num_tokens, max_seq_len)
     remaining_tokens = max_num_tokens
@@ -117,26 +120,23 @@ def create_dummy_context_requests(max_num_tokens: int, max_seq_len: int,
     return requests
 
 
-def get_token_num_for_estimation(executor_config, model_config):
+def get_token_num_for_estimation(executor_config: ExecutorConfig,
+                                 model_config: ModelConfig) -> int:
+    """Compute KV cache capacity required for estimate_max_kv_cache_tokens to succeed."""
     mapping = executor_config.mapping
-    if 'cp_type' not in mapping.cp_config:
-        end, _ = torch.cuda.mem_get_info()
-        fraction = get_fraction_from_executor_config(executor_config)
-        kv_size_per_token = get_cache_size_per_token(model_config, mapping)
-        max_tokens_limit = int(end * fraction // kv_size_per_token)
-        # When reusing KV cache blocks, we need to add extra tokens to account for partially filled blocks
-        # that cannot be reused. For each sequence of max_num_tokens length, we may need up to one extra
-        # block (tokens_per_block tokens) if the sequence length is not perfectly divisible by tokens_per_block.
-        # So we add math.ceil(max_num_tokens/max_seq_len) * tokens_per_block extra tokens.
-        return min(
-            max(
-                executor_config.max_batch_size, executor_config.max_num_tokens +
-                math.ceil(executor_config.max_num_tokens /
-                          executor_config.max_seq_len) *
-                executor_config.tokens_per_block, executor_config.max_seq_len),
-            max_tokens_limit)
-    else:
-        return None
+    if 'cp_type' in mapping.cp_config:
+        raise ValueError(
+            "KV cache size estimation not supported with context parallelism.")
+    # When reusing KV cache blocks, we need to add extra tokens to account for partially filled blocks
+    # that cannot be reused. Each sequence used during estimation (cf. estimate_max_kv_cache_tokens)
+    # has at most max_seq_len (or max_seq_len - 1) tokens. In total, the sequences used for estimation
+    # have max_num_tokens tokens (cf. create_dummy_context_requests). For each sequence, we may need up to one extra
+    # block (tokens_per_block tokens) if the sequence length is not perfectly divisible by tokens_per_block.
+    # So we add math.ceil(max_num_tokens/(max_seq_len-1)) * tokens_per_block extra tokens.
+    return (executor_config.max_num_tokens +
+            math.ceil(executor_config.max_num_tokens /
+                      (executor_config.max_seq_len - 1)) *
+            executor_config.tokens_per_block)
 
 
 def estimate_max_kv_cache_tokens(py_executor: PyExecutor,
@@ -223,97 +223,91 @@ def estimate_max_kv_cache_tokens(py_executor: PyExecutor,
 
 def create_kv_cache_manager(model_engine: PyTorchModelEngine, mapping: Mapping,
                             executor_config: ExecutorConfig) -> KVCacheManager:
-    kv_cache_manager = None
-    if executor_config.pytorch_backend_config.use_kv_cache:
+    assert executor_config.pytorch_backend_config.use_kv_cache, "Only construct KV cache when it is needed."
+
+    config = model_engine.model.model_config.pretrained_config
+    quant_config = model_engine.model.model_config.quant_config
+    spec_config = executor_config.speculative_config
+
+    hidden_size = config.hidden_size
+    num_attention_heads = config.num_attention_heads
+    num_key_value_heads = getattr(config, 'num_key_value_heads',
+                                  num_attention_heads)
+    head_dim = getattr(config, "head_dim", hidden_size // num_attention_heads)
+
+    if quant_config is not None and quant_config.quant_mode.has_fp8_kv_cache():
+        kv_cache_dtype = tensorrt_llm.bindings.DataType.FP8
+    else:
+        kv_cache_dtype = str_dtype_to_binding(
+            torch_dtype_to_str(model_engine.dtype))
+
+    num_hidden_layers = config.num_hidden_layers
+
+    if is_mla(config):
+        kv_cache_manager = KVCacheManager(
+            executor_config.kv_cache_config,
+            tensorrt_llm.bindings.internal.batch_manager.CacheType.SELFKONLY,
+            num_layers=num_hidden_layers,
+            num_kv_heads=1,
+            head_dim=config.kv_lora_rank + config.qk_rope_head_dim,
+            tokens_per_block=executor_config.tokens_per_block,
+            max_seq_len=executor_config.max_seq_len,
+            max_batch_size=executor_config.max_batch_size,
+            mapping=mapping,
+            dtype=kv_cache_dtype,
+            spec_config=spec_config,
+        )
+    elif is_nemotron_hybrid(config):
         config = model_engine.model.model_config.pretrained_config
-        quant_config = model_engine.model.model_config.quant_config
-        spec_config = executor_config.speculative_config
-
-        hidden_size = config.hidden_size
-        num_attention_heads = config.num_attention_heads
-        num_key_value_heads = getattr(config, 'num_key_value_heads',
-                                      num_attention_heads)
-        head_dim = getattr(config, "head_dim",
-                           hidden_size // num_attention_heads)
-
-        if quant_config is not None and quant_config.quant_mode.has_fp8_kv_cache(
-        ):
-            kv_cache_dtype = tensorrt_llm.bindings.DataType.FP8
-        else:
-            kv_cache_dtype = str_dtype_to_binding(
-                torch_dtype_to_str(model_engine.dtype))
-
-        num_hidden_layers = config.num_hidden_layers
-
-        if is_mla(config):
-            kv_cache_manager = KVCacheManager(
-                executor_config.kv_cache_config,
-                tensorrt_llm.bindings.internal.batch_manager.CacheType.
-                SELFKONLY,
-                num_layers=num_hidden_layers,
-                num_kv_heads=1,
-                head_dim=config.kv_lora_rank + config.qk_rope_head_dim,
-                tokens_per_block=executor_config.tokens_per_block,
-                max_seq_len=executor_config.max_seq_len,
-                max_batch_size=executor_config.max_batch_size,
-                mapping=mapping,
-                dtype=kv_cache_dtype,
-                spec_config=spec_config,
-            )
-        elif is_nemotron_hybrid(config):
-            config = model_engine.model.model_config.pretrained_config
-            num_layers = config.hybrid_override_pattern.count("*")
-            layer_mask = [
-                char == "*" for char in config.hybrid_override_pattern
-            ]
-            mamba_num_layers = config.hybrid_override_pattern.count("M")
-            mamba_layer_mask = [
-                char == "M" for char in config.hybrid_override_pattern
-            ]
-            kv_cache_manager = MambaHybridCacheManager(
-                # mamba cache parameters
-                config.hidden_size,
-                config.ssm_state_size,
-                config.conv_kernel,
-                config.expand,
-                config.n_groups,
-                config.mamba_head_dim,
-                mamba_num_layers,
-                mamba_layer_mask,
-                config.torch_dtype,
-                # kv cache parameters
-                executor_config.kv_cache_config,
-                tensorrt_llm.bindings.internal.batch_manager.CacheType.SELF,
-                num_layers=num_layers,
-                layer_mask=layer_mask,
-                num_kv_heads=num_key_value_heads,
-                head_dim=head_dim,
-                tokens_per_block=executor_config.tokens_per_block,
-                max_seq_len=executor_config.max_seq_len,
-                max_batch_size=executor_config.max_batch_size,
-                mapping=mapping,
-                dtype=kv_cache_dtype,
-                spec_config=spec_config,
-            )
-        else:
-            kv_cache_manager = KVCacheManager(
-                executor_config.kv_cache_config,
-                tensorrt_llm.bindings.internal.batch_manager.CacheType.SELF,
-                num_layers=num_hidden_layers,
-                num_kv_heads=num_key_value_heads,
-                head_dim=head_dim,
-                tokens_per_block=executor_config.tokens_per_block,
-                max_seq_len=executor_config.max_seq_len,
-                max_batch_size=executor_config.max_batch_size,
-                mapping=mapping,
-                dtype=kv_cache_dtype,
-                spec_config=spec_config,
-            )
-        # KVCacheManager (Non-draft) modifies the max_seq_len field, update it to executor_config
-        if model_engine.kv_cache_manager_key == KV_CACHE_MANAGER_KEY:
-            executor_config.max_seq_len = kv_cache_manager.max_seq_len
-
-    assert kv_cache_manager is not None
+        num_layers = config.hybrid_override_pattern.count("*")
+        layer_mask = [char == "*" for char in config.hybrid_override_pattern]
+        mamba_num_layers = config.hybrid_override_pattern.count("M")
+        mamba_layer_mask = [
+            char == "M" for char in config.hybrid_override_pattern
+        ]
+        kv_cache_manager = MambaHybridCacheManager(
+            # mamba cache parameters
+            config.hidden_size,
+            config.ssm_state_size,
+            config.conv_kernel,
+            config.expand,
+            config.n_groups,
+            config.mamba_head_dim,
+            mamba_num_layers,
+            mamba_layer_mask,
+            config.torch_dtype,
+            # kv cache parameters
+            executor_config.kv_cache_config,
+            tensorrt_llm.bindings.internal.batch_manager.CacheType.SELF,
+            num_layers=num_layers,
+            layer_mask=layer_mask,
+            num_kv_heads=num_key_value_heads,
+            head_dim=head_dim,
+            tokens_per_block=executor_config.tokens_per_block,
+            max_seq_len=executor_config.max_seq_len,
+            max_batch_size=executor_config.max_batch_size,
+            mapping=mapping,
+            dtype=kv_cache_dtype,
+            spec_config=spec_config,
+        )
+    else:
+        kv_cache_manager = KVCacheManager(
+            executor_config.kv_cache_config,
+            tensorrt_llm.bindings.internal.batch_manager.CacheType.SELF,
+            num_layers=num_hidden_layers,
+            num_kv_heads=num_key_value_heads,
+            head_dim=head_dim,
+            tokens_per_block=executor_config.tokens_per_block,
+            max_seq_len=executor_config.max_seq_len,
+            max_batch_size=executor_config.max_batch_size,
+            mapping=mapping,
+            dtype=kv_cache_dtype,
+            spec_config=spec_config,
+        )
+    # KVCacheManager (Non-draft) modifies the max_seq_len field, update it to executor_config
+    if model_engine.kv_cache_manager_key == KV_CACHE_MANAGER_KEY:
+        executor_config.max_seq_len = kv_cache_manager.max_seq_len
+
     return kv_cache_manager
 
 
diff --git a/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py b/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
index 3f5b8cffaf00..395107d84bbd 100644
--- a/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
+++ b/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
@@ -333,11 +333,16 @@ def create_py_executor(executor_config: ExecutorConfig,
             origin_seq_len, ctx_chunk_config, draft_model_engine)
         del py_executor  # free before constructing new
         del kv_cache_manager  # free before constructing new
+        del resources[KV_CACHE_MANAGER_KEY]
 
         executor_config.kv_cache_config.max_tokens = kv_cache_max_tokens
 
         with mem_monitor.observe_creation_stage(
                 _ExecutorCreationStage.KV_CACHE):
+            # Before estimating KV cache size, a minimal KV cache has been allocated using
+            # create_kv_cache_manager above, which caps executor_config.max_seq_len. Restoring
+            # the original value before creating the final KV cache.
+            executor_config.max_seq_len = max_seq_len
             kv_cache_manager = create_kv_cache_manager(model_engine, mapping,
                                                        executor_config)
             resources[KV_CACHE_MANAGER_KEY] = kv_cache_manager
@@ -350,6 +355,7 @@ def create_py_executor(executor_config: ExecutorConfig,
 
             if draft_model_engine is not None:
                 del draft_kv_cache_manager  # free before constructing new
+                del resources[DRAFT_KV_CACHE_MANAGER_KEY]
                 draft_kv_cache_manager = create_kv_cache_manager(
                     draft_model_engine, mapping, executor_config)
                 resources[DRAFT_KV_CACHE_MANAGER_KEY] = draft_kv_cache_manager


Reviews:
user: login: dcampora
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-30T08:14:12Z
user: login: ixlmar
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-30T08:25:06Z
user: login: dcampora
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-30T08:25:12Z

Review comments:
diff_hunk: @@ -223,97 +223,91 @@ def estimate_max_kv_cache_tokens(py_executor: PyExecutor,
 
 def create_kv_cache_manager(model_engine: PyTorchModelEngine, mapping: Mapping,
                             executor_config: ExecutorConfig) -> KVCacheManager:
-    kv_cache_manager = None
-    if executor_config.pytorch_backend_config.use_kv_cache:
+    assert executor_config.pytorch_backend_config.use_kv_cache, """Only construct KV cache when it is needed."""
path: tensorrt_llm/_torch/pyexecutor/_util.py
user: login: dcampora
body: Use single quotes here, no need for triple ones.
created_at: 2025-05-30T08:14:12Z
author_association: COLLABORATOR
diff_hunk: @@ -223,97 +223,91 @@ def estimate_max_kv_cache_tokens(py_executor: PyExecutor,
 
 def create_kv_cache_manager(model_engine: PyTorchModelEngine, mapping: Mapping,
                             executor_config: ExecutorConfig) -> KVCacheManager:
-    kv_cache_manager = None
-    if executor_config.pytorch_backend_config.use_kv_cache:
+    assert executor_config.pytorch_backend_config.use_kv_cache, """Only construct KV cache when it is needed."""
path: tensorrt_llm/_torch/pyexecutor/_util.py
user: login: ixlmar
body: Fixed in 4ea7495
created_at: 2025-05-30T08:25:06Z
author_association: COLLABORATOR

PR comments:
user: login: ixlmar
created_at: 2025-05-22T11:50:24Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-22T11:55:30Z
body: [PR_Github #6146](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6146/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-22T19:40:30Z
body: [PR_Github #6146](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6146/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4492](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4492/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-23T06:57:32Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T07:02:37Z
body: [PR_Github #6257](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6257/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T09:32:57Z
body: [PR_Github #6257](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6257/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4573](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4573/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-23T10:59:45Z
body: /bot run
author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-23T11:00:59Z
body: /bot kill
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T11:04:58Z
body: [PR_Github #6289](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6289/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T11:06:23Z
body: [PR_Github #6290](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6290/) [ kill ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T11:06:24Z
body: [PR_Github #6289](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6289/) [ run ] completed with state `ABORTED`

author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T11:06:54Z
body: [PR_Github #6290](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6290/) [ kill ] completed with state `SUCCESS`
Successfully killed previous jobs for commit `e10e7c7`

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-23T11:07:11Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T11:12:13Z
body: [PR_Github #6291](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6291/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T13:07:34Z
body: [PR_Github #6291](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6291/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4598](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4598/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-23T13:21:42Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T13:26:55Z
body: [PR_Github #6305](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6305/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T14:18:38Z
body: [PR_Github #6305](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6305/) [ run ] completed with state `FAILURE`
[/LLM/main/L0_MergeRequest_PR pipeline #4607](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4607/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-23T14:49:58Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T14:55:14Z
body: [PR_Github #6313](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6313/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-23T15:43:40Z
body: [PR_Github #6313](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6313/) [ run ] completed with state `FAILURE`
[/LLM/main/L0_MergeRequest_PR pipeline #4613](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4613/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-26T12:06:42Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-26T12:12:40Z
body: [PR_Github #6476](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6476/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-26T19:08:29Z
body: [PR_Github #6476](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6476/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4738](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4738/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-27T05:24:51Z
body: /bot run
author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-27T05:28:24Z
body: /bot kill
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-27T05:30:44Z
body: [PR_Github #6555](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6555/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-27T05:34:30Z
body: [PR_Github #6557](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6557/) [ kill ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-27T05:34:31Z
body: [PR_Github #6555](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6555/) [ run ] completed with state `ABORTED`

author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-27T05:35:01Z
body: [PR_Github #6557](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6557/) [ kill ] completed with state `SUCCESS`
Successfully killed previous jobs for commit `7800066`

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-27T07:15:13Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-27T07:21:18Z
body: [PR_Github #6589](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6589/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-27T21:06:11Z
body: [PR_Github #6589](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6589/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #4816](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/4816/) completed with status: 'SUCCESS'

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-28T08:22:41Z
body: /bot reuse-pipeline
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-28T08:40:44Z
body: [PR_Github #6740](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6740/) [ reuse-pipeline ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-28T08:47:02Z
body: [PR_Github #6740](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6740/) [ reuse-pipeline ] completed with state `SUCCESS`
Reusing [PR_Github #6589](https://prod.blsm.nvidia.com/sw-tensorrt-top-1/job/LLM/job/helpers/job/PR_Github/6589/) for commit `7228d5f`

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-28T10:04:41Z
body: /bot reuse-pipeline
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-28T10:23:15Z
body: [PR_Github #6761](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6761/) [ reuse-pipeline ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-28T10:33:45Z
body: [PR_Github #6761](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/6761/) [ reuse-pipeline ] completed with state `SUCCESS`
Reusing [PR_Github #6589](https://prod.blsm.nvidia.com/sw-tensorrt-top-1/job/LLM/job/helpers/job/PR_Github/6589/) for commit `c1f5a57`

author_association: COLLABORATOR
user: login: ixlmar
created_at: 2025-05-30T08:25:33Z
body: /bot reuse-pipeline
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-30T08:31:25Z
body: [PR_Github #7057](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/7057/) [ reuse-pipeline ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-30T08:37:55Z
body: [PR_Github #7057](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/7057/) [ reuse-pipeline ] completed with state `SUCCESS`
Reusing [PR_Github #6589](https://prod.blsm.nvidia.com/sw-tensorrt-top-1/job/LLM/job/helpers/job/PR_Github/6589/) for commit `4ea7495`

author_association: COLLABORATOR