TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators
Authors: jianling li, ShangZhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie, Jianrong Wang, Xu Han, Zhiyuan Liu, Maosong Sun
Accepted at: acl25

TRITONBENCH: Benchmarking Large Language Model Capabilities for Generating Triton Operators

Jianling Li1,*, Shangzhan Li2,*, Zhenye Gao3, Qi Shi4,† Yuxuan Li4,† Zefan Wang4, Jiacheng Huang4, Haojie Wang4, Jianrong Wang1, Xu Han4, Zhiyuan Liu4, Maosong Sun4
1 Tianjin University, Tianjin, China 2 Harbin Institute of Technology, Harbin, China
3 The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
4 Tsinghua University, Beijing, China
{jianlingl@tju,szli@ir.hit}.edu.cn, {qshi9510,yxuanl1995}@gmail.com

Abstract
Triton, a high-level Python-like language designed for building efficient GPU kernels, is widely adopted in deep learning frameworks due to its portability, flexibility, and accessibility. However, programming and parallel optimization still require considerable trial and error from Triton developers. Despite advances in large language models (LLMs) for conventional code generation, these models struggle to generate accurate, performance-optimized Triton code, as they lack awareness of its specifications and the complexities of GPU programming. More critically, there is an urgent need for systematic evaluations tailored to Triton. In this work, we introduce TRITONBENCH, the first comprehensive benchmark for Triton operator generation. TRITONBENCH features two evaluation channels: a curated set of 184 real-world operators from GitHub and a collection of 166 operators aligned with PyTorch interfaces. Unlike conventional code benchmarks prioritizing functional correctness, TRITONBENCH also profiles efficiency performance on widely deployed GPUs aligned with industry applications. Our study reveals that current state-of-the-art code LLMs struggle to generate efficient Triton operators, highlighting a significant gap in high-performance code generation. TRITONBENCH is available at https://github.com/thunlp/TritonBench.

1 Introduction
Triton (Tillet et al., 2019) language, a high-level Python-like programming language designed for implementing efficient GPU kernels, is playing an increasingly pivotal role in the ever-scaling deep learning ecosystems (Abadi et al., 2016; Paszke et al., 2019). Due to the superior portability, flexibility, lightweight design, and accessibility to less proficient programmers, Triton is prevalently adopted in modern Large Language Model (LLM) frameworks such as vLLM (Kwon et al., 2023), LightLLM (ModelTC, 2025), Liger-kernel (Hsu et al., 2024) and unsloth (Daniel Han and team, 2023). However, crafting high-performance operators remains challenging, especially for the intricate balance between memory hierarchy management, parallel thread coordination, and hardware-specific optimizations. Even though Triton abstracts away many complexities of low-level programming architectures like CUDA, it still requires developers to manually handle critical aspects such as pointer arithmetic and memory access patterns, making performance tuning a labor-intensive process that often involves extensive trial and error.

Current research in AI-assisted coding has reached a human-competitive level (Hui et al., 2024; Zhu et al., 2024), yet it is primarily restricted to general-purpose languages like Python. However, LLMs still face challenges in generating Domain Specific Language (DSL) code. Specifically for Triton, current models might be unfamiliar with Triton specification and the intricacies of GPU programming (Nichols et al., 2024). Most importantly, the ability of these models to produce high-quality Triton code remains unassessed. Therefore, a high-quality benchmark paired with performance-aware metrics is urgently required.

In this study, we present TRITONBENCH, a performance-aware benchmark framework for Triton generation, which contains two channels, namely TRITONBENCH-G and TRITONBENCH-T. Specifically, TRITONBENCH-G contains 184 carefully curated operators from existing GitHub repositories, reflecting the realistic demand for Triton operator development. As a complement, TRITONBENCH-T is composed of operator development tasks aligned with PyTorch interfaces, covering operators under-represented by public sources. Moreover, unlike the majority of code benchmarks merely prioritizing functional correctness (Chen et al., 2021; Austin et al., 2021a), TRI-
TONBENCH emphasizes efficiency performance profiling against reference programs on NVIDIA GPUs, better aligning industrial demands.

As shown in Figure 1, for TRITONBENCH-G, we follow three steps: 1) scrape and collect high-quality operators, 2) generate instructions via prompts, and 3) annotate test code with LLMs. Moreover, HPC experts evaluate GPU performance for all triton codes. For TRITONBENCH-T, we provide operator generation tasks aligned with PyTorch. To construct these tasks, we first perform a frequency analysis to select torch operators, combine them into diverse sets, and provide paired instructions and test code. Our evaluation metrics include similarity, call and execution accuracy, speed up, and GPU efficiency.

We conduct extensive experiments across a broad range of LLMs. Overall, the difficulty of TRITONBENCH-G is greater than that of TRITONBENCH-T. The highest execution accuracy on TRITONBENCH-G can reach 23.91%, while on TRITONBENCH-T, it can reach 53.01%. For all correctly executed operators generated by the models, the best speed up on TRITONBENCH-G is 1.56×, whereas, on TRITONBENCH-T, it is 1.91×. Additionally, we perform in-depth analyses of LLMs’ behavior on TRITONBENCH and summarize the challenges in Triton generation. The results reveal that current LLMs are not yet fully capable of handling TRITONBENCH, underscoring the challenge of enabling LLMs to generate Triton code effectively. We hope this work initiates evaluation in this under-explored area and fosters advancements in LLM-driven operator development.

![Illustration of the construction and evaluation of TRITONBENCH.](page_184_120_1077_312.png)

Figure 1: Illustration of the construction and evaluation of TRITONBENCH.

2 Related Work

2.1 Triton Development

Triton (Mitkov et al., 2021) is an open-source, Python-like language and a compiler designed to simplify GPU programming in AI and HPC. It abstracts the complexities of CUDA by introducing a block-based programming model, automating low-level optimizations such as memory coalescing and tensor core utilization, and making it more accessible to researchers without HPC background. Nonetheless, Triton provides explicit control over memory access patterns and parallelism. This balance of productivity and flexibility makes it prevalently adopted in both academia and industry (Kwon et al., 2023; ModeITC, 2025; Hsu et al., 2024; Daniel Han and team, 2023). However, Triton developers must still laboriously tune critical parameters to exploit hardware capabilities. LLM code generation poses prospects for automating Triton development, which calls for a systematic evaluation of generated operators.

2.2 Code Benchmarks

The demand for proper measurement of coding capability arises as the program synthesis research advances. The primary practice of coding benchmarks is functional correctness testing, usually realized by test case construction and sandbox execution. For example, HUMAN EVAL (Belz et al., 2021) curate hand-written programs and test cases, and MBPP (Austin et al., 2021b) create programming problems by crowd-sourcing. The functionality test has recently extended to automated test generation for better coverage (Liu et al., 2023) and broader applications, including software engineering (Jimenez et al., 2024). Another vital
aspect of coding benchmarking is performance profiling (Shypula et al.; Liu et al., 2023; Huang et al., 2024; Qiu et al., 2024). However, most existing frameworks focus on competition-style, single-process execution. While there are some frameworks for evaluating parallel programming on CPUs (Nichols et al., 2024; Chaturvedi et al., 2024), benchmarks targeting GPU code remain scarce. As the deployment of deep learning models scales up, a comprehensive evaluation framework that considers both correctness and performance on GPU code becomes increasingly necessary.

2.3 LLMs for Code Generation

LLMs have recently demonstrated impressive capabilities in generating code from natural language instructions, as evidenced by models such as DeepSeek-Coder (Guo et al., 2024; Zhu et al., 2024) and Qwen-Coder (Hui et al., 2024), which have achieved strong performance on broad coding benchmarks. Despite their versatility, they often struggle with Domain-Specific Languages (DSLs) designed for higher levels of abstraction and improved efficiency in targeted contexts (Wasowski and Berger, 2023). The main reason for this status is the limited availability of DSL datasets and benchmarks (Cassano et al., 2024; Pujar et al., 2023), coupled with their unique syntax and semantics (Pujar et al., 2023), posing significant challenges for LLMs (Buscemi, 2023). In this work, we focus on DSLs within the high-performance computing domain where the challenges we mentioned are more pronounced for involving the parallel programming model. We introduce the first comprehensive benchmark for Triton generation, providing a systematic evaluation framework that aims to guide future improvements in DSL-centric LLM code generation.

3 TRITONBENCH-G

Triton (Tillet et al., 2019) is a DSL that abstracts away low-level complexities to simplify GPU programming for computation-intensive tasks, with flexibility for specialized applications like machine learning. Typically, a Triton operator includes at least a kernel and a wrapper. The kernel comprises code executed on the GPU, focusing on tensor element addressing and thread parallel coordination. Meanwhile, the wrapper offers a Python function that encapsulates the kernel call. Figure 2 shows an example of Triton operator.

![Implementation of the Triton “add” operator. Lines 3-6 perform for tensor element addressing, followed by the calculation and storage in lines 7-10. The kernel is called in wrapper line 15.](page_1012_186_388_246.png)

Figure 2: Implementation of the Triton “add” operator. Lines 3-6 perform for tensor element addressing, followed by the calculation and storage in lines 7-10. The kernel is called in wrapper line 15.

We create TRITONBENCH-G by curating high-quality human-authored Triton operators from Github, which reflects Triton’s currently actual requirements. The following sections will explain data collection (§ 3.1), data statistics (§ 3.2), operator quality rating (§ 3.3), test code design (§ 3.4), and evaluation metrics (§ 3.5).

3.1 Data Collection

We begin by collecting Triton-related GitHub repositories with over 100 stars, yielding 95 repositories and a total of 845 Python files. As Triton repositories with higher star counts are rare, 100 stars serve as an optimal threshold, striking a balance between quality and quantity. To extract usable operator candidates, we apply the following preprocessing steps: 1) Remove Triton-irrelevant content from the collected files; 2) Complete missing imports or helper functions to ensure code executability; 3) Split files containing multiple independent Triton operators into separate samples.

Through this process, we curate 250 files containing clean Triton code snippets. Next, we perform a rigorous manual inspection to ensure code correctness: 4) For operators that consist solely of Triton kernels, we implement appropriate Python wrappers to make them callable; 5) We compile and debug each snippet manually to fix errors; 6) We write unit tests, and consult experts to fix functional bugs when necessary. Additionally, to ensure uniqueness, we leverage CODEBERTSCORE (Zhou et al., 2023) to eliminate duplicates.

Finally, we generate the LLM instruction for each operator based on prompt F. These instruc-
<table>
  <tr>
    <th rowspan="2">Difficulty</th>
    <th colspan="2">Instruction</th>
    <th colspan="3">Triton Operator</th>
  </tr>
  <tr>
    <th>tok#</th>
    <th>func#</th>
    <th>params#</th>
    <th>line#</th>
    <th>tok#</th>
  </tr>
  <tr>
    <td>d1 (1.6%)</td>
    <td>296.67</td>
    <td>2.00</td>
    <td>1.33</td>
    <td>26.00</td>
    <td>369.0</td>
  </tr>
  <tr>
    <td>d2 (14.7%)</td>
    <td>363.26</td>
    <td>2.41</td>
    <td>2.70</td>
    <td>45.56</td>
    <td>678.1</td>
  </tr>
  <tr>
    <td>d3 (35.3%)</td>
    <td>353.80</td>
    <td>3.80</td>
    <td>3.34</td>
    <td>102.42</td>
    <td>1510.4</td>
  </tr>
  <tr>
    <td>d4 (45.7%)</td>
    <td>394.48</td>
    <td>3.89</td>
    <td>6.04</td>
    <td>153.77</td>
    <td>2689.1</td>
  </tr>
  <tr>
    <td>d5 (2.7%)</td>
    <td>469.60</td>
    <td>6.60</td>
    <td>6.00</td>
    <td>249.80</td>
    <td>4581.4</td>
  </tr>
</table>

Table 1: Statistics of TRITONBENCH-G.

tions provide essential details, including the operator’s functionality, corresponding function names, and a comprehensive input/output demonstration. All instructions are carefully reviewed and manually verified to ensure they correctly reflect the intended behavior of each operator.

3.2 Data Statistics

Table 1 summarizes statistics of TRITONBENCH-G. In this benchmark, each operator is assigned a difficulty level, from d1 (easiest) to d5 (most challenging), by an LLM guided by prompt F, with subsequent manual verification by two domain experts. For each difficulty level, we report statistics including the average number of functions (func#), parameters (params#), lines (lines#), and tokens (tok#). Notably, the upward trend observed in these statistics as the difficulty level increases suggests the expert-driven grading scheme is largely reasonable.

Compared to existing code generation tasks (Chen et al., 2021; Austin et al., 2021a), the average instruction length in TRITONBENCH-G is substantially longer, which is a deliberate design decision. The extended instructions provide richer context, which can help the model understand nuanced requirements and generate high-quality operators. Additionally, this approach better reflects real-world operator development practices where detailed requirements are indispensable.

3.3 Operators Quality Rating

To systematically evaluate the quality of the Triton operators in TRITONBENCH-G, we compute the GPU efficiency for each operator. Detailed methodology for calculating GPU efficiency can be found in Appendix B. Our statistics indicate an average GPU efficiency of 43.0%, which reflects the overall reliability of the operators in TRITONBENCH-G. The distribution of efficiency scores is shown in Figure 3. As shown in the figure, 19.6% of operators developed by professional Triton programmers have GPU performance below 10%, which underscores the challenges in developing and optimizing Triton operators.

Figure 3: Distribution of GPU efficiency of the Triton operators in TRITONBENCH-G.

3.4 Test Code Design

In contrast to traditional CPU-language benchmarks (Shypula et al.; Liu et al., 2023; Huang et al., 2024; Qiu et al., 2024) that predominantly rely on scalar test inputs, TRITONBENCH-G is built around tensor-based test inputs. We employ PyTorch to generate random tensors as replacements for conventional test cases. Specifically, we leverage a prompt F to generate the corresponding test code for each operator. In the case of the multi-branch operators, the generated test code is designed to invoke every necessary branch within the operator. Moreover, we rigorously debug all branches to guarantee test reliability. On average, we generate 3.6 test branches per operator.

3.5 Evaluation Metrics

In contrast to traditional code evaluations, which mainly emphasize accuracy (Chen et al., 2021; Austin et al., 2021a), our TRITONBENCH-G introduces dedicated performance evaluations. Specifically, the systematic evaluation of Triton operators covers five key metrics:

Similarity assesses text-level resemblance using CODEBLEU (Ren et al., 2020). In our experiments, we assign equal weights of 0.25 to N-gram, weighted N-gram, syntax, and dataflow components to ensure a balanced evaluation.

Call & Execution Accuracy assess whether the code can run without error and whether its input-output behavior is correct, respectively.

Speed Up measures the relative execution time improvement for correctly executed operators. Specifically, if \( t_{gen} \) and \( t_{ref} \) represent the running times of the generated and reference operators, respectively, then \( \text{SpeedUp}(gen) = \frac{t_{ref}}{t_{gen}} \).
GPU Efficiency evaluates how effectively the generated operator utilizes GPU resources, following the operator quality rating in § 3.3. For further details, please refer to Appendix B.

4 TRITONBENCH-T

The real-world Triton operators introduced in § 3 primarily focus on highly frequent operations. As a complement, we propose TRITONBENCH-T, which aligns the Triton wrapper with interfaces of the PyTorch library (Paszke et al., 2019). Together, TRITONBENCH-G and TRITONBENCH-T form a complementary evaluation framework. The following sections elaborate on the data construction (§ 4.1), data statistics (§ 4.2), test code and metrics (§ 4.3), and benchmark comparisons (§ 4.4).

4.1 Data Construction

We construct TRITONBENCH-T by selecting PyTorch operators based on their usage frequency in real-world coding and then fusing them (hereafter referred to simply as “operators”). First, we select operators that require GPU interactions, ensuring alignment with Triton’s scope. Next, we sample 40 high-frequency operators and 40 low-frequency operators from the remaining pool. The frequency of each operator is determined by its usage probability in PyTorch-related code from The Stack V2 (Lozhkov et al., 2024) with those exceeding a predefined threshold 45% as common operators.

Subsequently, we fuse these operators in various configurations: combinations of common operators, combinations of common and uncommon operators, and combinations of uncommon operators. We provide an example of the selection and fusion process in Appendix C. All combinations are valid, as the outputs of preceding operators serve as appropriate inputs for subsequent ones. The final set includes 166 operators, based on the latest (v2.6.0) version of the PyTorch library. Each operator is paired with its corresponding standard PyTorch call and document, while fused operators combine descriptions from all involved operators.

4.2 Data Statistic

The statistics of TRITONBENCH-T are presented in Table 2. Similar to TRITONBENCH-G, the operators are categorized into five difficulty levels (d1 to d5) using an LLM guided by prompt F. These initial categorizations are then validated through manual review by two domain experts.

<table>
  <tr>
    <th rowspan="2">Difficulty</th>
    <th colspan="4">Torch-Align Operator</th>
  </tr>
  <tr>
    <th>torch-op#</th>
    <th>params#</th>
    <th>math#</th>
    <th>description#</th>
  </tr>
  <tr>
    <td>d1 (13.3%)</td>
    <td>1.36</td>
    <td>2.82</td>
    <td>23.50</td>
    <td>50.41</td>
  </tr>
  <tr>
    <td>d2 (22.3%)</td>
    <td>1.97</td>
    <td>3.78</td>
    <td>40.73</td>
    <td>61.19</td>
  </tr>
  <tr>
    <td>d3 (32.5%)</td>
    <td>2.70</td>
    <td>4.91</td>
    <td>74.64</td>
    <td>67.89</td>
  </tr>
  <tr>
    <td>d4 (29.5%)</td>
    <td>2.16</td>
    <td>5.24</td>
    <td>47.31</td>
    <td>71.02</td>
  </tr>
  <tr>
    <td>d5 (2.4%)</td>
    <td>2.75</td>
    <td>2.75</td>
    <td>30.50</td>
    <td>88.50</td>
  </tr>
</table>

Table 2: Statistics of TRITONBENCH-T.

We report the following statistics: (1)torch-op# the average number of PyTorch operators, (2)params# the average number of parameters, (3)math#, the average token number of mathematical expressions, and (4)description#, the average token count of the descriptions. These statistics generally increase with the operator difficulty, similar trend that aligns with the observations in TRITONBENCH-G.

4.3 Test Code and Metrics

The design of the test code in TRITONBENCH-T adheres to those of TRITONBENCH-G, employing randomly generated tensors for operator evaluation. For correctness and performance assessment, we utilize Call Accuracy, Execution Accuracy, and Speed Up, whose computation methods are consistent with those used in TRITONBENCH-G.

4.4 Benchmark Comparison

This section provides comparisons between TRITONBENCH-G and TRITONBENCH-T, which differ in key aspects and together provide a well-rounded evaluation.

Source & Distribution: TRITONBENCH-G is collected from GitHub and reflects real-world programming demands with a concentration of frequently used operators, e.g., Attention at 20.0%, MatMul at 10.9%, LayerNorm at 6.5%, SoftMax at 3.8%. In contrast, TRITONBENCH-T, sourced from PyTorch, presents a more diverse operator set including both common and uncommon operators.

Instruction Generation: TRITONBENCH-G combines LLM generation with expert verification while TRITONBENCH-T directly extracts instructions from PyTorch documentation. This difference underlines their complementary roles in probing different facets of the Triton generation.

Evaluation Metrics: Both benchmark channels assess correctness and performance. Additionally, TRITONBENCH-G incorporates a similarity-
<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>Similarity</th>
    <th colspan="2">Call Accuracy</th>
    <th>Execution Accuracy</th>
    <th>Speed Up</th>
    <th>GPU Efficiency</th>
  </tr>
  <tr>
    <th colspan="8">Domain-Specific Models</th>
  </tr>
  <tr>
    <td>Qwen2.5-Coder</td>
    <td>7B</td>
    <td>9.19 / 14.54</td>
    <td>0.54 / 0.54</td>
    <td>0.54 / 0.54</td>
    <td>1.11 / 0.99</td>
    <td>47.87 / 51.26</td>
  </tr>
  <tr>
    <td>DeepSeek-Coder</td>
    <td>6.7B</td>
    <td>9.38 / 14.52</td>
    <td>0.54 / 5.98</td>
    <td>0.54 / 5.98</td>
    <td>0.97 / 0.94</td>
    <td>45.89 / 53.77</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-sft</td>
    <td>7B</td>
    <td><b>29.98</b> / 25.96</td>
    <td>4.89 / 10.87</td>
    <td>4.89 / 10.87</td>
    <td><b>1.56</b> / <b>1.22</b></td>
    <td><b>51.71</b> / 46.70</td>
  </tr>
  <tr>
    <td>DeepSeek-Coder-sft</td>
    <td>6.7B</td>
    <td>25.52 / <b>30.34</b></td>
    <td><b>9.78</b> / <b>11.96</b></td>
    <td><b>9.78</b> / <b>11.96</b></td>
    <td>1.03 / 1.11</td>
    <td>47.68 / 42.26</td>
  </tr>
  <tr>
    <th colspan="8">General-Purpose Models</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>9.87 / 20.67</td>
    <td>10.87 / 17.93</td>
    <td>10.33 / 16.84</td>
    <td>0.97 / 1.19</td>
    <td>48.80 / <b>53.33</b></td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>-</td>
    <td>12.46 / 22.48</td>
    <td>10.33 / 20.11</td>
    <td>9.79 / 19.57</td>
    <td>0.90 / <b>1.54</b></td>
    <td><b>59.31</b> / 49.32</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>72B</td>
    <td>14.86 / 26.25</td>
    <td>11.41 / 16.85</td>
    <td>10.87 / 16.31</td>
    <td>0.96 / 1.19</td>
    <td>23.28 / 49.40</td>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>685B</td>
    <td><b>19.96</b> / 22.64</td>
    <td>13.59 / 22.83</td>
    <td>13.05 / 22.83</td>
    <td><b>1.11</b> / <b>1.22</b></td>
    <td>44.83 / 46.70</td>
  </tr>
  <tr>
    <td>GPT-o1</td>
    <td>-</td>
    <td>16.58 / <b>29.70</b></td>
    <td><b>15.22</b> / <b>23.91</b></td>
    <td><b>14.23</b> / <b>23.91</b></td>
    <td>0.92 / 1.14</td>
    <td>54.25 / 46.37</td>
  </tr>
</table>

Table 3: Main results of TRITONBENCH-G across baseline models, where the left side of “/” represents the zero-shot results and the right side represents the one-shot results.

based assessment that offers direct comparisons with established implementations. In summary, the different designs of TRITONBENCH-G and TRITONBENCH-T enable a comprehensive and nuanced evaluation of Triton operator generation.

5 Experiments

We conduct an extensive set of experiments on TRITONBENCH to rigorously evaluate the performance and capabilities of current LLMs.

5.1 Baselines and Setup

TRITONBENCH generally requires strong capabilities in code generation. Therefore, we select state-of-the-art LLMs that excel in programming tasks as baselines, including both specialized open-source models and general-purpose models. For specialized open-source models, we choose Qwen2.5-Coder-7B-Instruct (Hui et al., 2024) and deepseek-coder-6.7b-instruct (Guo et al., 2024). For general-purpose models, we include Claude-3.5-Sonnet-06201, GPT-4o-08062, qwen2.5-72B-Instruct (Yang et al., 2024), as well as the thought-driven models DeepSeek-R1 (Guo et al., 2025) and GPT-o1-2024-12-173.

In our experiments, all general-purpose models are deployed for direct inference. In contrast, domain-specific models undergo an additional supervised fine-tuning phase. Details of the training corpus can be found in § A. For evaluation, we consider both zero-shot and one-shot scenarios. In the one-shot setting, a BM25-based retrieval method (Robertson et al., 2009) is utilized to select the most relevant prompt from the training corpus.

5.2 Main results of TRITONBENCH-G

Table 3 illustrates the performances of baselines on TRITONBENCH-G. It is evident that domain-specific models generally underperform compared to general-purpose models. However, fine-tuning 7B domain-specific models with domain data significantly boosts accuracy. Qwen’s accuracy rises from 0 to 4.89%, and DeepSeek’s from 0 to 9.78% in zero-shot settings, with even more pronounced enhancements in one-shot settings due to the retrieval data from the same source as TRITONBENCH-G. The observed increase in Speed Up can be attributed to the relative simplicity of the correctly generated operators, which makes it easier for LLMs to produce efficient code. The high GPU efficiency shares the similar reasons.

General-purpose models, particularly DeepSeek-R1 and GPT-o1, excel across all metrics. Under one-shot conditions, DeepSeek-R1 achieves 22.83% in Call and Execution Accuracy, while GPT-o1 reaches 23.91%. The roughly 10% improvement from zero-shot to one-shot highlights the critical role of high-quality examples for Triton generation. Furthermore, the close alignment between Call Accuracy and Execution Accuracy indicates that only a few operators fail to produce correct results despite successfully invoked.

DeepSeek-R1 also leads in GPU execution times, with an improvement of 1.11× in zero-shot and 1.22× in one-shot settings. While GPU efficiency is strong across most models, Qwen2.5-72B exhibits lower efficiency in zero-shot settings, likely due to a higher proportion of less efficient operators. Finally, Similarity provides corroborative

1https://www.anthropic.com/news/clause-3-5-sonnet
2https://openai.com/index/hello-gpt-4o
3https://openai.com/o1
<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>Call Accuracy</th>
    <th>Execution Accuracy</th>
    <th>Speed Up</th>
  </tr>
  <tr>
    <th colspan="5">Domain-Specific Models</th>
  </tr>
  <tr>
    <td>Qwen2.5-Coder</td>
    <td>7B</td>
    <td>4.22 / 4.22</td>
    <td>4.22 / 4.22</td>
    <td><b>1.15</b> / 1.00</td>
  </tr>
  <tr>
    <td>DeepSeek-Coder</td>
    <td>6.7B</td>
    <td>8.43 / 3.61</td>
    <td>8.43 / 3.61</td>
    <td>0.74 / <b>1.02</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder-sft</td>
    <td>7B</td>
    <td>17.47 / 16.27</td>
    <td>17.47 / 15.67</td>
    <td>0.98 / 0.92</td>
  </tr>
  <tr>
    <td>DeepSeek-Coder-sft</td>
    <td>6.7B</td>
    <td><b>19.28</b> / <b>18.67</b></td>
    <td><b>19.28</b> / <b>16.26</b></td>
    <td>0.91 / 0.85</td>
  </tr>
  <tr>
    <th colspan="5">General-Purpose Models</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>36.75 / 32.53</td>
    <td>36.75 / 32.53</td>
    <td>0.98 / 0.94</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>-</td>
    <td>29.52 / 37.95</td>
    <td>29.52 / 33.70</td>
    <td>0.93 / 0.89</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>72B</td>
    <td>30.12 / 22.89</td>
    <td>30.12 / 16.30</td>
    <td>1.07 / 0.92</td>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>685B</td>
    <td><b>53.01</b> / <b>45.78</b></td>
    <td><b>53.01</b> / <b>45.78</b></td>
    <td>1.03 / <b>1.91</b></td>
  </tr>
  <tr>
    <td>GPT-o1</td>
    <td>-</td>
    <td>32.53 / 43.37</td>
    <td>32.53 / 43.37</td>
    <td><b>1.21</b> / 1.10</td>
  </tr>
</table>

Table 4: Main results of TRITONBENCH-T across baseline models, where the left side of “/” represents the zero-shot results and the right side represents the one-shot results.

insights, as its variations mirror trends observed in other metrics.

5.3 Main Results of TRITONBENCH-T

From Table 4, we can observe that domain-specific models generally underperform general-purpose models. Nonetheless, fine-tuning with an 8k corpus considerably improves their performance. For instance, Qwen’s zero-shot Execution Accuracy rises from 0 to 17.47%. In contrast, its one-shot improvement (15.67%) is slightly lower, likely due to the fact that the retrieved prompts and TRITONBENCH-T operators come from different sources (Github vs. Pytorch).

Among general-purpose models, DeepSeek-R1 demonstrates the strongest overall performance, achieving 53.01% Call and Execution Accuracy in the zero-shot setting. Although its accuracy drops by 7.23% in the one-shot setting, it still slightly surpasses GPT-o1. As for Speed Up, DeepSeek-R1 achieves the best performance of 1.91× improvements. Most performance improvements in successfully executed operators stem from operator fusion. Triton’s fused operators reduce redundant memory reads and writes compared to PyTorch, enhancing memory bandwidth utilization and boosting performance.

Overall, most models achieve better performance on TRITONBENCH-T than to TRITONBENCH-G, likely because TRITONBENCH-T features a more balanced distribution of operator difficulty, whereas TRITONBENCH-G is predominantly composed of higher-difficulty operators, namely, d3 and d4.

6 Analysis

In this section, we examine the distribution of correct and incorrect operators across difficulty levels (d1–d5) for the top-performing models, DeepSeek-R1 and GPT-o1, as shown in Figure 4 and Figure 5. Additionally, we analyze the error patterns of incorrect operators and summarize the main challenges for each benchmark as detailed in Table 3 and Table 4. The zero-shot and one-shot settings are annotated as 0 and 1 respectively.

6.1 Challenges for TRITONBENCH-G

Figure 4 clearly shows that most operators are generated incorrectly. Both DeepSeek-R1 and GPT-o1 exhibit similar trends, with DeepSeek-R1 outperforming GPT-o1. Notably, when moving from the zero-shot to the one-shot setting, both models achieve significant improvements on d4. These improvements may stem from the prevalence of Attention and Softmax operators in d4, enabling models to leverage similar examples. In contrast, the simpler operators in d2 and d3 show only limited gains in the one-shot setting, likely due to the smaller, more idiosyncratic nature of these datasets that leads to lower retrieval similarity.

For the incorrectly written operators, we classify the 16 error types into 4 major categories, detailed in Appendix D which is presented in Table 5. Note that only compiler-reported errors were considered. The results show that, compared to the zero-shot setting, both DeepSeek-R1 and GPT-o1 in the one-shot setting demonstrate a significant increase in Syntax and Name&Ref errors but a reduction in Attr&Type and Run&Logic errors. This trend suggests that the training corpus may provide helpful guidance on logical structure and Triton specifications, thus enhancing overall accuracy. Furthermore, error sensitivity differs between models: DeepSeek-R1 is less susceptible to syntax errors, whereas GPT-o1 handles logical errors better.
(a) DeepSeek-R1^0 results.   (b) DeepSeek-R1^1 results.

(c) GPT-o1^0 results.   (d) GPT-o1^1 results.

Figure 4: Execution results distribution across difficulty levels in TRITONBENCH-G.

<table>
  <tr>
    <th>Model</th>
    <th>Syntax</th>
    <th>Attr&Type</th>
    <th>Name&Ref</th>
    <th>Run&Logc</th>
  </tr>
  <tr>
    <td>DeepSeek-R1^0</td>
    <td>1.64</td>
    <td>42.62</td>
    <td>16.39</td>
    <td>39.34</td>
  </tr>
  <tr>
    <td>DeepSeek-R1^1</td>
    <td>9.27</td>
    <td>33.11</td>
    <td>35.76</td>
    <td>21.85</td>
  </tr>
  <tr>
    <td>GPT-o1^0</td>
    <td>10.3</td>
    <td>38.18</td>
    <td>28.48</td>
    <td>23.03</td>
  </tr>
  <tr>
    <td>GPT-o1^1</td>
    <td>20.83</td>
    <td>24.31</td>
    <td>43.06</td>
    <td>11.81</td>
  </tr>
</table>

Table 5: Error statistics of execution failures in TRITONBENCH-G.

(a) DeepSeek-R1^0 results.   (b) DeepSeek-R1^1 results.

(c) GPT-o1^0 results.   (d) GPT-o1^1 results.

Figure 5: Execution results distribution across difficulty levels in TRITONBENCH-T.

<table>
  <tr>
    <th>Model</th>
    <th>Syntax</th>
    <th>Attr&Type</th>
    <th>Name&Ref</th>
    <th>Run&Logc</th>
  </tr>
  <tr>
    <td>DeepSeek-R1^0</td>
    <td>0.00</td>
    <td>31.96</td>
    <td>14.43</td>
    <td>53.61</td>
  </tr>
  <tr>
    <td>DeepSeek-R1^1</td>
    <td>0.00</td>
    <td>36.79</td>
    <td>20.75</td>
    <td>42.45</td>
  </tr>
  <tr>
    <td>GPT-o1^0</td>
    <td>24.06</td>
    <td>26.32</td>
    <td>7.52</td>
    <td>42.11</td>
  </tr>
  <tr>
    <td>GPT-o1^1</td>
    <td>25.25</td>
    <td>25.25</td>
    <td>22.22</td>
    <td>27.27</td>
  </tr>
</table>

Table 6: Error statistics of execution failures in TRITONBENCH-T.

6.2 Challenges for TRITONBENCH-T

The execution results of TRITONBENCH-T (Figure 5) show the percentages of correctly generated operators. we can observe that DeepSeek-R1 generated more correct than incorrect operators, which proves the point that the difficulty distributions in TRITONBENCH-T are smoother than TRITONBENCH-G.

However, while DeepSeek-R1’s performance declines for difficulty d2-d4 in the one-shot setting, GPT-o1 shows improved accuracy on these subsets. This finding indicates that GPT-o1 might be more adept at logical reasoning for Triton generation tasks, allowing it to efficiently use the provided sample. The differing trends also imply that sample operators affect models in diverse ways.

For execution error statistics in TRITONBENCH-T (Table 6), DeepSeek-R1 notably avoids Syntax errors entirely, while GPT-o1 maintains a high rate of such errors. Under the one-shot setting, DeepSeek-R1 shows a rise in Attr&Type and Name&Ref errors alongside a decline in Run&Logc Errors. Conversely, GPT-o1 experiences a significant increase in Name&Ref errors with a notable drop in Run&Logc errors. Comparing TRITONBENCH-G and TRITONBENCH-T, the one-shot setting consistently reduces Run&Logc errors. These variations in error patterns likely stem from the mixed influence of useful and irrelevant information in the provided samples.

7 Conclusion

In this work, we present TRITONBENCH, a dual-channel benchmark specifically designed for evaluating LLMs’ generation for Triton operators. TRITONBENCH-G integrates real-world Triton operator samples from open repositories, while TRITONBENCH-T introduces complementary tasks that align with PyTorch interfaces. Our evaluation framework addresses both functional accuracy and the performance on NVIDIA GPUs. We also conduct extensive experiments and detailed analysis on our benchmark, and find that current LLMs struggle to generate high-quality Triton operators, un-
derscoring the necessity for further advancement in generating accurate as well as performance-aware Triton code. We anticipate TRITONBENCH will serve as an essential framework for advancing automated operator generation for Triton.

Limitations

This study initially focused on evaluations conducted on NVIDIA A100 GPUs due to their prevalence in deployment. Following reviewer suggestions, we have added experiments on H100 to reflect performance on newer hardware (refer to Appendix E). Broader hardware diversity remains a promising direction for future work.

Ethics Statement

This work adheres to ethical research practices and poses no potential risks. All code data used in TRITONBENCH are sourced exclusively from publicly available resources, including GitHub repositories and PyTorch documentations, ensuring no privacy concerns.

Acknowledgement

The work is initiated and supported by AI9Stars Team. We are grateful for the support of the OpenBMB and InfiniteTensor teams.

References

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. {TensorFlow}: a system for {Large-Scale} machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021a. Program synthesis with large language models. ArXiv preprint, abs/2108.07732.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021b. Program synthesis with large language models. ArXiv preprint, abs/2108.07732.

Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, and Anastasia Shimorina, editors. 2021. Proceedings of the Workshop on Human Evaluation of NLP Systems (HunEval). Association for Computational Linguistics, Online.

Alessio Buscemi. 2023. A comparative study of code generation using chatgpt 3.5 across 10 programming languages. ArXiv preprint, abs/2308.04477.

Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Carolyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. 2024. Knowledge transfer from high-resource to low-resource programming languages for code llms. Proceedings of the ACM on Programming Languages, 8(OOPSLA2):677–708.

Aman Chaturvedi, Daniel Nichols, Siddharth Singh, and Abhinav Bhatele. 2024. Hpc-coder-v2: Studying code llms across low-resource parallel languages. ArXiv preprint, abs/2412.15178.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374.

Michael Han Daniel Han and Unslot team. 2023. Unslot.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shiron Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv preprint, abs/2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. CoRR.

Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger kernel: Efficient triton kernels for llm training. ArXiv preprint, abs/2410.10989.

Dong Huang, Weiyi Shang, Yuhao Qing, Heming Cui, and Jie M Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code. ArXiv preprint, abs/2402.02037.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Owen2. 5-coder technical report. ArXiv preprint, abs/2409.12186.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E.
Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Xuyang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. ArXiv preprint, abs/2402.19173.

Ruslan Mitkov, Vilemmini Sosoni, Julie Christine Giguière, Elena Murgolo, and Elizabeth Deysel, editors. 2021. Proceedings of the Translation and Interpreting Technology Online Conference. INCOMA Ltd., Held Online.

ModelTC. 2025. Lightllm: A python-based llm inference and serving framework. https://github.com/ModelTC/lightllm.

Daniel Nichols, Joshua H Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. 2024. Can large language models write parallel code? In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, pages 281–294.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024–8035.

Saurabh Pujar, Luca Buratti, Xiaojie Guo, Nicolas Dupuis, Burn Lewis, Sahil Suneja, Atin Sood, Ganesh Nalawade, Matt Jones, Alessandro Morari, et al. 2023. Automated code generation for information technology tasks in yaml through large language models. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1–4. IEEE.

Ruizhong Qiu, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. 2024. How efficient is llm-generated code? a rigorous & high-standard benchmark. ArXiv preprint, abs/2406.06647.

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. ArXiv preprint, abs/2009.10297.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.

Alexander G Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, et al. Learning performance-improving code edits. In The Twelfth International Conference on Learning Representations.

Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19.

Andrzej Wałoszynski and Thorsten Berger. 2023. Domain-Specific Languages. Springer.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Owen2. 5 technical report. ArXiv preprint, abs/2412.15115.

Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. CodeBERTScore: Evaluating code generation with pretrained models of code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13921–13937, Singapore. Association for Computational Linguistics.

Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. ArXiv preprint, abs/2406.11931.

A Training Corpus

The training corpus for supervised fine-tuning comprises two distinct components: real-world data sourced from GitHub and synthetically generated data produced through compiler operations.

The real-world data component incorporates Triton code extracted from GitHub repositories, which undergoes basic cleaning procedures as outlined in prompt F, undergoes a debugging process that is less rigorous than the methodology applied to
Figure 6: The workflow of operator performance evaluation

TRITONBENCH-G. To prevent potential data leakage and ensure benchmark integrity, we systematically eliminate samples exhibiting high similarity to TRITONBENCH-G entries using the CODE-BERTSCORE similarity metric (Zhou et al., 2023).

The synthetic data component is generated using Ninetoohed4, a domain-specific language built upon Triton that offers enhanced abstraction capabilities. This framework facilitates the automated synthesis of valid Triton code through the processing of well-formed expressions. Each part of data containing \(4K\) samples. This combined corpus serves as the foundational training dataset for experimental models in one-shot learning settings. For all experiments, the fine-tuning process is carried out over 3 epochs with a learning rate of \(5e-5\).

B Operator Performance Evaluation

For operator performance evaluation, we refer primarily to the official examples provided by Triton5. We provide evaluation scripts for each operator in TRITONBENCH-G. Figure 6 illustrates the workflow of our operator performance evaluation.

First, we define a set of tensors with increasing dimensions based on the characteristics of the operator. Next, each tensor is sequentially fed into the operator for execution. During each execution, we use the expert annotations for each operator to determine the total memory bandwidth (Bytes) and the total number of floating-point operations (Flops) based on the input tensors. More importantly, we use the triton.testing.do_bench method from the official Triton library6 to measure the operator’s execution time on the GPU. Specifically, we gradually increase the warm-up time and repetition time until the measured execution time stabilized, which means that most operators are run hundreds of thousands of times to ensure that the running time is measured accurately. After obtaining the execution time, we calculate the operator’s performance metrics by dividing the total memory bandwidth and the total floating-point operations by the execution time to obtain throughput in GB/s and Tflops, respectively. We then calculate the GPU efficiency by calculating the ratio of the measured performance metrics (GB/s and Tflops) to the theoretical maximum performance of the NVIDIA A100 Tensor Core GPU. Repetition of the above process for tensors of increasing sizes obtains the performance metrics for each execution, which collectively form the operator performance report. We adopt the peak GPU efficiency from the performance report as the final measure of the operator’s quality.

By following the evaluation workflow described above, we generate a detailed performance report for each operator in TRITONBENCH-G. Figure 7 illustrates the performance curves of several common operators. As the input dimensions increase, as can be seen from the figure, the GB/s or Tflops of the operators show an upward trend, eventually stabilizing. This suggests that the performance of the operator reaches a bottleneck beyond a certain scale, and further increases in input size result in diminishing returns in performance, aligning with the expected trend of operator performance.

C Torch Operator Selection and Fusion

To address concerns regarding the clarity of operator extraction and composition in TRITONBENCH-T, we detail our two-stage pipeline below:

1. Selection: We begin by selecting 40 common and 40 uncommon PyTorch operators. The selection is based on their frequency from the Stack V2 dataset to ensure diversity across usage patterns.

2. Combination: With the assistance of large language models (LLMs), human experts manually combine the selected operators into valid compositions. Each composition ensures that the outputs of preceding operators match the inputs of subsequent ones. For example, a valid composition of operators \(A\), \(B\), and \(C\) may take the form:

4https://github.com/InfiniTensor/ninetoohed
5https://triton-lang.org/main/getting-started/tutorials/
6https://triton-lang.org/main/pythonapi/generated/triton.testing.do_bench.html
Figure 7: Performance Curves of Common Operators
<table>
  <tr>
    <th>Model</th>
    <th>Speed Up (A100/H100)</th>
    <th>GPU Efficiency (A100/H100)</th>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>0.85 / 0.90</td>
    <td>31.92% / 31.71%</td>
  </tr>
  <tr>
    <td>GPT-o1</td>
    <td>0.88 / 0.99</td>
    <td>36.38% / 40.23%</td>
  </tr>
</table>

Table 7: Comparison of representative models on A100 and H100 in TRITONBENCH-G

def comb_op():
    a = A()
    b = B()
    return C(a, b)

Here, comb_op represents a composite operator that integrates \( A \), \( B \), and \( C \). We verify the correctness of such compositions through carefully designed test cases. The process ensures that the functional dependencies among operators are preserved and the composed operator behaves as expected.

D Error Categories

We provide the error type statistics of failure operators in TRITONBENCH. A total of 16 error types are identified in the integrated Call and Execution error results. For convenience in presentation, we categorize them into four main groups:
Syntax Errors: including SyntaxError and IndentationError; Attrb&Type Errors: including AttributeError, TypeError, and NotImplementedError; Name&Ref Errors: including NameError, KeyError, IndexError, ModuleNotFoundError, and ImportError; Run&Loge Errors: including ValueError, ZeroDivisionError, RuntimeError, RecursionError, AssertionError, CompilationError, and ResultsError. ResultsError refers to the results inconsistency between reference and the generated operator.

E Experiments on H100

To address the limitation of evaluating performance only on the NVIDIA A100 GPU, we conducted additional experiments on the H100 GPU—a newer high-performance architecture. The results highlight the challenges of maintaining Triton operator performance across different hardware platforms.

Quality of Operators in TritonBench-G are shown in table 7. We observed that the average GPU efficiency of operators in TritonBench-G decreased from 43% on A100 to 31% on H100. This drop suggests that performance portability remains a significant challenge for Triton code, especially in newer hardware environments.

<table>
  <tr>
    <th>Model</th>
    <th>Speed Up (A100/H100)</th>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>1.17 / 1.19</td>
  </tr>
  <tr>
    <td>GPT-o1</td>
    <td>1.33 / 1.32</td>
  </tr>
</table>

Table 8: Comparison of representative models on A100 and H100 in TRITONBENCH-T

Performance Results on TritonBench-G are shown in table 8. We present the speed-up and GPU efficiency (A100/H100) for two representative models:

3. Performance Results on TritonBench-T.
These results emphasize the non-trivial challenge of achieving efficient and portable Triton code across hardware generations. We hope this expanded analysis will offer insights for future research on hardware-aware operator generation.

F Prompts

Here are the four prompts we use in our work: Filtering Prompt, Instruction Prompt, Difficulty Prompt, and Test Code Prompt. Specifically, the first is used to extract Triton-related code from crawled code files; the second instructs the large model to generate corresponding instructions based on Triton code; the third prompts the large model to score the difficulty of Triton operators according to the standards we proposed; and the last asks the large model to generate test code.
Filtering Prompt

{code}
Please help me select all triton kernel functions decorated with @triton.jit and all code that calls these kernels, while only keeping the necessary imports (e.g., triton, torch) and the calling functions.

Note 1: Retain necessary comments related to the Triton code. Code can be optimized, but do not remove all kernel code and its corresponding calls just for brevity.

Note 2: If the triton kernel is decorated with a custom or third-party decorator other than triton, discard that kernel.

Note 3: If @triton.jit appears as a string in the code or is nested within a function body, then discard it.

Note 4: If there are multiple triton kernel functions decorated with @triton.jit and their calling wrapper functions, retain all of them, not just a subset.
    1) Extract all triton operators (kernel functions decorated with @triton.jit and their calling functions) and output them in python code format. If no triton kernel function is found, discard it.
    2) Provide a concise English description of each extracted operator (including both kernel and calling code) in the form of a python dictionary: "description": "Use triton language to..."

Instruction Prompt

{code}
Based on the above Triton operator code, generate a detailed description so that the large model can accurately reproduce the corresponding kernel and wrapper function.
Be clear about the logic and main functionality of the operator, specify the function name, inputs, and outputs, and describe any public variables clearly.
Try to describe the function’s code implementation. Ensure that the large model can reproduce the corresponding function and parameter code based on these instructions.
Note that the output should maintain correct python syntax.

Test Code Prompt

{code}
Write a test code in Python for the above code. Ensure that all branch tests are in a single function starting with “test_”, with no parameters.

Note 1: Particular attention should be paid to the fact that tensor parameters are of GPU type.

Note 2: Try to limit the number of branches to no more than 4.

Note 3: In branch tests, avoid modifying parameters that are later in the argument list with default values (especially if they have out parameters, do not assign them).

Note 4: Store the results of all branch calculations in a dictionary, where the dictionary key is "test_case_n", with n representing the test case number.

Note 5: Ensure that the import paths match exactly as described in the operator documentation to maintain accuracy.

Note 6: The code should run directly, without if __name__ == "__main__".

Difficulty Prompt

{code}
Please evaluate the complexity of the code in the following two aspects based on the requirements of the Triton operator and score it from simple to complex on a scale from 1 to 5:
1) Memory layout complexity: Analyze the memory access pattern, including memory tiling, array transposition, address alignment, cache utilization, and the number of global memory accesses.
2) Computation scheduling complexity: Examine instruction-level parallelism, computation-memory pipeline, thread block design, inter-thread communication, thread branch divergence, and hardware resource utilization.
The final score is the ceiling of the average score from both aspects, and only one complexity score is output in [ ].

Reviewer paper summaries: This paper proposes a framework called TRITONBENCH, aimed at evaluating large language models' ability to generate corresponding Triton code based on operator descriptions. This benchmark includes two evaluation channels of different difficulties that comprehensively measure large models' ability to generate Triton code. TRITONBENCH-G contains 184 carefully selected operators from Github that reflect real-world Triton development needs. TRITONBENCH-T contains common, uncommon operators and their combinations aligned with PyTorch, addressing the lack of operators in open-source code. Unlike traditional code generation benchmarks, this benchmark is GPU-oriented and focuses not only on functional correctness but also on performance considerations. The paper provides dataset construction, experimental results, and error analysis.
Summary of strengths: 1.**Innovation**: TRITONBENCH is the first benchmark testing framework for Triton operator generation, filling the gap in LLM evaluation of GPU DSL-related code generation.

2.**Rich operator forms in the benchmark**: Two channels, TRITONBENCH-G aimed at real needs, TRITONBENCH-T completing various common and uncommon operators and their combinations, having strong inspirational and practical value.

3.**Comprehensive evaluation of large models**: Evaluated commonly used closed-source and open-source large models including Claude, ChatGPT, Deepseek, etc.

4.**Introduction of performance evaluation**: Compared to other code generation benchmarks, introduced relatively reasonable performance evaluation metrics and conducted corresponding tests.
Summary of weaknesses: 1.**Lack of ablation experiments**: For the evaluation of LLM's ability to generate Triton code, there is a lack of in-depth analysis on how each module and the data from both channels affect its comprehensiveness.

2.**Questionable one-shot definition**: The paper set up one-shot experiments, but the specific method is to find similar examples from the training set based on similarity, which sounds more like RAG rather than one-shot.

3.**Lack of training details**: The paper presents fine-tuning results in the experiments, but there is no detailed explanation of the training set in the main text. If the dataset used for training is an important contribution, its description should be placed in the main text rather than in the appendix.

4.**Statistical representativeness of benchmark construction**: For TRITONBENCH-G, the proportion of different difficulties is extremely uneven, with d1 accounting for only 1.6%, d5 for only 2.7%, and d3 and d4 accounting for 81% of the total, indicating that the distribution of data may have obvious bias.

5.**Benchmark set scale**: Doubtful whether 184 operators have sufficient representativeness. Triton operator writing styles vary greatly, and whether the Triton code contained in these operators can represent enough Triton programming needs requires further analysis.
Pretraining pollution: The paper isolates training corpus and test benchmarks, but lacks sufficient discussion on whether the mentioned models (especially commercial models) may have already seen these open-source codes in their corpus, which has a significant impact on zero-shot performance.

6.**Error classification**: It seems to merely classify Python error categories relatively crudely without in-depth analysis of mechanisms. Different error causes within the same category may be completely different.

7.**Experimental setup**: Using only A100 GPU for functional and performance evaluation limits the generality of conclusions.
Comments, suggestions and typos: The experimental hardware used only A100, which is insufficient for performance evaluation. It should be considered that hardware specifications of GPUs of different architectures may vary greatly, and whether the experiment has sufficient practicality needs to be supported by more hardware testing.

Table 1 shows that the operator difficulty distribution in TRITONBENCH-G is severely unbalanced, potentially affecting the objectivity of LLM performance under different difficulties. Section 3.1 mentions using prompts for filtering, but lacks specific selection and exclusion criteria for what conditions are selected and what situations are not selected, thus suspecting selection bias. Statistically, 184 is a bit small as a sample size and does not have enough persuasiveness to demonstrate the ability of LLMs to generate operators of different difficulties.

Lack of details on extracting interfaces from PyTorch, table-2 looks like each operator is composed through combination, details about the combination (apart from the descriptions in 4.1) and how to ensure the correctness of the combination lack detailed description, it is recommended to supplement this part in the appendix.

Section 3.4 mentions that for multi-branch operators, the generated test code should try to cover all branches, but lacks specific data on branch coverage.

Appendix A mentions using CODEBERTSCORE to eliminate overlap between training and testing, but cannot prevent the model's pretraining corpus from possibly including operators in the benchmark. Large language models' pretraining corpus may already include most Triton code on GitHub, so the "zero-shot" setting is not convincing.

For One-Shot, due to the lack of explanation of examples, it seems more like RAG processing from the appendix, looking for similar fragments from the training corpus to add to the context lacks specific detail description, which seems incompatible with the definition of one-shot.

GPU efficiency is placed in Appendix B, such important metric definitions should be placed in the main text. In addition, operator quality cannot be evaluated only through GPU efficiency, more evaluation methods are needed.

The error classification framework has limitations, under the same error category, the causes of AttributeError, TypeError, and NotImplementedError may be completely different. Additionally, ResultsError may be composed of calculation precision errors, implementation scheme-induced errors, and multiple causes of errors, lacking specific analysis. Overall, the classification granularity is coarse, limiting its guiding significance.

Insufficient planning for benchmark maintenance and evolution, Triton's API changes very quickly, and implementations in different repositories are not the same, neglecting these situations will limit the value of this benchmark test.
Overall assessment scores: 3
Metareview score: 3
Metareview: The paper presents a novel dataset for benchmarking LLMs’ code-generation abilities specifically for Triton, a high-level language used for GPU kernels. This represents an expansion of the type of code-generation that can be evaluated, as there are not currently any evaluations specifically for Triton, or for GPU code more generally. 

The reviewers overall found the use case to be interesting and the work to be acceptable. The primary concerns regard the validity of the benchmarks in various aspects–e.g., one reviewer points out that the distribution of the data might not be well balanced, another raises concerns about the lack of variation in how the models are prompted, another raises concerns about the size.

On my personal assessment of the reviews and skimming the paper, I find that the paper is missing more generalizable takeaways that would make it of interest to a wider community. I.e., is there something we can say about LLMs and code-generation in general, or is this paper but specifically about Triton? For a main conference paper, I prefer to see more wide-reaching insights. That, combined with the reviewers' sundry concerns, leads me to recommend Findings but not the main conference.
Summary of reasons to publish: * Novel benchmark dataset covering an area of programming not well represented in existing benchmarks