ICML 2025

KernelBench: Can LLMs Write Efficient GPU Kernels?
Anne Ouyang * 1 Simon Guo * 1 Simran Arora 1 Alex L Zhang 2 William Hu 1 Christopher R´e 1
Azalia Mirhoseini 1
Abstract
Efficient GPU kernels are crucial for building per-
formant machine learning architectures, but writ-
ing them is a time-consuming challenge that re-
quires significant expertise; therefore, we explore
using language models (LMs) to automate kernel
generation. We introduce KernelBench, an open-
source framework for evaluating LMs’ ability to
write fast and correct kernels on a suite of 250
carefully selected PyTorch ML workloads. Ker-
nelBench represents a real-world engineering en-
vironment and making progress on the introduced
benchmark directly translates to faster practical
kernels. We introduce a new evaluation metric
fastp, which measures the percentage of gener-
ated kernels that are functionally correct and offer
a speedup greater than an adjustable threshold
p over baseline. Our experiments across vari-
ous state-of-the-art models and test-time meth-
ods show that frontier reasoning models perform
the best out of the box but still fall short overall,
matching the PyTorch baseline in less than 20% of
the cases. While we show that results can improve
by leveraging execution and profiling feedback
during iterative refinement, KernelBench remains
a challenging benchmark, with its difficulty in-
creasing as we raise speedup threshold p.
1. Introduction
AI relies on efficient GPU kernels to achieve high perfor-
mance and cost and energy savings; however, developing
kernels remains challenging. There has been a Cambrian
explosion of ML architectures (Tay et al., 2022; Peng et al.,
*Equal contribution 1Department of Computer Science, Stan-
ford University, Stanford, California, USA 2Department of Com-
puter Science, Princeton University, Princeton, New Jersey, USA.
Correspondence to: Anne Ouyang <aco@stanford.edu>, Simon
Guo <simonguo@stanford.edu>.
Proceedings of the 42 nd International Conferen...