ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm
  Engineering
Authors: name: Yuki Imajuku, name: Kohki Horie, name: Yoichi Iwata, name: Kensho Aoki, name: Naohiro Takahashi, name: Takuya Akiba

How well do AI systems perform in algorithm engineering for hard optimization
problems in domains such as package-delivery routing, crew scheduling, factory
production planning, and power-grid balancing? We introduce ALE-Bench, a new
benchmark for evaluating AI systems on score-based algorithmic programming
contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench
presents optimization problems that are computationally hard and admit no known
exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench
encourages iterative solution refinement over long time horizons. Our software
framework supports interactive agent architectures that leverage test-run
feedback and visualizations. Our evaluation of frontier LLMs revealed that
while they demonstrate high performance on specific problems, a notable gap
remains compared to humans in terms of consistency across problems and
long-horizon problem-solving capabilities. This highlights the need for this
benchmark to foster future AI advancements.