SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research
  Repositories
Authors: name: Ben Bogin; name: Kejuan Yang; name: Shashank Gupta; name: Kyle Richardson; name: Erin Bransom; name: Peter Clark; name: Ashish Sabharwal; name: Tushar Khot
Published: 2024-09-11 17:37:48

Abstract: Given that Large Language Models (LLMs) have made significant progress in
writing code, can they now be used to autonomously reproduce results from
research repositories? Such a capability would be a boon to the research
community, helping researchers validate, understand, and extend prior work. To
advance towards this goal, we introduce SUPER, the first benchmark designed to
evaluate the capability of LLMs in setting up and executing tasks from research
repositories. SUPERaims to capture the realistic challenges faced by
researchers working with Machine Learning (ML) and Natural Language Processing
(NLP) research repositories. Our benchmark comprises three distinct problem
sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems
derived from the expert set that focus on specific challenges (e.g.,
configuring a trainer), and 602 automatically generated problems for
larger-scale development. We introduce various evaluation measures to assess
both task success and progress, utilizing gold solutions when available or
approximations otherwise. We show that state-of-the-art approaches struggle to
solve these problems with the best model (GPT-4o) solving only 16.3% of the
end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of
this task, and suggests that SUPER can serve as a valuable resource for the
community to make and measure progress.