Title: Measuring AI Ability to Complete Long Software Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: benchmarks

evaluations

Natural language processing

Software engineering

forecasting
Abstract: Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as o3 have a 50% time horizon of around 110 minutes. Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated since 2024. The increase in AI models’ time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes, combined with better logical reasoning and tool use capabilities. We discuss the limitations of our results—including their degree of external validity—and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.

Measuring AI Ability to Complete Long Software Tasks

Abstract

Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon , the time humans typically take to complete tasks that AI agents can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, agents built using current frontier AI models such as o3 have a 50% time horizon of around 110 minutes. Furthermore, frontier AI time horizon has doubled approximately every seven months since 2019, though the trend may have accelerated since 2024. The increase in AI agents' time horizons seems to be primarily driven by greater reliability, ability to adapt to mistakes, logical reasoning, and capacity for tool use. We discuss the limitations of our results-including their degree of external validity-and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.

1 Introduction

In the last five years, frontier AI systems have undergone a dramatic transformation in capabilities, evolving from basic text generation [1] to autonomously executing complex multi-hour machine learning research projects [2]. Sufficiently capable AIs could perform dangerous, highly complex actions like autonomous development of chemical, biological, radiological or nuclear weapons (CBRN) and self-replication and adaptation outside human control [3]. Understanding AI capabilities helps inform the development of safety guardrails as systems become increasingly powerful. In particular, many frontier AI developers have committed to using measures of specific AI capabilities to determine the necessary risk mitigations for their frontier AI systems. Robust benchmarks that can accurately track and forecast AI capabilities thus form the foundation for responsible AI governance and risk mitigation.

LengthoftasksAlagentshavebeenabletocompleteautonomously

for169softwareengineering,cybersecurity,generalreasoning,andMLtasks

4 hrs

2hrs

1hr

30 min

15 min

50%successrate

8min

GPT40

4 min

GPT4

2 min

1min

30 sec

GPT3.5

15 sec

8sec

GPT3

4 sec

2 sec

GPT2

Doublingtime:207days

2018-09-03+data

1sec

R2: 0.97

2019

2020

2021

2022

2023

2024

2025

2026

2027

Modelreleasedate

However, existing benchmarks face several key limitations. First, they often consist of artificial rather than economically valuable tasks. Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 1 biasing the comparison to human performance. Most critically, individual benchmarks saturate increasingly quickly [6], and we lack a more general, intuitive, and quantitative way to compare between different benchmarks, 2 which prevents meaningful comparison between models of vastly different capabilities (e.g., GPT-2 versus Claude 3.7 ). As a consequence, while the last few years have seen dramatic increases in AI performance on many individual benchmarks, understanding the progress of AI capabilities in general has required estimating the qualitative difficulty of the latest benchmarks AI systems can pass.

Wepropose tracking AI progress over time using the task completion time horizon : the duration of tasks that AI agents can complete at a certain success probability, providing an intuitive measure of real-world capability compared to humans. As AI agents based on current models may not reliably complete all tasks of a given length, we operationalize this by measuring the X%-(task completion) time horizon -the length of tasks that AI agents can complete approximately X% of the time.

We prototype this methodology using three datasets designed to capture skills required for research or software engineering (Section 2.1), totaling 170 tasks with a wide range of difficulty: general software agent tasks from HCAST [8], machine learning research engineering tasks from RE-Bench [2], and Software Atomic Actions (SWAA), a new suite of shorter software tasks that provides signal on pre-2023 models (Appendix B.1.3). Using skilled human baseliners, we estimate the duration that a domain knowledgeable human (without task-specific context) takes to complete these tasks (Section 2.2). We evaluate the performance of AI agents based on 12 frontier models from 2019 to 2025 on these tasks (Section 2.3). Using methodology inspired by human psychometric studies, we then estimate the duration of tasks that models can complete with 50% success rate-the 50% time horizon (Section 3.1). 3

1DiverseTaskSuite

2TaskPerformance

HCAST

Human Runs

Agent Runs

Diverse tasks that require

agency

?

1hrs

2hrs

3hrs

√

1 min-30 hrs

97 tasks

个

#Time Estimate

% Success Rate

SWAASuite

Single-step tasks

sampled from SWE work

3

Time Horizon Analysis

1-30 sec

66 tasks

Find Time Horizon

Doubling Time

RE-Bench

Horizon Length Per Model

7AlR&D Research

Engineering tasks

150%

8hrs

7 tasks

Dates

Release Date

Time Estimate

We find that the 50% time horizon has been growing exponentially from 2019-2025 on these tasks, with a doubling time of approximately seven months (Figure 1). We compare our main result with our exploratory results on non-SWAA tasks, finding that the 2023-2025 horizon growth rate is about 20% faster than the 2019-2025 rate (Section F.1). We also measure the 80% time horizon of AI agents (Figure 17) and find a similar trend, though horizons are roughly 5x shorter. This progress appears to be driven by several key factors: improved logical reasoning, better tool use, and greater reliability and self-awareness in task execution (Section 3.3). We also note several limitations of current systems-notably, performance is much lower on less structured, 'messier' tasks (Section F.2).

Since our tasks do not perfectly represent the average segment of intellectual labor by researchers and software engineers, this raises the question of external validity (Section 4): whether the exponential trend holds on real-world tasks. We include four supplementary external validity experiments, which find little evidence of performance trends being slower on the somewhat more realistic tasks we tested, but do not rule out the possibility that trends are meaningfully slower on the distribution of tasks required to automate software engineer jobs. We also find evidence that AI agent time horizons can differ by a large factor depending on the task domain and reference human population.

We conclude by discussing implications for AI capabilities forecasting (Section 5). Naively extrapolating the trend in horizon length implies that AI will reach a time horizon of > 1 month (167 work hours) between mid-2028 and mid-2031 (Figure 6). However, extrapolation is affected by both external validity concerns and future changes in the trend.

1.1 Related Work

Here, we briefly discuss related work. See Appendix A for an expanded discussion.

Agentic AI capability benchmarks Many recent benchmarks aim to measure the capability of AI systems to act as agents in general, including AgentBench [9], ToolBench [10], GAIA [11], TheAgentCompany [12], the Berkeley Function Calling Leaderboard [13]. In this work, we study AI agent performance on RE-Bench[2], HCAST [8], and SWE-Bench Verified [7], which focus on machine learning and software engineering tasks. Other similar benchmarks include MLAgentBench [14], MLEBench [15], and DSBench [16]. In concurrent work, Miserendino et al. [17] create the SWE-Lancer benchmark, consisting of 1400 freelance software engineering tasks gathered from Upwork with real-world payouts totalling above $1 million.

AI capability forecasting This work builds on previous efforts to forecast the capabilities of AI systems over time, including efforts to contextualize benchmark performance using human experts

Table 1: Example tasks for different durations; see Rein et al. [8] and Wijk et al. [2] for more examples.

Phuong et al. [3]; Murray et al. [18] and efforts to forecast performance on particular benchmarks Owen [19]; Pimpale et al. [20]. Contextualizing AI capabilities using human time horizon was proposed by Ngo [21], and similar concepts were discussed in Carlsmith [22] and Cotra [23].

Human psychometric methods Our methodology draws inspiration from human psychometric testing, especially Item Response Theory [24; 25; 26].

2 Measuring AI agent performance on realistic tasks

2.1 Task suite / dataset

Our tasks comprise three distinct task suites:

A subset of HCAST [8]: 97 diverse software tasks ranging from 1 minute to 30 hours. 4

RE-Bench [2]: 7 difficult ML research engineering tasks, all eight hours long.

Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from 1 second to 30 seconds.

All tasks are automatically scored with a continuous score or binary threshold; details of how we normalize and process scores are given in Section 3.1, and other task suite details in Appendix B.

Example tasks See Table 1 for five example tasks. Tasks under 1 minute measure knowledge relevant to professional software engineering. At 10 minutes, tasks can include the easiest meaningful step of a real software project (e.g. configuring a common open-source package). The shortest tasks that could be standalone economically relevant projects take around one hour; by eight hours tasks represent valuable software projects. Many more examples of these tasks are available in Wijk et al. [2] and Rein et al. [8].

2.2 Baselining

In order to ground AI agent performance, we also measure the performance of multiple human 'baseliners' on most tasks and recorded the duration of their attempts. In total, we use over 800 baselines totaling 2,529 hours, of which 558 baselines (286 successful) come from HCAST and REBench, and 249 (236 successful) from the shorter SWAA tasks. 148 of the 169 tasks have human baselines, but we rely on researcher estimates for 21 tasks in HCAST.

Our baseliners are skilled professionals in software engineering, machine learning, and cybersecurity, with the majority having attended world top-100 universities. They have an average of about 5 years of relevant experience, with software engineering baseliners having more experience than ML or cybersecurity baseliners. For more details about baselines, see Appendix C.1.

ModelSuccessRatevs

HumanCompletionTime

AverageSuccessRateforCeneralAutonomy+SWAA+RE-BenchTasks

Tasksdiversity-weighted

R2=0.80

100% 1

1.0

X

Mean Model Success Rate

Average successrate

80%

0.8

60%

0.6

40%

0.4

20%

0%

0.2

HCAST

RE-Bench

GPT-2

GPT-40

04-mini

00

SWAA

GPT-4 0314

GPT-41106

ol-preview

davinci-002 (CPT-3)

Claude 3.7 Sonnet

心

30

gpt-35-turbo-instruct

Claude 3.5 Sonnet (Old)

Claude 3.5 Sonnet (New)

Human Time-to-Complete

2.3 Results

We measure 12 frontier (and 4 near-frontier) models released between 2019 and 2025; full information on the models and scaffolds used can be found in Appendix C.3. We perform 8 runs 5 per agent/task pair and report the average results in Figure 3. There is a strong upwards trend over time, with recent models completing approximately 50% of all tasks, while earlier models perform substantially worse. We find substantial correlation between the tasks that models can complete (Figure 22), with average correlation of approximately 0.73.

There is a negative correlation between the time it takes a human baseliner to complete a task and the average success rate (across all models) on the task. This decrease in success rate over length (Figure 3) is well-fit by an exponential model ( R 2 ≈ 0 . 80 when regressing model success rate against the logarithm of human time-to-complete). As expected, more recent models can complete longer tasks, with Claude 3.7 Sonnet and o3 completing some tasks that take human baseliners > 4 hours (Figure 4).

3 Time horizon

3.1 Computing time horizon

To convert agent run data to task completion time horizon, we first convert the agent performance on each task to a binary value (success or failure). Many tasks are naturally binary, including all SWAA tasks. Continuously scored tasks are binarized via a task-specific threshold chosen to represent human performance. For HCAST, the task-specific threshold is the same 'target score' the human baseliner tries to achieve, which we also use to filter for successful runs. RE-Bench tasks have a fixed time rating of 8 hours, so the task-specific threshold is the average score of 7-9-hour human runs.

Once we have agent success rates and time ratings for each task, we fit time horizons by performing the following logistic regression: 6

Lengthof tasksAl agentshavebeen abletocomplete autonomously

Claude3.7Sonnet

Claude3.5Sonnet(New)

Claude3.5Sonnet(Old)

Claude3Opus

03

1.0

Success Probability

TinNHorizon:

me Horizon:

Time Horizon:

Horizon:

54min

28min

18 min

6min

1h

somin

00

1ssnsnon

hrad

Ssiminninmfhrhshed

Tseesessetninrminmifhrhisohed

04-mini

[O

ol-preview

GPT-40

Success Probability

Horizon:

18min

Horizon:

39min

me Horizon:

22min

ime Horizon:

9min

Time Horizon:

6min

00

1sesessermin

GPT-4 1106

CPT-4 0314

gpt-3.5-turbo-instruct

davinci-002 (CPT-3)

GPT-2

1.0 1

Success Probability

0.8

o6-

me Horizon:

8min

Time Horizon:

5min

Time Horizor

Time Horizon:

9 sec

Time Horizon

02

00

Task length (human time)Task length (human time)Task length (human time)Task length (human time)Task length (human time)

where t task is the geometric mean time of successful human baselines, and h agent and β agent are learned parameters, with h agent representing the 50% time horizon.

3.2 Time horizon vs. release date

In Figure 1, we plot the time horizons of each model against their release date-the date at which the lab first publicly announced the frontier model. 7 In addition, we linearly regress 8 log( time horizon ) against release date, finding that time horizon has doubled every 207 days with a 95% bootstrapped confidence interval 166-240 days (roughly ± 19% ). Error bars are calculated via 10,000 samples from a three-level hierarchical bootstrap over task families, then tasks, then runs.

While there are wide error bars on each individual models' horizon lengths, these errors are highly correlated between models. This is because tasks at the same human time rating vary widely in difficulty for models, and sampling easy (or hard) tasks will result in a higher (or lower) horizon estimate for all models. Therefore, we are more confident in the slope of the time horizon trend than in the time horizon of any particular model. The fit is not sensitive to various hyperparameters such as regularization, weighting of tasks, and WLS vs. OLS (see Figure 6).

Horizon length increases substantially over the entire time period from 2019 to early 2025. GPT-2 has a 50% time horizon of only 2 seconds, while o3 has a 110-minute time horizon and succeeds at several tasks over 4 hours. Also, o3 lies above the long-run trend (p = 0.006); this is robust to methodological ablations like using continuous scoring (Appendix H), which may imply the trend in 2024 and early 2025 is faster.

3.2.1 Time horizons at 50% success rate vs 80% success rate

To check whether our choice of 50% success rate affects the long-run trend, we also compute the time horizon at which AI agents succeed at tasks with 80% success rate, shown in Figure 17. The doubling time in 80% time horizon (204 days) is similar to the doubling time of 50% time horizon (207 days), within margin of error. However, models' 80% time horizons are 4-6x shorter, suggesting that even models that sometimes succeed on difficult and diverse tasks cannot reliably perform tasks of moderate length. Due to a limited amount of tasks of sufficiently short length, we cannot confidently measure time horizons at very high success rates (e.g. 95%) - see Section E.3 for more discussion.

3.3 Qualitative analysis

By examining transcripts from 31 randomly sampled unsuccessful GPT-4 1106 agent runs and 32 o1 runs, we observe that models seem to have improved greatly in terms of tool use capabilities, demonstrate a markedly greater ability to adapt to mistakes (as opposed to repeating unsuccessful actions), and perform much better at parts of tasks requiring logical reasoning or code generation. However, we noticed that AI agents still seem to struggle in intuitively 'messier' environmentsspecifically, environments without clear feedback loops, or where the agent needs to proactively seek out relevant information.

Table 3.3 shows the categorized failures by model. We provide definitions of these categories, as well as examples of qualitative improvements and continuing limitations in Appendix D.

4 External validity and robustness

We first check whether the 2023-2025 trend without the SWAA dataset retrodicts the trend since 2019, and find that the trends agree (Appendix F.1). We also perform three experiments to check external validity.

First, we replicate our methods on SWE-bench Verified. We find that the exponential trend still holds (Figure 4.1), albeit with an even shorter doubling time, possibly because SWE-bench Verified difficulty annotations, meant to represent high-context maintainer time, may differentially underestimate how long human contractors take to perform easier SWE-bench tasks.

Second, we score the HCAST and RE-Bench tasks against a list of 16 'messiness' factors that aim to capture some systematic differences between the tasks studied here and hard'real-world' tasks. Some examples include whether the task is resource limited, novel, or involves a dynamic environment (Section H.6). Controlling for task length, we find models perform worse on tasks that have higher messiness scores. Notably, trends in AI agent performance over time are similar for the lower and higher messiness subsets of these tasks (see Figure 12). In particular, we find no evidence of plateaus in performance trends specific to the higher messiness subset.

Third, we measure AI agent performance on a small, uncontaminated set of our internal pull requests (PRs) (Section C.2). We find large differences in the speed at which different human groups complete the internal PR tasks, with contractors taking 5-18x longer to fix issues than repo maintainers. We also find that AI agent performance is worse than would be predicted by maintainer time-to-complete as a measure of task length, but is consistent with contractor time-to-complete,

In addition, following the original publication of our paper, we performed some preliminary analysis exploring the time horizon of models on other domains, including math competition, video QA, and autonomous driving (Appendix I). We found that while the specific estimates of the current time

Performance of frontier AI models using reported SWE-bench Verified results (Section 4.1). We observe a similar exponential trend to Figure 1, albeit with a steeper slope.

TimeHorizonforSWE-BenchVerified Tasks

Task time (for humans)that model completes with

4 hrs ]

Claude 3.5 Sonnet (New)

2 hrs

Claude 3.5 Sonnet (Old)

1 hr

+

ol

30 min

GPT-40

50%successrate

15 min

CPT-4 1106

8min

4 min

2 min

1min

Fit on our tasks

30 sec

Models ≥CPT-4 1106

15 sec

Doubling time:143days

FitonSwE-Benchtasks

8 sec

Doubling time:70 days

4 seC

R2:0.87

2023

2024

2025

Modelreleasedate

horizons of models sometimes differ between domains, most of these domains also show exponential trends with similar doubling times (Figure 27).

SWE-bench Verified information is provided below. Details of the retrodiction and messiness analysis can be found in Appendix F, and details of the internal PRs experiment in C.2. Details of our cross-domain preliminary analysis can be found in Appendix I.

4.1 SWE-bench Verified

To check whether we observe similar performance trends on other benchmarks, we apply our methodology to SWE-bench Verified, an industry standard benchmark for evaluating language model performance on software engineering tasks [27]. All tasks in the SWE-bench Verified dataset were harvested from large open source repositories, like matplotlib or django, and then filtered to ensure they are automatically checkable and well-specified [28].

Model time horizon computed from SWE-bench Verified tasks seem to follow an exponential trend from late 2023 through 2024. However, while the doubling time predicted by HCAST + SWAA + RE-bench using 2024 models is 143 days, the doubling time on SWE-bench Verified results is shorter-around 70 days.

SWE-bench Verified time annotations were based on the expected time it would take an 'engineer who has had a few hours to familiarize themselves with the codebase' to solve the issue [28]. We found that annotator time estimates differentially underestimate how long our contract baseliners take to complete the easiest SWE-bench verified tasks. As a result, our time horizon estimates for SWE-bench Verified (which use annotator times) are likely to underestimate the time horizon of less capable models relative to contractor times, in turn shortening doubling times. For more details see Appendix H.5.

5 Extrapolation

Extrapolating towards one-month-horizon AI To forecast when AI systems may be capable of autonomously producing large economic value and potential catastrophic actions, we chose one month (approximately 167 working hours for a fair comparison with humans, since humans cannot work 24/7) for two reasons. First, Ngo [21] writes that a 1-month AGI (defined as an AI that outperforms most knowledgeable humans who are given 1 month of work hours, i.e. 167 hours, to perform the task) would necessarily exceed human performance at economically valuable endeavors like writing large software applications, founding startups, and making novel scientific discoveries. 9 Second, one month is around the period when new hires at a company begin to complete onboarding and generate economic value, 10 and so an AI with a horizon of 1 month could be capable of acquiring context like a human employee, allowing it to complete high-context as well as low-context

Uncertaintyinextrapolated dateof1monthAl(167workinghours)

Bootstrap (tasks)

Bootstrap (runs)

Bootstrap(models)

Weighting/Regularization

IIDBaselineNoise

Overall(2019-2025 trend)

Overall(2024-2025trend)

2026

2027

2028

2029

2030

Extrapolateddateof1monthAl(167workinghours)

tasks. We cannot rule out that a system capable of 1-month tasks, even at 50% reliability, could be transformative for our society, potentially including the ability for catastrophic harm.

Sensitivity analysis Figure 6 shows a sensitivity analysis of our results over various sources of noise. 'Bootstrap (tasks)', 'Bootstrap (runs)' and 'Bootstrap (models)' represents the limited number of tasks, run-to-run variance, and limited models respectively. 'Weighting/regularization' represents methodological choices; this perturbation includes 10 hyperparameter combinations with and without task diversity weighting, and with a logistic regression regularization parameter between 0.01 and 0.2. 'IID Baseline Noise' multiplies every task duration by a random factor based on the empirical distribution of baseline times. Note our confidence in the 2024-2025 only trend is low because there are only seven frontier models in this time span.

Because our analysis puts low probability on the growth rate of time horizon on our tasks being much slower than one doubling every 8 months, the uncertainty in the extrapolated date of 1-month AI is fairly small (80% CI width about 2 years, central estimate mid-2029). If future progress instead follows the 2024-2025 trend, 1-month AI would arrive sooner, with half the probability in 2027 and even late 2026.

It is possible that systematic biases and alternate methodologies have a greater impact on forecasts, and we discuss these in Appendix H. In addition, due to the inherent difficulty of predicting the future, real forecasts will have larger error than this naive extrapolation, which we discuss below.

Task distribution limitations We note several differences between this task suite and realistic tasks in Appendix B.2: in short, realistic tasks rarely have automatic scoring, and often require dealing with other agents, resource constraints, dynamic environments, and high reliability requirements. Such differences cast doubt on whether the rapid performance improvements seen on this task suite (and other benchmarks) will generalize. It may be that benchmarks like HCAST and SWEBench Verified are insufficient for forecasting real-world AI capabilities, and accurate forecasting will require deliberate construction of more realistic benchmarks without these limitations.

Future changes in time horizon trends The time horizon trend may significantly change in the future due to factors including agency training, compute scaling, and automation of AI research and development. Agency training and eventual automation of AI R&D are more likely to speed up the trend, whereas compute scaling could slow it (but be partially offset by algorithmic improvements);

a future artificial general intelligence (AGI) could technically have an infinite time horizon. See Appendix E for details.

6 Discussion

Summary In this paper, we proposed an intuitive, quantitative metric for AI capabilities: the task completion time horizon, which relates AI performance on tasks to the typical length of time human experts require to complete the tasks. We constructed a dataset of 66 shorter SWAA tasks, combined these with existing benchmarks, and conducted baselines to estimate their difficulty. To measure the trend in time horizon, we benchmarked 11 frontier AI models released between 2019 and 2025 on this dataset, and found (Section 3) that the 50% task completion time horizon on these tasks has been growing exponentially from 2019-2025 with a doubling time of approximately seven months (Figure 1).

We explored the robustness and external validity of our results in several ways: we confirmed that a similar trend also holds on the 80% time horizon (Figure 17). We performed preliminary analyses of 50% time horizons for other domains in Appendix I, and found that similar trends hold across other domains. That being said, we noted important limitations of current systems, particularly their lower performance on less structured, 'messier' tasks (Appendix F.2), complicating a naive interpretation of our results.

Finally, we attempt to extrapolate the trend on these tasks to one-month (167 hours) AI (Section 5), finding that if the trend continues and observed performance trends generalize to real-world tasks, an 80% confidence interval for the release date of AI that can complete 1-month long software tasks spans from mid-2028 to mid-2030 (Section 5)- or even as soon as early 2027 if the 2024-2025 trend continues.

Limitations and future work Even though we believe the exponential trend to be robust, we caution that time horizon is always measured relative to a domain, task distribution, and human baseliners' level of skill and context, and can be difficult to measure in practice, especially for longer horizons or success rates near 100%. We discuss nuances with interpreting time horizon in Appendix E.3. Finally, in Appendix E.4, we discuss ways of extending or improving our results, such as by studying multiple domains in detail, more models, better elicitation, inference scaling, or other statistical methods.

Acknowledgments and Disclosure of Funding

The authors thank the following reviewers for feedback on draft versions of this paper. Ryan Greenblatt, Aaron Scher, Romeo Dean, Mike Knoop, Jeff Wu, Steve Newman, Rohit Krishnan, Taren Stinebrickner-Kauffman, JS Denain, Jacob Pfau, Seb Krier, Anton Troynikov, Max Henderson, Ajeya Cotra, Max Nadeau, Tamay Besiroglu, Nate Thomas. The authors thank Charles Foster and Michael Chen for their support with the publication.

We especially thank the following reviewers for substantial feedback: Sara Fish, David Duvenaud, Eli Lifland, Holden Karnofsky, and Rif A. Saurous. We also thank Stephanie He for her graphic design work on Figure 2, and Ryan Greenblatt for input on the scoring methodology.

References

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114 , 2024.

Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.13793 , 2024.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791-4800, 2019.

Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249 , 2025.

Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. Artificial intelligence index report 2024, 2024. URL https://arxiv.org/abs/2405.19522 .

Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified. https://openai.com/index/introducing-swe-bench-verified/ , 2024. Accessed: 2025-02-26.

David Rein, Joel Becker, Amy Deng, Seraphina Nix, Chris Canal, Daniel O'Connell, Pip Arnott, Ryan Bloom, Thomas Broadley, Katharyn Garcia, Brian Goodrich, Max Hasin, Sami Jawhar, Megan Kinniment, Thomas Kwa, Aron Lajko, Nate Rush, Lucas Jun Koba Sato, Sydney Von Arx, Ben West, Lawrence Chan, and Elizabeth Barnes. HCAST: Human-Calibrated Autonomy Software Tasks. Forthcoming, 2025.

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688 , 2023. URL https://arxiv.org/abs/2308.03688 .

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. StableToolBench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.

Gr´ egoire Mialon, Cl´ ementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id= fibxvahvs3 .

Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161 , 2024.

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs. berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html , 2024.

Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Conference on Machine Learning , ICML'24. JMLR.org, 2024.

Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench: Evaluating machine learning agents on machine learning engineering, 2024. URL https://arxiv.org/abs/2410.07095 .

Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents to becoming data science experts? arXiv preprint arXiv:2409.07703 , 2024.

Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?, 2025. URL https://arxiv.org/abs/2502.12115 .

Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-Franc ¸ois Gimenez, and Simeon Campos. Mapping AI benchmark data to quantitative risk estimates through expert elicitation. arXiv preprint arXiv:2503.04299 , 2025.

David Owen. How predictable is language model benchmark performance?, 2024.

Govind Pimpale, Axel Højmark, J´ er´ emy Scheurer, and Marius Hobbhahn. Forecasting frontier language model agent capabilities, 2025. URL https://arxiv.org/abs/2502.15850 .

Richard Ngo. Clarifying and predicting AGI. https://www.lesswrong.com/posts/ BoA3agdkAzL6HQtQP/clarifying-and-predicting-agi , 2023. Accessed: 2024-03-21.

Joseph Carlsmith. How much computational power does it take to match the human brain?, 2020. URL https://www.openphilanthropy.org/research/ how-much-computational-power-does-it-take-to-match-the-human-brain/ . Open Philanthropy report, August 14, 2020.

Ajeya Cotra. Draft report on AI timelines, 2020. URL https://www.alignmentforum. org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines . Draft report on AI timelines; technical report.

Frank B Baker. The basics of item response theory . ERIC, 2001.

R. J. de Ayala. Handbook of Item Response Theory: Models, Applications, and Issues . Guilford Press, 2017.

Fernando Mart´ ınez-Plumed, Ricardo B.C. Prudˆ encio, Adolfo Mart´ ınez-Us´ o, and Jos´ e Hern´ andez-Orallo. Item response theory in AI: Analysing machine learning classifiers at the instance level. Artificial Intelligence , 271:18-42, 2019. ISSN 0004-3702. doi: https:// doi.org/10.1016/j.artint.2018.09.004. URL https://www.sciencedirect.com/science/ article/pii/S0004370219300220 .

OpenAI. OpenAI o3-mini. https://openai.com/index/openai-o3-mini , 2025. [Accessed 18-03-2025].

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=VTF8yNQM66 .

Dave Zielinski. How to optimize onboarding, June 2019. URL https://www.shrm. org/topics-tools/news/hr-magazine/how-to-optimize-onboarding . Accessed on March 17, 2025.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR) , 2018. URL https://gluebenchmark.com/ .

Alex Wang, Yada Pruksachatkun, Naman Nangia, Amanpreet Singh, Julian Michael, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems (NeurIPS) Workshop , 2019. URL https://super.gluebenchmark.com/ .

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR , abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300 .

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world apis, 2023. URL https://arxiv.org/abs/ 2307.16789 .

Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, Vishaal Udandarao, Jingyi Lu, Shiyang Chen, Sam Purkis, Tianshuo Yan, Wenye Lin, Gyungin Shin, Qiaochu Yang, Anh Totti Nguyen, David I. Atkinson, Aaditya Baranwal, Alexandru Coca, Mikah Dang, Sebastian Dziadzio, Jakob D. Kunz, Kaiqu Liang, Alexander Lo, Brian Pulfer, Steven Walton, Charig Yang, Kai Han, and Samuel Albanie. ZeroBench: An impossible visual benchmark for contemporary large multimodal models, 2025. URL https: //arxiv.org/abs/2502.09696 .

[Author] Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374 . arXiv preprint, arXiv:2107.03374.

Jeremy Austin, Augustus Odena, et al. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732 . arXiv preprint, arXiv:2108.07732.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track , 2021. URL https://arxiv.org/abs/2105.09938 .

Dario Amodei and Danny Hernandez. AI and compute, 2018. URL https://openai.com/ blog/ai-and-compute/ . OpenAI blog post, May 16, 2018.

Epoch AI. Data on notable AI models, 2024. URL https://epoch.ai/data/ notable-ai-models . Accessed: 2025-03-04.

Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2022.

Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.08926 , 2024.

Hao Song and Peter Flach. Efficient and robust model benchmarks with item response theory and adaptive testing. International Journal of Interactive Multimedia and Artificial Intelligence , 2021.

Denis Paperno, Germ´ an Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern´ andez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1525-1534, 2016.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2023. URL https: //arxiv.org/abs/2210.03629 .

Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench - a sparse benchmark of reasoning and knowledge in large language models, 2025. URL https://arxiv.org/abs/2407.12844 .

Ege Erdil and Tamay Besiroglu. Algorithmic progress in computer vision, 2023. URL https: //arxiv.org/abs/2212.05153 .

Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress in language models, 2024. URL https://arxiv.org/abs/2403.05812 .

Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIDE: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138 , 2025.

METR. Measuring automated kernel engineering. https://metr.org/blog/ 2025-02-14-measuring-automated-kernel-engineering/ , 02 2025.

David Malin Roodman. On the probability distribution of long-term changes in the growth rate of the global economy: An outside view, 2020. URL https://api.semanticscholar. org/CorpusID:221641799 .

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: a graduate-level google-proof qa benchmark, 2023. URL https://arxiv.org/abs/2311.12022 .

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874 .

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/ abs/2403.07974 .

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arxiv.org/abs/2404.07972 .

Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL https://arxiv.org/abs/ 2307.13854 .

Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2025. URL https://arxiv.org/abs/2405.21075 .

A Expanded Related work

Below, we expand on our discussion of related work in Section 1.1.

A.1 Agent and capability benchmarks

The evaluation of AI capabilities has evolved significantly from single-task benchmarks to complex, multi-step evaluations designed to assess agent-like behavior. While traditional benchmarks such as GLUE [30], SuperGLUE [31], and MMLU [32] have provided valuable insights into language model performance, they primarily measure static knowledge rather than the dynamic problemsolving capabilities essential for real-world applications. Recent work has developed more complex agent benchmarks. AgentBench [9] evaluates agents across diverse environments including web browsing, coding, and game playing. MLAgentBench [14] and MLEBench [15] focus specifically on machine learning research tasks, while ToolBench [33; 10] assesses tool use capabilities. The recent ZeroBench [34] involve difficult reasoning, but in the context of visual puzzles rather than economically valuable tasks. Other noteworthy benchmarks include GAIA [11], which evaluates reasoning across multiple modalities, and BIG-bench [35], which contains hundreds of diverse tasks including many requiring multi-step reasoning.

Software engineering has emerged as a particularly informative domain for evaluating AI capabilities. HumanEval [36] and MBPP [37] provide programming challenges of varying complexity, while more complex benchmarks like SWE-bench [28] and APPS [38] test more sophisticated programming abilities. We use SWE-bench Verified [7]'s human time estimates for task completion in our work. RE-Bench [2], which we use in this work, evaluates models on complex research engineering tasks that may require hours of human effort and compares AI performance to human machine learning engineers. SWE-Lancer [17] generates tasks

While these benchmarks provide valuable insights into specific capabilities, they often lack a unified metric that allows for tracking progress over time and comparing models of vastly different capabilities. Our time horizon approach aims to address this gap by providing a continuous metric that can be used to measure progress across different capability levels.

A.2 Forecasting AI progress

Quantitative forecasting of AI progress has employed various approaches, often starting with the observation that the compute used in AI training has increased dramatically over time. Amodei and Hernandez [39] observed that AI training compute usage has been increasing exponentially, doubling approximately every 3.4 months between 2012 and 2018; Epoch AI [40] included more recent data as well as trends in training dataset size and energy usage.

Other work has studied how AI performance has increased over time, relating benchmark performance to release date, compute usage, and other inputs. Sevilla et al. [41] found that compute usage growth rate increased at the start of the 'deep learning era' in 2010, coinciding with increases in performance. More recently, Owen [19] and Pimpale et al. [20] use compute and other metrics to forecast future benchmark performance.

Several recent efforts have been made to contextualize AI benchmark performance. One such effort is the annual AI Index Report [6], which tracks performance across various benchmarks, including the date at which models achieved human-level performance. Murray et al. [18] had cybersecurity experts relate AI performance on Cybench [42] to the ability to autonomously develop malware. Phuong et al. [3] commissioned professional forecasters to predict whether AI ranks amongst top public concerns by 2030, conditioned on benchmark performance. These efforts generally lack a unified, quantitative metric for cross-benchmark comparison.

Carlsmith [22] and Cotra [23] developed the 'bio-anchors' framework, in which they related the compute involved in training AI models to the 'effective horizon length' of tasks required for AI to have transformative impacts. Ngo [21] proposed using the time horizon for which AI systems outperform most human experts at most tasks to measure general AI capabilities. In our work, we empirically evaluate the relationship between task duration and AI agent success rate, which we convert into a quantitative metric of AI agent performance.

A.3 Psychometric methods and Item Response Theory

Our methodological approach draws inspiration from psychometric testing, particularly Item Response Theory (IRT) [24], which models the relationship between latent traits (such as ability) and observed responses to test items. In traditional IRT, item difficulty is a parameter in a logistic model predicting response correctness based on respondent ability. Our approach inverts this, using task completion time (a proxy for difficulty) to predict AI performance. Our methodology also relates to difficulty estimation techniques in educational testing [25], where multiple metrics including completion time are used to estimate the difficulty of tasks. IRT has been applied to machine learning classifiers by Mart´ ınez-Plumed et al. [26], and was used to design efficient benchmarks in Song and Flach [43].

B Task suite details

As with most benchmarks, the three task suites we study were designed to isolate a specific unit of work that can be reliably accomplished within a time limit. This usually means that the tasks require much less context than the average task in the middle of a larger project. 11 We confirmed that all tasks were possible given the instructions provided by having humans successfully complete each task at least once. 12

B.1 Task subsuites

The HCAST and SWAA suites are divided into task families , which are groups of tasks that are similar. For example, the 'crossword' task family consists of tasks such as creating a 3x3 crossword puzzle, or a 5x5 crossword puzzle, etc. We segment tasks into families because performance within families is correlated and we down-weight families with many tasks for diversity.

B.1.1 HCAST suite

We use 97 tasks from 46 task families in HCAST, a diverse set of challenges in cybersecurity, machine learning, software engineering, and general reasoning.

Tasks in this suite range from easy tasks that take humans a couple of minutes (e.g. looking up a basic factual question on Wikipedia) to tasks that take expert humans multiple hours (e.g. writing CUDAkernels, or fixing a subtle bug in PyTorch). Because modern frontier AI systems are relatively more proficient at text-based tasks, the majority of these tasks do not require visual/multimodal capabilities, and all tasks are solvable by text editing via a bash shell.

Compared to many recent benchmarks, tasks in HCAST are not designed to be as difficult as possible for either human domain professionals or current AI systems. Instead, most tasks are designed to be realistic, such that doing well on the task requires skills we expect to be economically useful. As a result, we expect that most of these tasks are solvable by humans with a few years of professional experience in the relevant domain.

Tasks are defined by their instructions, starter resources, and an algorithmic scoring function. Task instructions are strings, typically between 1-2 sentences and a few paragraphs, although they can refer to other sources of information included as starter resources, or externally available via the internet. Starter resources typically consist of code, data, and documentation.

Each task is automatically scored between 0 and 1, with higher scores indicating better performance. Many tasks only return scores of 0 or 1, but for tasks with continuous scoring, we manually define a success threshold score, which we use in some of our analysis to binarize agent scores. 13

DistributionofTaskDifficulty

25

TaskSource

HCAST

RE-Bench

20

SWAA

15

10

5

1 sec

4sec

15 sec

1 min

4 min

15 min

1 hr

4 hrs

16 hrs

Human tasktime

More details about HCAST tasks are provided in Rein et al. [8].

B.1.2 RE-Bench suite

RE-Bench consists of 7 challenging open-ended ML research engineering environments, each of which are intended to take a human expert approximately 8 hours to complete. See Wijk et al. [2] for more details.

B.1.3 Software atomic actions (SWAA) suite

HCAST is designed to be a diverse set of tasks, but the shortest tasks are around 1 minute long, limiting both the representativeness and the achievable resolution in measuring AI agent performance on shorter tasks. To fill this gap, we observed that real-world intellectual labor consists in part of measurable, single-step actions shorter than 1 minute. We created the SWAA task suite, which comprises 66 small tasks corresponding to < 1 minute atomic actions commonly performed in software engineering work. The SWAA subset includes both multiple-choice and completion questions.

Example SWAA task (file selection)

Which file is most likely to have a password in it?

credentials.txt

installation_notes.txt

main.py

launcher_win.exe

In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable to software engineering, the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study. SWAA consists of 5 task families, three representing common decisions, one for code completion, and one for math; see Appendix B.1.3 for more details.

Development of SWAA tasks was blind to AI agent performance; that is, all tasks were written before seeing AI attempts, and elicitation (development of the few-shot prompt used for evaluation) was carried out on a separate development task suite.

SWAAconsists of five task families: three representing common decisions, one for code completion, and one for math.

Decisions (multiple choice)

File selection: Which file has a certain property, or is appropriate to read in a situation?

Alert triage: Which team at a company should investigate an alert?

Request routing: Which action needs to be taken in response to some request?

Fill-in-the-blank

Code completion: Complete a single word of code.

Math: Solve simple arithmetic problems, either standalone or software engineering-themed word problems.

When developing SWAA tasks, we attempted to avoid biasing the results by blinding task authors to model performance, so the preliminary set of 71 tasks was written without running any models on them during development. These were filtered down to the final set of 66 by excluding tasks that more than one baseliner failed.

RE-Bench See Figure 8 for a description of the RE-Bench tasks. 14

B.2 Limitations of the task suite

The tasks we use to benchmark AI capabilities are systematically different from real tasks. These differences could result in the trends we observe on these tasks not generalizing to real world tasks. For instance, all SWAA, HCAST, and RE-Bench differ from real-world tasks in the following ways:

Automatic scoring All tasks we use are automatically scorable, meaning a piece of code running in the task environment determines the final score. This imposes constraints on e.g. the format of solutions, that tend to reduce task open-endedness, and the need for sensible value judgments.

No interaction with other agents No tasks involve interacting with other autonomous agents. Coordinating with, or competing with, other agents seems likely to increase task difficulty. For instance, by increasing the importance of strategic decision-making, realtime coordination, and predicting the actions of other complex agents.

Lax resource constraints NoSWAAtasks, and few HCAST tasks saliently involve making efficient use of a limited resource-a common constraint in real-world tasks.

Unpunishing Similarly, very few tasks are punishing of single mistakes. 15 This is in part to reduce the expected cost of collecting human baselines. Real world tasks can often be more punishing, for instance, when they involve competing against other agents. For instance, a single blunder in a chess game can greatly reduce the chance of winning the game.

Static environments Tasks in this suite typically use environments that do not significantly change unless directly acted upon by the agent. In contrast, real tasks often occur in the context of a changing environment.

We attempted to measure how these systematic differences might affect AI agent performance in Section F.2, by including the above properties as 'messiness' factors. We found that though the absolute performance on 'messier' tasks was lower, the trends in performance were similar to less messy tasks.

C Methodological details

C.1 Human baselines

C.1.1 HCAST tasks

We use existing baselines collected as part of HCAST. These baselines are collected from domain professionals with relevant experience in software engineering, ML, and cybersecurity. Baseliners work in the same environment as agents, using Vivaria (an open source platform for language model agent evals) with their screens and audio recorded for manual review, to prevent cheating. They are incentivized with bonuses for successful completion and for completing tasks faster than other baseliners. After screening out attempts on tasks other than our subset of HCAST, failed attempts, and those with issues (such as using disallowed AI tools), we include 286 successful baselines from approximately 460 total attempts. Task durations are calculated using the geometric mean of successful baselines, with manual estimates for tasks lacking successful baselines. 16

For further information about HCAST baselines, see Rein et al. [8].

C.1.2 RE-Bench baselines

For RE-Bench, we used the baselines from [2]. As baseliners were instructed to achieve the best performance for each task, we consider the task duration of each of these 6 tasks as 8 hours, and instead use the mean score achieved by baseliners who spent between 7 and 9 hours to convert raw score into success threshold.

Baselines for RE-Bench were conducted with the same baseliner pool and incentives as with HCAST, but with a few differences. Rather than being told to achieve a threshold score, baseliners were given a fixed time limit of 8 hours and instructed to maximize their score; the threshold was set based on average baseliner performance. RE-Bench baseliners had access to AI tools, so the human scores on these tasks may be biased upwards.

C.1.3 SWAA baselines

Unlike HCAST and RE-Bench, which were baselined by external contractors, SWAA is baselined by internal employees with relevant expertise using a custom webapp that enables more accurate timing. Because these tasks are intended to be a single step and exclude context acquisition, the timer for SWAA tasks ends as soon as the user chooses a response. For decision based tasks, only one selection is allowed to avoid random guessing; for fill in the blank tasks, baseliners can try multiple times until getting the answer correctly or opting to skip. We baselined each decisionbased task 4 times and each fill-in-the-blank style task 3 times.

Unlike the HCAST and RE-Bench sets, SWAA is baselined by internal employees with relevant expertise using a custom webapp that enables more accurate timing. Because these tasks are intended to be a single step and exclude context acquisition, the timer for SWAA tasks starts after reading general instructions and ends as soon as the user chooses a response, so that baseline time only includes reading the question itself and choosing an answer. Baselines done with our usual setup have timing overhead of seconds to tens of seconds, which would be unacceptably high for single-step tasks with duration shorter than 2 seconds.

C.1.4 Baseline success rate

We aggregate human baseline times into a task length rating by taking the geometric mean time of successful baseline runs, which is more predictive of model performance than mean time because baseline times are roughly log-normally distributed.

We chose to filter successful runs for two main reasons - one practical and one principled. Firstly, collecting enough baselines to estimate the time-versus-success curve for each task would be practically difficult. Secondly, we wanted to exclude cases where the baseline failed for reasons that are not applicable to models. A substantial fraction of human failures appeared to fall in this category including humans having insufficient expertise for the task, or giving up on a task for unclear reasons (possibly due to getting interrupted, or getting bored). In particular, because we were optimizing for obtaining successful baselines, our payment scheme incentivized contractors to make a quick guess or give up early, in order to move on to other tasks where their expected earnings were higher.

However, conditioning on success biases towards shorter task length ratings, thereby underestimating model performance. This is especially the case if many of the human failures are due to problems that are also relevant for models - for example, if some of the tasks require guessing. This bias is most significant on longer tasks, which often have a baseliner success rate below 50%; therefore, we have lower confidence in the difficulty ratings of these longer tasks, and expect they may be underestimates. If this is the case, we may be underestimating the pace of model improvement.

Humantime horizon Analternative approach would be to calculate the human time horizon using the same methodology as we do for models. One natural interpretation of time horizon would imply that the time horizon of 'a human given x hours' is x hours. Since our baseliners were paid for spending up to 8 hours per task, we would expect their time horizon to be around 8 hours. However, in practice it's much lower, at around 1.5 hours (which would imply that the best models will surpass humans in under 7 months). As discussed above, we think this is artificially low, given that many human failures seemed to be artifacts of our incentive scheme.

Figure 9 shows a graph of baseliner success rate by task length.

C.1.5 Baseline time analysis

We manually watched and annotated the recordings of 50 baselines to get a sense of how our baseliners spent their time. Based on this small survey of baselines:

On short (5-15 minute) tasks, a substantial fraction of the time (between 25-75%) is spent reading instructions or understanding how to submit correctly. Particularly for short tasks, our time estimates may be hard to interpret as there is some fixed overhead for reading the instructions, independent of the difficulty of actually performing the task per se.

Our tasks are designed to require relatively low amounts of context but base liners still spend over 25% of their time reminding themselves of relevant information (e.g. how to do port forwarding).

Humanbaselinerperformanceontasks

human

1.0

Fitted curve

Empirical success

rates w/ ± 2SE

SuccessProbability

0.8

0.6

TimeHorizon:

1hr,37 min

0.4

0.2

0.0

Tasklength(human time-to-complete)

C.2 Internal PR tasks

Summary We ran GPT-4o, Claude 3.5 Sonnet (New), and o1 on five uncontaminated issues from an internal repository. Resolving these issues was real work performed by internal staff, so we might expect results on these tasks to better represent performance on real economically valuable tasks than a typical benchmark task.

We find that our contract baseliners take 5x-18x longer to resolve issues than repository maintainers. Additionally, AI agent performance on these issues is not inconsistent with AI agent success rate curves derived from HCAST, SWAA, and RE-Bench performance if contractor time-to-complete is used to measure the tasks length. However, it takes much longer for our contract baseliners to complete these tasks than repository maintainers. This suggests that time horizons may have better correspondence to the labor of a low-context human, rather than a high-context human.

Methodology We collected five real, recent, uncontaminated issues from an internal repository. We then ran GPT-4o, Claude 3.5 Sonnet (New), and o1 on these issues. We recorded how long it took repository maintainers to solve these issues, and also had external baseliners attempt to fix these issues. Unlike SWE-bench Verified, we did no filtering to require these issues to be automatically verifiable. Instead, maintainers of the relevant repository scored model and baseliner solutions manually, as if they were reviewing PRs:

0 if the PR was incorrect and required a fundamental refactor.

0.25 if there were minor changes that must be made before merging.

0.75 if there were nits but the PR could still be merged.

1.0 if the PR could be merged as is.

Example issues We include two example issues - one easy (Issue 1), and the other (Issue 8) more challenging. Issue 8 demonstrates how context on the codebase may provide a dramatic speedup in human time to complete between baseliners and repository maintainers. Results of these baselines can be seen in table 3.

Issue 1

Issue 8

Repo maintainer time vs. baseliner time There was a dramatic difference in how long it took baseliners to complete issues, compared to how long it took repository maintainers to complete issues (Table C.2). In practice, repo maintainers were 5-18x faster at completing the given issues.

Manual scoring methods and results Maintainer, baseliner, and model solutions were manually scored as if they were pull requests. We experimented with both contractors scoring PRs, as well as different repository maintainers scoring PRs. In practice, repo maintainers scored results much more consistently than contractors. Contractor correlations were between 50-60%. Repo maintainer correlations were between 88-91% .

Time to score For models to be cost competitive on real work, their total cost to complete a PR must be less than the cost for a repository maintainer to implement this PR themselves (plus ancilliary costs, e.g. the time of a code reviewer). As such, we kept track of the time that it took to score model solutions, to better understand the net cost of using a model to fix a real issue. Repo maintainers scored submissions much faster than contractors, although the ratio was less dramatic than time to complete-contractors took an average of 8 minutes per run, while repo maintainers took an average of 3.5 minutes per run. Different issues had different average times to score, with easier issues generally being much quicker to score, and all issues being faster to score than to complete. In the future, we plan to create an approximate cost metric that takes into account scoring effort, model cost, and model success rate. We expect that this will be a useful framing in understanding how the ratio of model / maintainer work may shift in the future, as the cost of agent usage continues to fall.

Contract baseliners vs repository maintainers Model success rates were reasonably consistent with those predicted by contract baseliner times. In practice, baseliners also did much worse on easy tasks than experienced maintainers. Issue 11, for example, took a baseliner upwards of 1.5 hours to complete, and so we would not expect models to consistently complete this task.

Qualitative impressions Naively comparing to per-time bucket success rates on HCAST + SWAA + RE-bench tasks, repository maintainer time-to-complete is not a good predictor of model performance on these tasks. For example, Issue 11 was the simplest task for maintainers. It requires writing simple comments across 10 Python files, and it took maintainers less than five minutes to complete. However, models did not successfully complete the task successfully once in 30 runs. There were also tasks where models did succeed, as we would expect from per-time bucket success rates on other tasks. For example, models were consistently able to add a missing folder creation step to a single stage in a data pipeline. While baseliners and models both struggle compared to repository maintainers, they struggle in different ways. Baseliners often lacked knowledge about the tools and techniques used in the codebase: e.g. 'what is DVC?' or 'what is bootstrapping again?'. Current AI models, on the other hand, consistently demonstrate knowledge of tools and techniques, but appear to struggle with the larger context required for working in a real codebase.

C.3 How models were run

All models were run using agent scaffolds using the Anthropic or OpenAI APIs.

We used the same agent scaffolds across the evaluation suite, with no task-specific prompting or scaffolding, except for the SWAA tasks, which used a simple prompting scaffold. All agents were provided with the same affordances provided to human baseliners.

GPT-2 is incompatible with our scaffolding due to low context length, so we imputed a score of zero for GPT-2 on all tasks in RE-Bench and HCAST. We think this is reasonable because the far more capable davinci-002 (GPT-3) scores zero on this set. Removing these imputed GPT-2 zero scores has a negligible effect on all subsequent results in this paper.

C.3.1 Scaffold and platform details

Most AI models were evaluated with modular-public-a basic agent scaffold. This scaffold provides the model with Python and Bash commands and some very simple context management to keep the input within the context window length of the LM. We used a slightly different scaffold for o1-preview and o1, because they seemed to struggle with tool use, responding to environmental feedback, and generally acting as an agent.

For both human baselines and AI agent runs, our experiments use the open-source platform Vivaria. CPU and GPU resources are provided inside the secure VMs based on the resource specifications of tasks. Table 6 displays the scaffolding used for each model. Both modular-public and triframe/duet incorporate principles from the ReAct framework [46], which interleaves reasoning traces with actions. These agents are developed on a held-out dev set of tasks, not included in HCAST, to reduce the likelihood of overfitting to these tasks. See the respective repositories for implementation details about these scaffolds.

Both scaffolds allow agents to plan via chain-of-thought reasoning, before calling a selection of tools. The agents interact with their task environment primarily through running Python code and Bash commands.

In Modular, the model generates a single command for execution, then the scaffold executes the function call, returning relevant information from the environment (e.g. STDOUT/STDERR ). This process repeats until either the model determines that the answer is ready for submission or the system reaches its predefined usage threshold.

In Triframe, to decide on a command to execute, the model generates one suggested plan, then generates three possible commands to execute based on the plan, as well as three suggested commands that ignore the plan (to diversify the ideas it generates). Then, the model generates two scores between -2 and 2 for each of the six proposed actions, and the scaffold executes the top scoring function call averaged across the two scores.

C.3.2 Compute usage

The total amount of compute used included about 2,000 H100-hours for RE-Bench environments and 50,000 CPU hours for other environments on a combination of cloud and internal machines, plus roughly 50,000 H100-hour equivalents of compute used internally or from API providers for inference. This includes compute used for preliminary experiments. Compute used for data analysis is minimal.

C.4 Comparison to Item Response Theory

Item response theory (IRT) is a collection of statistical techniques used to predict the probability that a person will answer a question correctly, based on their ability level. The standard IRT twoparameter logistic (2PL) model is (simplified from Cai et al.):

Where T i (1 | η agent ) indicates the probability that a person with ability level η agent answers a task correctly. 17 IRT is usually used to regress on both the question difficulty and the ability level

simultaneously; in contrast, we define the question difficulty as the logarithm of human baseline time, and therefore only regress on ability level. In particular, our model is as follows.

Note the following changes from the 2PL model:

α task = log t task where t task is the geometric mean of successful human baseline times.

η agent can now be interpreted as log h agent where h agent is the 50% time horizon of the agent rather than an abstract ability parameter. This is because when h agent = t task , our model predicts a success probability of σ (0 · β agent ) = σ (0) = 0 . 5 .

β agent is an agent-dependent learned parameter, rather than being task-dependent.

To fit equation 1, we weight tasks by diversity, such that a task from a family of size n gets weight 1 / √ n . We then use logistic regression to fit the time horizon h agent and slope β agent for each model. For example, following this methodology o1 has a time horizon of about 39 minutes (see Figure 4).

These changes allow us to compute an interpretable time horizon without several complexities of IRT. First, IRT has an initial calibration/joint optimization stage in which the task difficulty is estimated from all agent scores, so the ability score of one agent may depend on other agents' performance; in our method each agent's time horizon is computed individually. Second, IRT outputs 'ability scores' which then need to be converted to a time horizon estimate; our method eliminates that step by estimating horizon directly. Third, the 2PL model uses one slope parameter per task, whereas we use one slope parameter per agent. In our setting with many fewer agents than tasks, this reduces the overall number of parameters.

In exchange, we forego the ability to use IRT metrics designed for test design, such as Fisher information functions for individual tasks (used in e.g. Kipnis et al. [47]). Future improved methodologies could include these metrics to design sample-efficient benchmarks and evaluations grounded in human data.

D Qualitative analysis

D.1 Failure categories

To better understand the differences between current and older AI agent failures, we separately sampled 31 unsuccessful agent runs from our GPT-4 1106 agent and 32 unsuccessful runs from our o1 agent, and manually labeled them for the following exclusive categories of failures:

Poor planning and tool choice : the agent generates a high level plan that seems unworkable on its own merits, or picks tools for the plan that would not accomplish the desired purpose.

Incorrect mental math or reasoning : the agent performs incorrect mental math or logical reasoning at a crucial step, causing the run to fail.

Premature task abandonment : The agent abandons the task in the middle of the attempt and either submits a nonsensical answer or submits a solution without checking for correctness. These failures often result from the agent submitting their answer before looking at all the pieces of code or information required to arrive at the correct solution.

Repeating failed actions : The agent repeats the same behavior that doesn't make progress toward the problem, such as running a command that leads to an error over and over again, without trying other approaches.

We report the results in Table 3.3. We find that over a third of the GPT-4 failures resulted from repeating failed actions, compared to 2 out of 32 for o1, which we see as quantitative evidence for our claim that models seem to have improved in their ability to adapt to mistakes. Half of the o1 failures resulted from abandoning the task prematurely, while only a quarter of the GPT-4 failures resulted from the same-this may result from o1 failures occurring on qualitatively more difficult tasks, or may reflect idiosyncrasies of o1.

D.2 Specific skills of newer models

To better contextualize the observed trend of improved model performance, we examined the transcripts for the tasks where earlier models (e.g. GPT-4) do substantially worse than current models. Specifically, we categorized all task families based on the type of expertise required, and noticed that current models tend to outperform earlier models for tasks that involved ML training, reverse engineering compiled software binaries, and cybersecurity CTFs. In addition, we noticed that agents improved greatly on tasks requiring situational awareness of the AI agent's limitations or defeating an opposing strategy. This gave us five sets of task families to examine.

For each of these task family sets, we then used contractors to manually read through all runs from all models for all tasks in each of these five task families, and identified possible explanations for the improvement in AI agent performance on those tasks, as well as potential limitations.

We provide examples of both ways that models have improved, as well as their major limitations, as described in Section 3.3.

D.2.1 What do newer models do better?

We start by discussing some of the ways in which current models have improved compared to pre2023 models. For some of our public tasks, we also provide links to the transcripts of the full runs.

Greater robustness to mistakes Models have become increasingly robust over time, and especially seem to be able to avoid repeating failed actions. Early models typically fail to correct even trivial mistakes, instead engaging in looping behavior or introducing additional errors. However, starting with GPT-4o, models demonstrate improved ability to recover from mistakes and coursecorrect, an ability that is especially valuable in the harder variants of the pico ctf tasks, or other coding related tasks that require iteration. Consider examples from a task where models need to debug a small amount of Python code:

In one run gpt-4-turbo-2024-04-09 introduced syntax errors related to having a misplaced backslash character in a Python file, and despite copious attempts is unable to understand or fix the issue until it gives up.

Claude 3.5 Sonnet (New) first adds code at the wrong place, and then attempts several times to add the code to the correct file using the CLI tool sed . However, when doing so it encounters IndentationErrors and duplicated methods. Finally it successfully pivots to writing the entire file from scratch.

Agent output

I apologize for the confusion. It seems that our previous attempt didn 't fully remove the old 'save' method. Let's try one more time to fix the 'base_model.py' file. We'll remove both 'save' methods and add the correct one

...

Agent output

I apologize for the persistent issue. It seems that our previous attempts didn 't fully resolve the problem. Let's try one more time to fix the 'base_model.py' file by completely rewriting the content of the file:

...

Tool use Early models like davinci-002 (GPT-3) struggle with syntactically correct tool use, while later models improved with clear explanations and examples in their prompt. Current frontier models generally use the tools afforded in our scaffolding without issue. For example, we noticed that when tasks required editing Python scripts, GPT-3 struggled with formatting even when provided explicit

instructions, and GPT-4o would sometimes write scripts line-by-line using echo commands in bash, while o1-preview would consistently edit files efficiently.

Despite the reminder in the user message about the expected format, davinci-002 fails to use the correct syntax with angle brackets.

User

No function call was included in the last message. Please include a function call in the next message using the <[tool_name]> [args] </[ tool_name]> syntax.

GPT-4o sometimes struggles to write the Python script effectively. In a run on the same task, it ineffectively writes each line of the script with a separate command.

o1-preview consistently edits files on this task without any issue.

Improved logical and programmatic problem solving Consistent with increasing performance on coding and reasoning benchmarks, we saw models' ability to write syntactically and semantically correct code gradually increase. This allowed models to more reliably interpret task instructions correctly and use code to find correct answers.

For example, on the interpret data task, GPT-4o fails in one run due to an incorrect interpretation of the request to find 'the tallest building in the world in 1972' as finding 'the tallest building in the world built in 1972', as opposed to the common sense interpretation that the task is to find the tallest building in the world in 1972. For example, it writes code like:

In contrast, o1-preview correctly interprets the request as finding the tallest building in the world built in or before 1972.

D.2.2 What do they still do poorly?

Despite improvements, even current frontier models still struggle in substantial ways. In general, our observerations indicate that agents tend to do worse on tasks that are qualitatively 'messier'. We investigate this claim further in Section F.2.

AI agents struggle without clear feedback Without clear feedback or cheap ways of verification, such as unit tests or answer checking scripts, all models we tested often have difficulty understanding whether their solution is correct. For example, two of the tasks hardest for models in their respective time buckets are 'blackbox' and 'symbolic regression'. Both of these tasks involve guessing a hidden function under the constraint that probing for information is costly.

AI agents often fail to proactively seek out relevant information We also observe that all models still exhibit deficiencies in understanding their own limitations or proactively seeking out helpful information; instead, models tend to assume that they already know how to complete the tasks and then only reevaluate after they fail to do so.

For example, in a task about experimenting with an API interface the agents are informed that they can read more about the API in a locally present markdown doc. Even if agents succeed, they typically start off hallucinating or guessing the API endpoints, and only when they encounter an error from the task environment do they read the API.

In a capture the flag task, agents have to open a file with unknown encoding in python, and often fail on the first try because they specify an incorrect encoding. Even the best models respond by trying out different encodings in the python script, which is inefficient and wastes tokens. We never saw an agent running the bash file command that would identify the necessary encoding directly.

E Further discussion

E.1 Constant factors versus slope changes

Although extrapolation is always imperfect, forecasting AI time horizons far into the future is much more sensitive to changes in doubling rate than to constant factors in time horizon. For example, a naive extrapolation based on o1's time horizon of 39 minutes and a past doubling time of 218 days (3.2x/year) predicts that AIs will reach a 1-month 50% time horizon (roughly 8 doublings over o1) about 4.8 years after the release date of o1. A 2x increase in doubling time would delay the 1 month point by a further 4.8 years, but a 2x constant factor decrease in time horizon would only delay it by 0.6 years.

E.2 Important factors that could change time horizon slope

Agency training Horizon growth since 2024, which may be faster than the long-term trend, could be explained by researchers post-training models to be more agentic (that is, capable of taking many sequential actions towards completing a task) using outcome-based RL. Research into making models capable and agentic is likely to continue. Future agency training could be faster than the long-run trend (since post-training may be more compute-efficient than pretraining at increasing horizon length). But 2024-2025 agency training could also be a one-time boost from picking lowhanging fruit, in which case horizon growth will slow once these gains are exhausted. Overall, we think agency training is more likely to increase the time horizon growth rate compared to the 2019-2024 trend.

Compute scaling Between the release of GPT-2 and today, the compute used to train the most impressive frontier language models has increased by at least a factor of 10,000x [40], with training compute usage doubling every 6-10 months [41]. More recently, models like o1 and o3 have began to use more compute at inference time. It is unclear whether there is sufficient capacity to expand either training or inference compute by many more orders of magnitude in the next 5 years. However, algorithmic improvements, which have historically decreased the compute requirements for a fixed performance level [48] [49], can substitute for compute limitations. We think that limits to compute scaling will slow the growth of AI agent time horizons somewhat, but be partially compensated by more investment into algorithmic improvement.

Automation of AI R&D The main inputs to AI research and development are compute and researcher time. If future AI systems are capable of substituting for human research engineers and/or increasing the compute-efficiency of training, the rate of AI progress will increase. We think it is likely that there will be substantial AI R&D automation once frontier AI time horizon reaches tens of hours, shortening the time from then until one-month-horizon AI.

AGI will have 'infinite' time horizon An infinite time horizon does not mean an arbitrarily capable AI, merely the ability to complete tasks that take humans an arbitrarily long length of time. If an artificial general intelligence (AGI) is capable of completing all tasks expert humans can with a success rate of at least X%, its X% time horizon will necessarily be infinite. Therefore, if such systems are ever developed, the long-term trend in time horizon will be faster than exponential, with an asymptote at the date of AGI deployment.

E.3 Interpreting time horizon

Although time horizon is an intuitive measure of AI agent capability, measuring it requires a large dataset annotated with human time, and time horizon is always measured relative to a task distribution and baseliners' levels of context and skill.

Context and skill effects At real companies, junior software engineer hires often take weeks of onboarding to begin contributing economic value. The human baseliners that determine the length of most tasks have much less context than average employees, potentially increasing measured task length. Our tasks are designed to require minimal context, which somewhat mitigates this problem; our internal PRs (Section C.2) were not designed this way, and so baseliners took many times longer than employees. However, highly skilled baseliners can also complete tasks far faster than average employees. Our expert baseliners are likely much more skilled than the average software engineer, potentially decreasing our measured task length.

Task distribution effects Figure 3 shows that AI agent success rate is imperfectly predicted by human time-to-complete, meaning that other factors also substantially influence the difficulty of tasks. When models are measured in different domains of intellectual labor like research mathematics, computational biology, or law, we expect their time horizons to differ.

Measuring extreme time horizons Accurately measuring that the X%-time horizon of an AI agent is about t minutes requires many tasks of human length t that the AI agent completes with a success rate of about X%. This has two implications. First, accurately measuring very long time horizons requires a dataset of difficult tasks with long human baseline runs, which can be impractical to construct, especially because success criteria for realistic difficult tasks are often complex enough to require manual grading. Second, measuring time horizons at extremely high success levels 95%, 98%, or higher - requires very large task datasets containing many easy tasks, with near-zero label noise that cover the population of tasks they are meant to represent.

Human time horizon measurements In theory, one could also measure the time horizon of a human or population of humans. However, there are both theoretical and practical difficulties to doing so. We discuss this more in Appendix C.1.4.

E.4 Limitations and future work

We believe that there are several ways in which our work could be improved.

More models with better elicitation In general, we find that properly eliciting models can make a very large difference in their performance. 18 We have put a limited amount of effort into eliciting models to get good performance on our tasks, so while our results are a reasonable lower bound, some models may have somewhat greater capabilities than we demonstrate. The most work was done to elicit o1 and the original Claude 3.5 Sonnet, each of which had around 2-3 engineer weeks of iterative development. All other models use the same scaffolding with at most minor changes.

Future work could replicate our results with more effort spent on eliciting the full capabilities of frontier models.

More rigorous human baselining Our per-task human time estimates are likely noisy due to relatively small sample size, and potentially also systematically skewed in various ways. Most notably, we select only successful completions of a task, and encourage baseliners to give up on tasks they may not complete in a reasonable amount of time. Our baseliners' skills also vary significantly, and a wide variety of skills are relevant to these tasks. Though we attempt to match baseliners with appropriate tasks, this process is unlikely to be perfect. From manual reviews of baseline attempts, we also observe that humans sometimes simply give up even when the task seems within their capabilities, and it is unclear what selection effects are produced on the distribution of success times as a result. See Appendix C.1.4 for further discussion of how bias in human baseline times could affect our results. Future work could replicate our results with more rigorous human baseliner selection or explore how sensitive the results are to methodological choices around human baselining.

More natural, varied tasks There are reasons to believe that our task distribution is systematically different from the distribution of economically valuable work (and perhaps systematically different than the distribution of risk-model relevant tasks). We explored some of these reasons in Sections 4, but there remain many differences that we did not explore. For example, the modality of interaction in these tasks is also relatively narrow-for example, no tasks require the use of a mouse. No tasks require cooperating or competing with humans or other agents, 19 while real software engineering or ML research involves communicating and coordinating with managers and other engineers or researchers. Many real-world tasks require very high reliability, and these are underrepresented in our dataset due to the difficulty of measuring models on these tasks. Most importantly, the tasks we study are heavily skewed toward software engineering and ML research. Future work could explore how the capabilities of AI agents are progressing in other domains.

More use of inference compute Our scaffolds made relatively limited use of inference-time compute. When assuming that the human expert is compensated at $143.61/hour (average L4 Engineer salary at Google divided by 2,000 hours), more than 80% of successful runs cost less than 10% of what it would cost for a human to perform the same task. (Figure 14). This implies that if inferencetime computation could be used to improve performance, there is substantial room to do so while still remaining economically competitive with human experts. Previous research has found that techniques such as best-of-k can substantially improve performance on a subset of these tasks,[2] and better use of inference-time compute may lead to substantially different scores.

Data analysis The 2024-2025 trend appears faster than the 2019-2025 trend, but due to the small number of models it is unclear if this is noise. Future work should include hypothesis testing to detect a possible 2024 slope change, and more sophisticated statistical methods to create credible intervals for an overall forecast. We also lose some information in the multiple stages of estimationconversion of baseline data to task difficulty ratings, time horizon computation, and linear regression to find the trend-and end-to-end methods could be more data-efficient.

F External validity details

F.1 Retrodiction from 2023-2025 data

As part of exploratory work for this paper, we measured the time horizon of 9 frontier and nearfrontier models released in 2023 and 2024, using only our HCAST and RE-Bench suites. This trend (with Claude 3.7 Sonnet and o3 added) is shown in Figure 10: time horizon doubles about every six months. Since we only had two 2023 models (GPT-4 0314 and GPT-4 1106) and a small data range (release date spanning 2 years and time horizon spanning 5 doublings), error bars were very wide. In addition, restricting our data further to 2024-only models produced a different trend with time horizon doubling about every three months, so any extrapolation into the future would not be robust.

To address these issues, we collected more data to extend the trendline into the past, developing the Software Atomic Actions (SWAA) suite to decrease the minimum human time of the task suite from 1 minute to under 2 seconds, and enabling us to measure GPT-2, davinci-002 (GPT-3) and GPT-3.5-turbo-instruct on the combined suite.

The 2023-2025 trend may be faster than the longer-term trend (Figure 11). The measured doubling time over the whole 6 year period 2019-2025 inclusive was 207 days, which is less steep than the 172-day trend based on data from non-SWAA tasks and 2023-2025 models. However, this difference is still within margin of error.

F.2 Messiness factors

Real-world intellectual labor often involves messy details that benchmarks usually don't include, such as being under-specified or poorly scoped, having unclear feedback loops or success criteria, or requiring coordination between multiple streams of work in real-time. We generally observed that agents struggle more on tasks that have these 'messy' details (Section 3.3). A natural question is therefore whether agents showed similar rates of improvement on 'less messy' and 'more messy' tasks.

We rated HCAST and RE-Bench tasks on 16 properties that we expected to be 1) representative of how real world tasks might be systematically harder than tasks we study and 2) relevant to AI agent performance. Some example factors include whether the task involved a novel situation, was constrained by a finite resource, involved real-time coordination, or was sourced from a real-world context. We labeled RE-bench and HCAST tasks on the presence or absence of these 16 messiness factors, then summed these to obtain a 'messiness score' ranging from 0 to 16. Factor definitions can be found in Appendix H.6.

The mean messiness score amongst HCAST and RE-Bench tasks is 3.2/16. None of these tasks have a messiness score above 8/16. For comparison, a task like 'write a good research paper' would score between 9/16 and 15/16, depending on the specifics of the task.

On HCAST tasks, AI agents do perform worse on messier tasks than would be predicted from the task's length alone (b=-0.081, R 2 = 0.251), see Figure 13. An increase in task messiness by 1 point reduces mean success rates by roughly 8.1% 20 .

However, trends in AI agent performance over time are similar for lower and higher messiness subsets of the task suite. For example, on sub hour tasks, success rates increased by 40 percentage points between Jan. 2023 and May 2025 in both high and low messiness splits (Figure 12). In particular, we find no evidence of either much slower performance trends, or a plateau, specific to our higher messiness subset.

G Miscellaneous tables and figures

The following are several figures not included in the main body, from external validity and other sections.

H More ablations and robustness checks

There are many ways to measure 'time to complete' a 'well-defined task', so our analysis involved many somewhat arbitrary choices. To ensure our results are robust, we've reproduced our results with alterations to our methodology and find the results are largely robust to these changes.

Because we used these analyses to inform our methodology choices, these ablations were performed relative to a slightly different version of the pipeline than the one in the final report. In particular, they include a slightly different set of baselines and different filtering for task success.

Alternative curve-fits (H.1)

Re-normalizing the task suite to other distributions than log-uniform

50% Time Horizon for HCAST + RE-Bench Tasks

Task time (for humans) that model completes with

2hrs

Claude3.7Sonnet

Claude3.5Sonnet(New)

1 hr

Claude3.5Sonnet(Old)

03

ol

30 min

ol-preview

50% success rate

GPT-40

GPT-41106

15 min

GPT-4 0314

8min

4 min

2min

Doubling time:172days

2023-01-01+ data

R2:0.89

1min

2023

2024

2025

Model release date

50%Time HorizonRetrodicted from 2023-2025 Data

4 hrs 1

Claude3.7Sonnet

2hrs

Claude3.5Sonnet(New)

1 hr

30 min

03

+

15 min

ol-preview

50% success rate

8min

GPT-40

4 min

GPT-4 1106

2min

CPT-4 0314

gpt-3.5-turbo-instruct

1min

O

davinci-002 (GPT-3)

30 sec

★

GPT-2

15 sec

Fit on non-SWAAtasks,2023-2025 models

8sec

Doubling time: 172 days

4sec

2023-01-0l+data

Task time

2sec

Fit on all data

Doubling time:207days

1sec

2019-01-01+ data

2019

2020

2021

2022

2023

2024

2025

Modelrelease date

2022-

Model Success Rates overTime,byTaskLength and Messiness

(Weighted by Task Diversity)

TaskLength:<1hour

Task Length:1+hours

50%least MessyTasks

50%least MessyTasks

1.0 1

(Messiness score<2.3,N tasks=32)

1.0

(Messiness score <3.0, N tasks =18)

Claude 3.7 Sonnet

Claude 3.5 Sonnet (New)

Claude 3.5 Sonnet (Old)

80

0.8

04-mini

03

Weighted Success Rate

0l

ol-preview

GPT-40

0.6

0.6

GPT-4 1106

CPT-4 0314

gpt-3.5-turbo-instruct

0.4

0.4

0.2

0.2

0.0

2-01-01

2-05-01

.05-

01

09

.01

05-1

5-01

.09-

0.0

2-01-01

2-05-01

.9-

01

.01

.05

9

.01

5

o

60

05

2022

2022-

2022-0

2023-0

2023-

2024

2025-01-01

2025-

2022-0

2022-

2022-

2023-

2023

2025

2025

TaskLength:<1hour

Task Length:1+hours

50%most MessyTasks

50%most Messy Tasks

1.0

(Messiness score >=2.3,N tasks =32)

1.0 1

(Messiness score >=3.0,N tasks =22)

0.8

0.8

Weighted Success Rate

0.6

0.6

0.4

0.4

0.2

0.2

0.0

+00

2022-

2-01-01

2-05-01

2023-0

3-05-01

2023-C

2024-

2024-

4-05-01

2024

2025-01-01

2025-0

5-05-01

10-10-7

2022-

2022-

.09-

.01

2023-

2023-

05

2023-

2024

05

Model Release Date

Model Release Date

Excess Success Rate vs Task Messiness Score

TaskMessinessScore

(Observed - Predicted Success Rate)

X

R2=0.283

0.4

X美X

×

0.2

×xx

X！

Xx

0.0

xx.

X

×

！x

xx×

女

一

-0.2

×

Residual(

-0.4

xx

-0.6

2

3

4

F5

6

CostRatiovsTaskLength

n=1460 (10.2% outside visible range)

。

D

100

8

8

Cost Ratio (Model Cost / Human Cost)

10-1

··

.

i.

。

10-2

8

?

10-3

.

?

.8

10-4

10-5

8

10-6

1sec

2 sec

4sec

8sec15sec

30 sec

1min

2min4min8min15min30min

2 hrs

4 hrs

8hrs

16 hrs 1d

TaskLength

Lengthoftasks Al agentshavebeenable tocompleteautonomously for169software engineering,cybersecurity,generalreasoning,and MLtasks

4hrs

2hrs

1hr

30 min

15 min

50%successrate

8min

4 min

2 min

Hyperbolic Fit

1min

2019-0l-01+ data

R2: 0.69

30 sec

Linear Fit

15 sec

2019-01-01+data

8sec

R2:0.36

Task time(t

2 sec

4 sec

Doubling time:207days

2019-01-01+data

Exponential Fit

1sec

R2:0.97

2019

2020

2021

2022

2023

2024

2025

2026

2027

Alternative means of estimating task difficulty

Sensitivity to baseliner ability:

Baseliners whose abilities we are subjectively very confident in

Restricting the task suite to tasks with at least 2 baselines and using the best baseline time on each task as our difficulty estimate

Restricting the task suite to tasks with at least 2 baselines and using the worst baseline time on each task as our difficulty estimate

Adding noise to baseline times (included in Figure 6)

Task choice: Removing RE-Bench tasks

Different weightings of task families: 1 √ family size vs uniform (included in Figure 6); also 1 family size

Estimated training date vs estimates of release date

Continuous scoring: Figure 16

H.1 Alternative curve fits

In our main result, we fit an exponential curve (linear with a log y axis) because the fit is very good ( R 2 ≥ 0 . 96 , depending on the exact data and methodology). Linear and hyperbolic curves have poor fits (Figure 15). Because there are only 12 frontier models in the time span we studied, and the exponential fit has such high R 2 with only two parameters, we think applying fits with more parameters would be more likely to overfit than to give accurate predictions. In particular, we considered:

Adouble exponential function log(horizon) ∼ a + b exp( c · (release date + d )) is strictly more expressive than an exponential (because exp( x ) ≈ x for small x , they are equivalent when c is small)

Likewise, the initial part of any saturating logistic function horizon ∼ a · σ (release date+ d ) looks very similar to an exponential, and without any evidence that AI horizon is leveling off (on our metric), it is essentially impossible to predict when or if it will plateau, or for how long.

Sophisticated stochastic models that asymptote to infinity often have large uncertainties; see Roodman [52], which applied a superexponential diffusion model to economic data. As we discuss in Section 5, large uncertainty may be appropriate when constructing a prediction interval for a long-term forecast, but quantitatively modeling the potential impact of the factors we mentioned is out of scope of this paper, so we prefer to discuss these qualitatively.

LengthoftasksAlagentshavebeenabletocompleteautonomously

ContinuousScoring

Task time (for humans) that model completes with

4 hrs

2hrs

1hr

30 min

15 min

50% success rate

8min

Claude3.7Sonnet

4 min

Claude3.5Sonnet(New)

2 min

Claude3.5Sonnet(Old)

1min

★

03

+

30 sec

x

ol-preview

15 sec

GPT-40

8sec

GPT-4 1106

4sec

CPT-4 0314

2 sec

gpt-3.5-turbo-instruct

davinci-002(GPT-3)data

ys

1sec

￥

GPT-2

R2: 0.97

2019

2020

2021

2022

2023

2024

2025

2026

2027

Modelreleasedate

Length of tasks Al agents have been able to complete autonomously

80%successrate

:(for humans) that model completes with

4 hrs

Claude3.7Sonnet

2hrs

Claude3.5Sonnet(New)

1hr

Claude3.5Sonnet(Old)

03

30 min

ol

15 min

ol-preview

80% success rate

8min

GPT-40

4 min

CPT-41106

CPT-4 0314

2 min

gpt-3.5-turbo-instruct

1min

davinci-002 (GPT-3)

30 sec

15 sec

8sec

4sec

Task time (

Doublingtime:204days

2 sec

2020-0l-01+ data

1sec

R2: 0.96

2019

2020

2021

2022

2023

2024

2025

2026

2027

Model release date

H.2 Horizons at 80% success rate

In Figure 17, we show the trend for the 80% time horizon for models over time. The doubling time is similar to the 50% plot, but horizons are substantially lower.

H.3 Success rate versus inference cost

In Figure 18, we plot the success rate of models across our tasks as a function of token costs. All of the models studied perform better with larger token budgets, but the majority of models show clear evidence of plateaus far below the maximum number of tokens allocated to each model.

(WeightedbyTaskDiversity)

1.0

human

best human for each task

0.8

Claude3 Opus

Claude 3.5 Sonnet (New)

Weighted Success Rate

03

ol-preview

0.6

gpt-3.5-turbo-instruct

davinci-002 (GPT-3)

GPT-40

0.4

GPT-4 1106

CPT-4 0314

GPT-2

0.2

Claude 3.7 Sonnet

Claude3.5 Sonnet (Old)

04-mini

0.0

10-3

10-2

(-OL

100

101

102

103

104

Allowed Cost per Run ($)

2019-2025 and 2024-2025Trendlinesin50%TimeHorizon

for169 software engineering,cybersecurity,general reasoning,and MLtasks

Tasktime (for humans) that model completes with

4 hrs

Claude3.7Sonnet

2 hrs

口

Claude3.5Sonnet(New)

Claude 3.5 Sonnet (Old)

1 hr

03

30 min

ol

15 min

ol-preview

50%successrate

8min

GPT-40

4 min

GPT-4 1106

口

CPT-4 0314

2 min

gpt-3.5-turbo-instruct

1 min

davinci-002 (GPT-3)

30sec

★

GPT-2

15sec

8sec

Doubling time: 109 days

2024-01-0l+data

4sec

2sec

Doubling time:207days

1sec

2019-01-01+data

2019

2020

2021

2022

2023

2024

2025

2026

2027

Modelreleasedate

H.4 2024-2025 Horizon growth trend

If the 2024-2025 horizon growth trend continues to exceed the 2019-2023 slope, future work should apply change point analysis to determine whether the difference in slope is statistically significant.

H.5 SWE-bench Verified

Data Collection We collected per-model, per-task results from the official SWE-bench evaluation results repository for frontier models: Claude 3 Opus, Claude 3.5 Sonnet (Old), Claude 3.5 Sonnet (New), GPT-4 1106, GPT-4o, and o1. Note that this is only 6 of the 11 models we include in the main result.

Time estimates SWE-bench Verified contains time estimates created by contractors that split tasks into four buckets: < 15 min fix, 15 minutes-1 hour, 1 hour-4 hours, or > 4 hours. These estimates were based on the expected time it would take an 'engineer who has had a few hours to familiarize themselves with the codebase' to solve the issue.

We verified annotator time buckets by running seven baselines across 6 SWE-bench Verified tasks. We baselined four tasks in the ' < 15 min fix' bucket, and found that they took our baseliners 8, 26, 67, and 84 minutes respectively. We baselined two tasks in the '1 hour-4 hours' bucket, and both baseliners took between 2-3 hours.

Table 9: The time horizon of less capable models is substantially longer on our tasks than on SWEbench Verified.

Analysis methodology We converted annotator time range estimates into per-task time estimates. We do this by taking the geometric mean of the starting and ending times of the time buckets. We select 16 hours as the upper limit for SWE-bench Verified tasks, but there are only 3 tasks in this time bucket, and they do not meaningfully affect the results. We then applied the same methods as used in Section 3 to convert per-model, per-task results and task time estimates into a model time horizon. To reduce the influence of similar tasks, we consider tasks that involve issues belonging to the same repository as part of the same task family, and down-weight the contribution of tasks by the square-root of the number of tasks in the same family.

Annotator vs contract baseliner times As shown in Table 8, we found that annotator time estimates are likely more inaccurate on the ' < 15 min fix' time bucket, as in practice baseliners often take much longer than this to complete tasks. This results in easier tasks' difficulty being underestimated, meaning that the SWE-bench Verified time horizon estimates likely underestimate true model performance for less capable models, which in turn leads to a steeper exponential trend.

H.6 Messiness factors

Messiness factor labels were collected from 2 internal contractors and one researcher.

The full list of messiness factors and definitions used are shown in Tables 10 and 11.

In Figure 20, we plot the messiness scores of HCAST tasks against log human time-to-complete. We find that messiness is correlated with task length, and that we have very few short tasks with a high messiness score. We also color each point by the performance of the model. Tasks with high messiness and which take humans a long time to complete tend to have low success rates, and vice versa for shorter, lower messiness tasks.

Alimitation of our messiness factors is that they are at least somewhat adversarially selected against current models. This is due to the combination of selecting for perceiving the factor to be relevant to model performance, and all of these factors being expected to make tasks more difficult. As a result, we might expect model performance to be more negatively correlated with our task messiness measure than it ought to be-and for predictive adjustments based on messiness to work less well for models released long before, or after, the time of this analysis.

H.7 Randomizing file names

To better understand the effect of messiness on agent and human performance, we modified an existing HCAST task, which involved looking through local LaTeX and PDF files to find a specific piece of information, by randomizing file names, folder names, and providing less information on what to search for.

H.7.1 Methods

The specific HCAST task modified is called Local Research TeX . The unmodified version of this task initializes the agent environment to have a collection of nested folders and content, and then prompts the user to find some specific information, for example, with the prompt 'According to the ARreport, how many tasks did the best-performing agent complete?' Agents (or humans) must then search the local folders and content to find the specific answer to this question (in this case, 'four'). We modified this task in a variety of ways, but the most notable variant involved:

Modifying the prompt from 'According to the AR report' to 'Find the answer to the following question in one of the resources in /home/agent/resources/':

Modifying the all folder names to random integers, and all file names to meaningless names like 'final' or 'my file.'

We then ran agents for about a hundred runs across the original variant of this task and the modified version of this task, and compared the respective success rates. We also calculated the human time to complete using our standard baseline methodology.

Observed Model Success Rates

by Human Time-to-Complete and Messiness

3.0

0.8

0.7

2.5

Task Length for Human

ObservedSuccessRate

0.6

(logl0 minutes)

2.0

0.5

1.5

0.4

1.0

0.3

0.2

0.5

0.1

0.0

0.0

2

3

4

5

6

7

TaskMessinessScore

H.7.2 Results

Surprisingly, o1's performance is on average better on the messier variant of the report, and human baseliners performed much worse. Qualitatively, o1 appears to do much better on the scrambled-

filename variant of this task because it is not given the term 'AR report' in its prompt. In practice, o1 often decides to grep for the term 'AR report' and fails to find it (as the actual report is named 'ARA report'), and then makes a guess. On the other hand, when told to look in one of the resources, o1 performs a more general search, and finds the report it's looking for (more often). While this is only one limited example, we believe this further illustrates the challenges of quantifying and predicting how aspects of any task may effect agent performance - indeed, factors that make humans worse at some tasks may indeed improve agent performance.

H.8 Correlation in performance between agents

In Figure 22, we report the correlation matrix of per-task success rates between each pair of models.

Excess success rates Excess success rates ( S observed -S predicted S predicted ) are a metric for how much better (or worse) an AI agent performed compared to what we would expect, given a task's length and that model's ability (expected success can be seen in Figure 4).

In Figure 23, we report the correlation matrix for the excess success rate ( S observed -S predicted ) /S predicted (where S predicted is the success rate predicted from the model's time horizon). While the correlations for excess success rate are lower than the correlation of raw task success rate (0.38 instead of 0.71), the fact that it is positive suggests additional factors that explain model success rates across tasks which are common across models.

Model Success Rates over Time, split by Messiness (Weighted by Task Diversity)

50% least Messy Tasks

(Messiness score<2.3, N tasks= 55)

1.0

Claude 3.7 Sonnet

Claude 3.5 Sonnet (New)

Claude 3.5 Sonnet (Old)

Weighted Success Rate

0.8

1★++x+

03

04-mini

ol-preview

0.6

GPT-40

GPT-4 1106

GPT-4 0314

gpt-3.5-turbo-instruct

0.4

0.2

0.0

2-09-01

3-05-01

3-09-01

4-05-01

2022-01-01

10-50-Z207

2022-C

10-10-207

2023-

2023-

2024-01-01

2024-

10-60-07

10-10-507

2025-05-01

50% most Messy Tasks

1.0 1

(Messiness score >=3.0, N tasks =59)

Weighted Success Rate

80

0.6

0.4

0.2

0.0

.01

3-05-01

2022-01-01

2022-05-01

10-60-Z207

2023-01-01

2023-0

2023-C

2024-01-01

2024-05-01

2024-09-01

2025-01-01

2025-05-01

Model Release Date

I Time Horizons of Benchmarks in Other Domains

In Figure 27 we show derived time horizons on nine other benchmarks across a wide range of domains, including software, GUI usage, driving, and video understanding. When subset-level performance data were available we used maximum likelihood estimation to fit the 50% time horizon and the logistic slope parameter. When only overall benchmark scores are available, we assume the slope parameter value is 0.6 (the same value as METR-HRS).

Time horizons on these benchmarks show similar exponential trends to the trends on the three datasets studied in this work (METR-HRS), but absolute performance varies greatly. Software and reasoning domains like scientific QA (GPQA [53]), math contests (MATH [54], Mock AIME), semi-realistic software (METR-HRS), and competitive programming (LiveCodeBench [55]) all have 50-200+ minute horizons which are currently doubling every 2-6 months, similar to METR-HRS. Visual computer use (OSWorld [56], WebArena [57]) time horizons are 40-100x shorter than the software and reasoning cluster but increase at similar rates. Self-driving (Tesla FSD) is improving more slowly at around 0.6 doublings / year. In video understanding (VideoMME [58]), models are capable of answering questions on 1 hour videos with > 50% success rates. However, since video understanding benchmarks show a weak correlation between length and difficulty-for-models (see

Correlationof ObservedSuccessRatesBetweenAgents

1.0

GPT-2

0.55

0.54

0.44

820

0.39

0.39

0.37

0.32

0.31

davinci-0O2 (GPT-3) -

0.55

1

0.87

0.7

0.65

0.63

650

0.56

0.52

0.47

0.43

0.9

gpt-3.5-turbo-instruct-

0.54

0.87

0.83

0.78

0.75

0.7

0.65

0.63

0.56

0.51

0.15

0.16

0.8

CPT-4 0314 -

0.44

0.7

0.83

1

60

160

0.87

0.84

0.72

0.72

0.53

0.57

GPT-4 1106 -

0.38

0.65

0.78

160

0.94

0.9

0.86

0.84

0.78

0.6

0.65

0.7

GPT-40 -

0.39

0.63

0.75

160

0.94

1

0.88

0.88

0.85

0.8

0.78

0.61

0.65

0.6

Claude 3.5 Sonnet (Old) -

620

650

0.7

0.87

0.9

880

1

0.87

0.9

0.79

0.83

0.63

0.68

ol-preview -

0.37

0.56

0.65

0.84

0.86

0.88

0.87

1

580

0.89

0.89

0.74

0.77

0.5

Claude 3.5 Sonnet (New) -

0.32

0.52

0.63

0.79

80

0.85

60

0.85

1

0.83

0.88

0.69

0.75

0.4

-Lo

0.47

0.56

0.72

0.78

80

0.89

0.83

1

60

0.81

0.85

Claude 3.7 Sonnet -

0.29

0.43

LS0

0.72

0.78

0.78

0.83

680

880

60

1

0.83

0.87

0.3

03 -

0.53

0.6

0.61

0.63

0.74

0.69

0.81

0.83

1

0.93

0.2

04-mini -

0.16

0.57

0.65

0.65

0.68

0.77

0.75

0.85

0.87

0.93

GPT-2 -

davinci-002(GPT-3) -

gpt-3.5-turbo-instruct

GPT-4 0314

GPT-41106-

GPT-40

ol-preview

7Sonnet

-mini

1-0

alias

Correlation of Fractional Excess Success Rates BetweenAgents

1.0

GPT-2

0.49

0.17

0.0044

-0.019

-0.0099

0.015

0.0039

L200-

-0.043

0.041

davinci-002 (GPT-3) -

0.49

0.34

0.048

0.046

0.015

0.025

0.0073

-0.019

1900-

0.062

gpt-3.5-turbo-instruct-

0.17

0.34

0.094

0.15

0.12

0.077

0.1

0.09

0.027

0.11

0.024

0.028

0.8

GPT-4 0314 -0.0044

0.048

0.094

1

0.66

0.75

0.42

0.56

0.34

0.35

0.46

0.19

0.2

GPT-4 1106 -

-0.019

0.046

0.15

0.66

1

0.82

0.56

0.7

0.56

0.54

0.7

0.28

0.4

0.6

CPT-40 - -0.0099

0.015

0.12

0.75

0.82

0.44

0.68

0.49

0.52

0.6

0.25

0.28

Claude 3.5 Sonnet (Old)-

0.015

0.025

0.077

0.42

0.56

0.44

0.54

0.65

0.49

0.64

0.14

0.28

ol-preview -

0.0039

0.0073

0.1

0.56

0.7

0.68

0.54

1

0.72

0.77

0.75

0.26

0.36

0.4

Claude 3.5 Sonnet (New) -

L200-

6100-

0.09

0.34

0.56

0.49

0.65

0.72

0.71

0.74

0.34

ol-

-0.043

1900-

0.027

0.35

0.54

0.52

0.49

0.77

0.71

490

0.45

0.58

0.2

Claude 3.7 Sonnet -

0.041

0.062

0.11

0.46

0.7

0.6

0.64

0.75

0.74

0.67

0.35

0.49

03-

0.024

0.19

0.28

0.25

0.14

0.26

0.45

0.35

0.73

- 0.0

04-mini -

0.028

0.2

0.4

0.28

0.28

0.36

0.34

0.58

0.49

0.73

GPT-2 -

davinci-002 (CPT-3)-

gpt-3.5-turbo-instruct

GPT-4 0314 -

GPT-4 1106 -

GPT-40

Claude 3.5 Sonnet (Old)-

ol-preview-

Sonnet

04-mini

3

alias

LengthoftasksAlagentshavebeenabletocompleteautonomously

for169softwareengineering,cybersecurity,generalreasoning,andMLtasks

2 hrs

03

1 hr, 45 min

50%successrate

1hr,30 min

1hr,15 min

1hr

45 min

30 min

Sonnet3.6

olpreview

Task time (

Sonnet3.5

15 min

GPFPT 4 Nov '23]

GPT2

GPT3

GPT3.5

GPT40

Osec

2019

2020

2021

2022

2023

2024

2025

Modelreleasedate

4 hrs

Claude3.7Sonnet

2hrs

Claude3.5Sonnet(New)

1 hr

Claude3Opus

30 min

★

03

15 min

?

04-mini

50% success rate

←+<←×+<

8min

ol

4 min

ol-preview

GPT-40

2 min

CPT-4Turbo

1 min

30 sec

CPT-4 1106

口

CPT-4 0314

15 sec

gpt-3.5-turb-instruct

8sec

davinci-002[(GPT-3)

Task time (

4sec

GPT-2

Doublingtime:208days

2 sec

★

2019-01-01+ data

1sec

2019

2020

2021

2022

2023

2024

2025

2026

2027

Model release date

LengthoftasksAlagentshavebeenabletocompleteautonomously

for 169 software engineering,cybersecurity, general reasoning, and ML tasks

e (for humans) that model completes with

4 hrs

2hrs

1hr

30 min

15 min

50%successrate

8min

CPT40

4 min

GPT4

2 min

1 min

30sec

GPT3.5

15 sec

8sec

GPT3

4sec

Tasktime(

2sec

GPT2

Doubling time:207days

2018-09-03+data

1sec

R2: 0.97

2019

2020

2021

2022

2023

2024

2025

2026

2027

Modelreleasedate

4 hrs ]

MockAIME

GPQADiamond

1 hr

TeslaFSD

LiveCodeBench

15 min

SWE-benchVerified

MATH

4 min

1 min

osworld

15 sec

WebArena

4sec

RLBench

METR-HRS

(Original Time Horizons)

1 sec

2020

2022

2024

2026

CC-BY

Modelrelease date

metr.org

4.00

Model success

13.33x

strongly related

2.59

to tasklength

5.35x

1.68

2.96x

1.09

(odds ratio)

(ass b0)

2.02x

0.70

1.58x

0.46

β

1.34x

0.29

1.21x

0.19

Model success

1.13x

weaklyrelated

0.12

to task length

1.08x

0.08

Video-MME

LiveCodeBench

Figure 28, time horizon may not be an appropriate metric for these benchmarks. No domain we examined displayed trends were clearly sub-exponential.

For more details, see our blog post:

https://metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/

J Code

The code to reproduce the results in the main body of the paper can be found at: https://github.com/METR/eval-analysis-public

K Author contributions

Thomas Kwa led the development of SWAA tasks, wrote most of the data analysis code, and wrote the plurality of the final draft including generating about half of the figures. Ben West performed the initial analysis, and co-led the project. Joel Becker led human baselining processes and data collection for RE-Bench and HCAST tasks, and contributed to early data analysis code. Amy Deng created most of the SWAA tasks, collected human baselines and AI agent results on the SWAA tasks. She also investigated agent and task failures and assisted with obtaining agent results on HCAST tasks. Katharyn Garcia wrote evaluation and data analysis code. Max Hasin led the qualitative impressions work and wrote key sections, gathered internal PRs, and contributed to data analysis code and baselining collection. Sami Jawhar contributed to the task-running and data analysis infrastructure. Megan Kinniment performed the messiness experiments, excess success rate and initial reliability analyses, oversaw the creation of many HCAST tasks, and contributed to figures and writing. Nate Rush led the SWE-Bench and Internal PR experiments and drafted those sections of the paper. Sydney Von Arx oversaw internal validity analysis.

Brian Goodrich contributed to the development of the modular-public agent, and created the flock and duet agents. Nikola Jurkovic baselined SWAA and RE-Bench tasks and tested RE-Bench tasks. Seraphina Nix contributed to writing and revising the paper and figures. David Rein managed the development and collection of HCAST tasks and oversaw the human baselining process for those tasks. Lucas Jun Koba Sato led the development of the AI agents and collection of agent results

on HCAST tasks. Chris Painter suggested substantial framing and presentation changes. Neev Parikh contributed to running external validity experiments.

Elizabeth Barnes helped develop the basic concept, suggested experiments/analyses, gave feedback, and assisted with writing/early task development/analysis code. Lawrence Chan co-led this project, including setting the overall direction, helping decide what experiments to run, and contributing much of the writing for our paper.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We summarize all important claims, assumptions, results, and limitations, except some minor results in the appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the conclusion for a short summary of our limitations, appendices E.4, B.2, and E.3 for more details, and 4 for results that partially address these limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper is mainly empirical and does not prove theorems.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The models and scaffolds used are described, models are all publicly available (except those which have been deprecated), and raw data and analysis code are provided in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Although we provide basic analysis code sufficient to reproduce the main results from raw data, there are several challenges to sharing all code used; it is split across several repositories, difficult to anonymize, and some benchmarks (both internal and external) are private to prevent data contamination. We believe the analysis notebook provides better usability for reviewers and sufficient reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: For details of our methodology, see 2 and various other sections.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See 3; our error bars are generated using hierarchical bootstrap, which accounts for covariance between similar tasks and thus gives wider, more conservative error bars. For figure 6 we clearly describe the boxplot, which is 80% rather than 95%.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See section C.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All workers were fairly and legally paid and consented to this experiment. Data collected from humans is limited to success/failure and time taken on tasks, and is anonymized to the extent practical.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the impacts of capabilities forecasting in several places in the introduction, conclusion, and appendix, including economic value created by future powerful AI, possible catastrophic risks, and the application to determining appropriate risk mitigation measures. This paper does not develop specific technologies vulnerable to direct negative social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The only datasets we use are either original or previously published.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all benchmarks and data used; the only models or other assets we use are the OpenAI and Anthropic models we evaluate through their APIs.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

Justification: The full text of instructions would be impractical to share, as it includes around 170 separate task instructions for private internal and external tasks, many of which reveal the details of their tasks. As for compensation, SWAA baselines were collected on internal employees rather than contractors or crowdworkers.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: As above, all original human data was collected on internal employees who consented.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were used for exploratory work and editing but not core methods.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper evaluates the performance on agentic tasks (including e.g. SWE and ML tasks) of 12 models released over the past 6 years, and analyses how models get better over time. To do that, it uses RE-Bench, HCAST and a newly created set of short tasks called Software Atomic Actions, and draws on extensive human baseline data comprising hundreds of human baseline trajectories taking in total >2,500 hours by skilled SWEs and ML researchers. Based on this data, each task is assigned a task length (how long humans took on average to solve it). The paper’s main contributions are: a) The definition of a model’s time horizon as the (estimated) length of tasks which the model can do with 50% reliability. This is an intuitive measure of a model’s agentic capabilities. b) The discovery of an exponential trend in models’ time horizons: since 2019, models’ time horizons have doubled approximately every 7 months. The trend approximately holds up under various ablations and can be used to forecast the arrival of future highly agentic models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This is a highly significant and original model evaluation paper. Most evaluation papers focus on a narrow capability and are quickly saturated, as a result of which they cannot be used for tracking and forecasting model capabilities across multiple years and model generations. The discovered trend is simple to understand and highly consequential for timelines to AGI and beyond, and any efforts aimed at preparing society for that transition.\n- The human benchmarking effort that went into this paper (as well as HCAST and RE-Bench), is immense, especially in light of the fact that all the baseliners were experienced SWEs / ML researchers / cybersecurity experts. This makes the time-horizon metric grounded and highly interpretable.\n- The paper is very rigorously done. It has many ablations, e.g. accounting for task messiness, a sensitivity analysis for the final forecast, a follow-up experiment on real-life pull-requests, … Many limitations or follow-up questions I had while reading the paper were discussed in detail in the appendix.\n- The writing is clear, high information-density, and the figures / tables are informative.\n\nWeaknesses (minor)\n- While the logarithmic axis makes sense for longer time horizons, I’m not sure that the distinction between a 1s, 2s, 4s, 8s, 15s, or 30s task is that meaningful or reliably measurable, and so I’m not sure how seriously to take the results for the early models.\n- One interesting observation is that models’ competence at different levels of reliability spans quite a few orders of magnitude in terms of time horizons. E.g. tasks the model can do 90% of the time can be ~60 times shorter than tasks the model can do 10% of the time. An analysis of this in more detail could improve the paper.\n\nQUESTIONS:\n- IIUC, the human baseline methodology is slightly different between the different datasets (SWAA, HCAST, RE-Bench). How worried are the authors that these differences systematically bias the fitted trend?\n- How far do the authors think the trend will continue? Does it seem possible that e.g. week-long or month-long tasks will tend to be more messy, and therefore the trend fitted on non-messy tasks won’t hold up?\n- (Nit) Table 2 would be more informative if the analysis was done on the same set of tasks\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel measurement for assessing AI capabilities: the 50% task-completion time horizon, i.e. the amount of time it takes a human to complete tasks that the AI model achieves 50% success rate. The authors tested over 100 tasks across three datasets. Based on the results, they conducted backtesting and provided forward-looking analyses, such as estimating when AI systems might be able to complete one-month-long software tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed 50%-task-completion time horizon is an interesting and reasonable metric for evaluating AI model capabilities. Unlike traditional success rate or performance-based metrics, it better captures model behavior in more realistic and automated settings.\n\n2. The authors validate this metric across 170 diverse tasks and 12 frontier models. The results are reasonable and informative.\n\n3. The authors further apply the metric to perform verification and exploration, including replicating their measurement on SWE-bench Verified and extrapolating toward one-month-horizon AI. These analysis contributes to both a robust measurement approach and valuable insights.\n\nOverall, using time horizons as a metric provides a fresh perspective on evaluating large language models. Although similar ideas were mentioned in the GPT-4o technical report, this paper stands out by offering a more systematic and comprehensive treatment of the concept.\n\nWeaknesses:\n\n1. The human baseline sample size is relatively small, which could bring significant bias into the overall evaluation.\n\n2. The paper may be better suited for a datasets and benchmarks track.\n\n3. The paper lacks discussion of recent works that also explores LLM capability evaluation beyond traditional success-based performance metrics. For example: Miserendino, Samuel, et al. \"SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?.\" arXiv preprint arXiv:2502.12115 (2025).\n\nQUESTIONS:\nConsidering the high cost of collecting human baseline data, do you have any insights on reducing variance or bias in the measurement process to obtain a more robust result.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposes a new metric to evaluate AI abilities, the 50% task-completion time horizon, which is the time humans typically take to complete tasks that AI models can complete with a 50% success rate.\n- The authors evaluate a dozen or so models on three datasets and compute the task-completion time horizon metric.\n- The authors find that the task-completion time horizon has been growing exponentially in the past few years.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n- I agree with the author's reasoning about why existing metrics are flawed. However, they fail to provide a sufficient justification for why X% task completion time horizon is the “right” metric that the community should be thinking about.\n- The reliance of the metric on human baseliners means the results might depend a lot on the population of humans that is recruited. Can the authors comment on this?\n\nClarity:\n- Since the authors focus on software tasks, why do you only evaluate LLMs on their own? Many software tasks (e.g., SWE-Bench) cannot be solved with one LLM call and rather need more agentic frameworks.\n- While the authors claim to focus on realistic tasks, the collection seems like a somewhat adhoc combination of software-related tasks. More discussion on how the set of tasks covers software engineering development and life cycle would be helpful.\n- I noticed there is consistent writing. For example, L47 states that 13 models were evaluated, L111 says 12 frontier and 3 “near frontier” models, and then L256 states 11 models. The inconsistencies throughout this paper are concerning.\n\nSignificance:\n- I am finding it difficult to understand what the significant takeaways or contributions are of this work and how it implicates future work. The primary contribution of the work seems to be evaluating models on existing datasets and baseliner results with relatively simple methods. \n\nOriginality:\n- This type of metric seems fairly novel in the literature. Most other evaluations do not have an axis of human task completion time.\n\nQUESTIONS:\n- Can the authors discuss why this particular metric was selected and how to mitigate concerns about the population of human baseliners that are used?\n- Can the authors discuss how benchmarking more agentic frameworks might affect their results?\n- Can the authors explain what the primary takeaways from this work are beyond the exponential trends observed in existing data?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Measuring AI Ability to Complete Long Software Tasks\" proposes a new metric called the 50% task completion time horizon, which quantifies how long a task takes a human to complete that an AI model can solve with 50% reliability. By benchmarking 13 frontier AI models from 2019 to 2025 on 170 software and research tasks, the authors find that this time horizon has been doubling approximately every seven months, indicating rapid progress in AI capabilities. The study uses datasets like HCAST, RE-Bench, and a new SWAA suite to evaluate tasks ranging from seconds to hours, comparing AI performance to skilled human baselines. Key drivers of improvement include better logical reasoning, tool use, and adaptability. However, AI agents still struggle with “messier” tasks that lack structure or clear feedback. Authors extrapolate the results to say that by 2028–2030, AI systems may autonomously complete software tasks that currently take humans a month.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n*.- Novel Metric for Long-Horizon Tasks: The introduction of the 50% task completion time horizon offers a fresh, intuitive way to measure AI progress on complex, real-world tasks, bridging the gap between toy benchmarks and practical capabilities.\n*.- Empirical Breadth and Historical Scope: The study evaluates 13 models across six years (2019–2025) and 170 tasks, providing a rich longitudinal view of AI progress, especially in software engineering and research domains.\n*.- Insightful Trend Analysis: The finding that AI capability (as measured by the 50% horizon) doubles every ~7 months is a compelling and quantifiable insight into the pace of AI advancement.\n\nWeaknesses\n*.- Neglect of Task Complexity Growth: The paper does not analyze whether the complexity of tasks increases over time. If task difficulty grows exponentially, the gap between 50% and 100% completion could also grow non-linearly, potentially biasing the metric and overstating progress.\n*.- Potential Benchmark Contamination: There’s a risk that models have been exposed to benchmark data during training, especially given the popularity of certain datasets. This could inflate performance and obscure whether improvements stem from genuine generalization or memorization.\n*.- Limited External Validity and Real-World Testing: While the benchmarks are diverse, the paper lacks a clear discussion of how well these results translate to real-world software development environments, where ambiguity, collaboration, and evolving requirements are common.\n\nQUESTIONS:\nI think that answering the next questions could help to improve the paper:\n\nHow do you account for the potential exponential growth in task complexity over time, and could this affect the interpretation of the 50% completion time horizon metric?\n\nWhat measures have you taken to ensure that the models evaluated were not exposed to benchmark data during training, and how do you mitigate potential contamination effects?\n\nCould you analyze or report how the gap between 50% and 100% task completion evolves, especially for longer or more complex tasks?\n\nHave you considered evaluating model performance on unseen or adversarial variants of the benchmarks to better assess generalization?\n\nCan you provide evidence or case studies showing how well the 50% time horizon metric correlates with real-world software development performance?\n\nLIMITATIONS:\nI would recommend the authors to read the paper: Larger and more instructable language models become less reliable (https://www.nature.com/articles/s41586-024-07930-y) as maybe the models work better and better with complex problems but forget the simple and routine ones (much more useful) because the large models have become too focused on solving increasingly complex problems (leaving aside the comments of possible contamination).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}