Measuring AI Ability to Complete Long Software Tasks
Authors: Thomas Kwa
Ben West
Joel Becker
Amy Deng
Katharyn Garcia
Max Hasin
Sami Jawhar
Megan Kinniment
Nate Rush
Sydney Von Arx
Ryan Bloom
Thomas Broadley
Haoxing Du
Brian Goodrich
Nikola Jurkovic
Luke Harold Miles
Seraphina Nix
Tao Roa Lin
Neev Parikh
David Rein
Lucas Jun Koba Sato
Hjalmar Wijk
Daniel M Ziegler
Elizabeth Barnes
Lawrence Chan
Venue: NeurIPS 2025 poster ()
Keywords: benchmarks
evaluations
Natural language processing
Software engineering
forecasting
Primary area: evaluation
TL;DR: The length of software tasks AIs can complete has doubled every 7 months since 2019

Abstract: Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as o3 have a 50% time horizon of around 110 minutes. Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated since 2024. The increase in AI models’ time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes, combined with better logical reasoning and tool use capabilities. We discuss the limitations of our results—including their degree of external validity—and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.

Measuring AI Ability to Complete Long Software Tasks
Thomas Kwa', Ben West"", Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx
Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler
Elizabeth Barnes, Lawrence Chan
Model Evaluation & Threat Research (METR)
Abstract
Despite rapid progress on Al benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms human capabilities, we propose a new metrie: 30%--aask-complete zon, the time humans typically take complete tasks that Al agents can complete with 50% success rate. We first timed humans with relevant domain expertise combination of RE- Be-BBechh, HCAST, and 66 novel shorter tasks. On these tasks, agents built using current frontier AI models such as o3 have 50% time horizon of around 110 minutes. Furthermore, frontier AI time horizon has dou- bled approximately every seven months since 2019, though the trend may have accelerated since 2024. The increase in Al agents' time horizons seems to be pri- marily driven by greater reliability, ability to adapt to mistakes, logical reasoning. and capacity for tool use. We discuss the limitations of our results-including their degree of external validity-and and the implications of increased autonomy for dangerous capabilities, these results generalize to real-world software tasks, ex- trapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.
time hori-
Introduction
In the last five years, frontier Al systems have undergone a dramatic transformation in capabili- ties, evolving from basic text generation [1] to autonomously executing complex multi-hour ma- chine learning research projects [2]. Sufficiently capable AIs could perform dangerous, highly com- plex actions like autonomous development chemical, biological, radiological or nuclear weapons (CBRN) and self-replication and adaptation outside human control [3]. Understanding Al capabili- ties helps inform the development of safety guardrails as systems become increasingly powerful. In particular, many frontier Al developers have committed to using measures of specific AI capabili- ties to determine the necessary risk mitigations for their frontier AI systems. Robust benchmarks that accurately track and forecast Al capabilities thus form the foundation for responsible AI governance and risk mitigation.
"Equal contribution.
Corresponding author, thomas www.metr.orrr..mo......... org.
t 'Ohm Chip. Work done at METR.
S Anthropic. Work done at METR.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Length agents able complete autonemously
for 169 software or ongindriingg syborsacuuity, general reasoning, and ML tasks
Tass timee ("or hummens)) that model complletes with
41/5
30
50% Success rate
mmh
GPT
30
GPT 3S
15
GPT2
Doubling time: 207 days 2018-09-03+ data
1900
0790
2020
2023
2025
2027
Model rellesse date
Figure I: The length of tasks (measured by how long they take human professionals) that generalist autonomous frontier model agents can complete with 50% reliability has been doubling approx- imately every months for the last 6 years (Section 3). The shaded region represents 95% CI calculated by hierarchical bootstrap over task families, tasks, and task attempts. Even if the absolute measurements are off by a factor of 10, the trend predicts that in under a decade we will see AI agents that can independently complete large fraction of software tasks that currently take humans days or weeks (Section 5).
However, existing benchmarks face several key limitations. First, they often consist of artificial rather than economically valuable tasks. Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans," ! biasing the comparison to human performance. Most critically, individual benchmarks saturate increasingly quickly [6]. and we lack more general, intuitive, and quantitative way to compare between different benchmarks, which prevents meaningful comparison between models of vastly different capabilities (c.g.. GPT-2 versus Claude 3.7 ). As consequence, while the last few years have dramatic increases in Al perfor- mance on many individual benchmarks, understanding the progress of AI capabilities in general has required estimating the qualitative difficulty of the latest benchmarks AI systems can pass.
the duration of tasks that Al agents can complete at certain success probability, providing intuitive measure of real-world capability compared to humans. As AI agents based on current models may not reliably complete all tasks of given length, operationalize this by measuring the X X%-(task completion) time horizon the length tasks that Al agents can complete approximately X% of the time.
We propose tracking Al progress over time using the task completion time horizon:
We prototype this methodology using three datasets designed to capture skills required for research or software engineering (Section 2.1), totaling 170 tasks with wide range of difficulty: general software agent tasks from HCAST [8], machine learning research engineering tasks from RE-Bendi [2], and Software Atomic Actions (SWAA), new suite of shorter software tasks that provides signal on pre-2023 models (Appendix 1.1.3). Using skilled human baseliners, we estimate the duration that a domain knowledgeable human (without task-specific context) takes to complete these tasks (Section 2.2). We evaluate the performance of AI agents based on 12 frontier models from 2019 to 2025 on these tasks (Section 2.3). Using methodology inspired by human psychometric studies, we then estimate the duration of tasks that models complete with 50% success rate-the 50% time horizon (Section 3.1).²
For example, HellaSwag [4] and Humanity's 's Last Exam [5] were both generated by adversarially filtering problems against the best performing language models available at the time.
PSWE-bench Verified [7] does come with outan-estimated task completion times. We SWE-bench Verified tasks and accompanying time estimates to validate our main result in Section 4.1.
'The code to reproduce our results can be found at: https: // thub. com/METR/ eval wal-amalyeis-publy

---

Diverse Task Suite
Task Performance
HCAST
Human Runs
Agent Runs
Diverse that
agency
1hre hrs
trm
Time Estimatee
Success Rate
SWAA
Singre stop tasks sempled from SWE work
Time Horizon Analysis
+-30
66
Find Time Horizon
Hortzon
Doubling Time
RE-Bench
Al R&O Buswarch ch Eng neering taaks
Model Release Dates
7 tasks
Figure 2: Our methodology for measuring Al agent time horizon. First, we create a diverse task suite of 170 tasks. Second, we have both humans and AI agents (consisting of an Al model and scaffold) attempt these tasks, recording the time taken by successful humans and the success rate for Al agents. Third, we fit logistic model find the time horizon which each Al agent has a 50% chance of success, and plot this against the release date of the model.
We find that the 50% time horizon has been growing exponentially from 2019-200000 2025 on these tasks, with doubling time of approximately seven months (Figure 1). We compare our main result with our exploratory results on non-SWAA tasks, finding that the 2023-2025 horizon growth rate is about 20% faster than the 2019-2025 rate (Section F.I). We also measure the 80% time horizon of Al agents (Figure 17) and find a similar trend, though horizons are roughly 5x shorter. This progress appears be driven by several key factors: improved logical reasoning, better tool use, and greater reliability and self-awareness in task execution (Section 3.3). We also note several limitations of current systems-notably, performance is much lower less structured, "messier" tasks (Section F.2).
Since tasks do not perfectly represent the average segment of intellectual labor by researchers and software engineers, this raises the question of external validity (Section 4): whether the exponen- tial trend holds real-world tasks. We include four supplementary external validity experiments, which find little evidence of performance trends heing slower on the somewhat more realistic tasks tested, but do not rule out the possibility that trends meaningfully slower the distribution of tasks required to automate software engineer jobs. We also find evidence that Al agent time horizons can differ by a large factor depending on the task domain and reference human population.
We conclude by discussing implications for AI capabilities forecasting (Section 5). Naively ex- trapolating the trend in horizon length implies that AI will reach time horizon of > month (167 work hours) between mid-2028 and mi3-2031 (Figure 6). However, extrapolation is affected by both external validity concerns and future changes in the trend.
1.1
Related Work
Here, we briefly discuss related work. See Appendix A for expanded discussion.
Agentic AI capability benchmarks Many recent benchmarks aim to measure the capability of AI systems to act as agents general, including AgentBench [9], ToolBench [10], GAIA [11], TheAgentComppny [12], the Berkeley Function Calling Leaderboard [13]. In this work, we study AI agent performance on RE-Bench(22, HCAST [8], and SWE-Bench Verified [7], which focus on machine learning and software engincering tasks. Other similar benchmarks include MLAgent- Bench [14], MLEBench [15], and DSBench [16]. In concurrent work, Miserendino et al. [17] create the SWE-Lancer benchmark. consisting of 1400 freelance software engineering tasks gathered from Upwork with real-world payouts totalling above SI million.
AT capability forecasting This work builds on previous efforts forecast the capabilities of AI systems over time, including efforts to contextualize benchmark performance using human experts

---

Family find shell script
Length
Description
3 seconds
shell script?" Choices: "run.sh", txt". "run.py". "run.md"
Multiple choice: "Which
rexeardia.eeeearch
minute
Research simple factual information from Wikipedia and pro- vide accurate answers straightforward questions.
oxdna_simple
9 minutes
Detect and fix a bug in the input files for molecular dynam- ics simulation using the oxDNA package.
munge data
56 minutes
Write Python script transform JSON data from for- mat to another by inferring the conversion rules from pro- vided example files.
Speed up a Python backtesting tool for trade executions by implementing custom CUDA kernels while preserving all functionalityy aiming for a 30x performance improvement. Table 1: Example tasks for different durations; see Rein et al. [8] and Wijk et al. [2] for more examples.
cuda backtesting
8 hours
Phuong et al. [3]; Murray et al. [18] and efforts to forecast performance on particular benchmarks Owen [19]; Pimpale et al. [20]. Contextualizing AI capabilities using human time horizon was proposed by Ngo [21], and similar concepts were discussed in Carlsmith [22] and Cotra [23].
Human psychometric methods Our methodology draws inspiration from human psychometric testing, especially Item Response Theory [24; 25; 26).
2 Measuring AI agent performance on realistic tasks
2.1 Task suite / dataset
Our tasks comprise three distinct task suites:
1. A subset of HCAST [8]: 97 diverse software tasks ranging from I minute to 30 hours. 4
2. RE-Bench [2]: difficult ML research engineering tasks, all eight hours long.
3. Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from second to 30 seconds.
All tasks are automatically scored with continuous score binary threshold; details of how normalize and process scores are given in Section 3.1, and other task suite details in Appendix B.
Example tasks See Table for five example tasks. Tasks under 1 minute measure knowledge rel- evant to professional software engineering. At 10 minutes, tasks can include the easiest meaningful step of real software project (e.g. configuring common open-source package). The shortest tasks that could be standalone economically relevant projects take around one hour; by eight hours tasks represent valuable software projects. Many more examples of these tasks available in Wijk al. [2) and Rein et al. [8].
Baselining
In order to ground AI agent performance, we also measure the performance of multiple human "baseliners" on most tasks and recorded the duration their attempts. total, we use over 800 baselines totaling 2,529 hours, of which 558 baselines (286 successful) come from HCAST and RE- Bench, and 249 (236 successful) from the shorter SWAA tasks. 148 of the 169 tasks have human baselines, but we rely researcher estimates for 21 tasks in HCAST.
Our baseliners are skilled professionals in software engineering, machine learning, and cybersecu- rity, with the majority having attended world top-100 universities. They have an average of about years of relevant experience, with software engineering baseliners having more experience than ML or cy cybersecurity baseliners. For more details about baselines, see Appendix C.I.
'Our results also include one task from GAIA [11], and five tasks involving writing code that is robust to an adversary.

---

Model Success Rote vs Human Completion Time
Average Success
Autonomyr SWAAt RE-Bench
Tasks diversit--weiiiittee
0225
Mear Model Eucoees Datte
0.6
201
02
Cluat Faius
oet Tantu
Clson la Seveett grag
CPP
digrenivv
devind end
Claue 35 Server
00-
Human Twwee-to-Cooe Comppeee
Figure 3: Left: Average task success rate across the entire combined suite, for each model. As with all of the results reported in the main body of this work, to reduce the influence of large task families, we weight each task by the inverse square root of the number of tasks the family belongs to. Right: Model success rates are negatively correlated with how much time it takes a human complete the task. (y- 0 + 0.66, Ri 0.83)
2.3 Results
We measure 12 frontier (and 4 near-frontier) models released between 2019 and 2025; full infor- mation on the models and scaffolds used can be found in Appendix C.3. We perform 8 runs per agent/task pair and report the average results Figure 3. There is a strong upwards trend over time, with recent models completing approximately 50% of all tasks, while earlier models perform substantially worse. We find substantial correlation between the tasks that models can complete (Figure 22), with average correlation of approximately 0.73.
There is negative correlation between the time it takes human baseliner complete a task and the average success rate (across all models) on the task. This decrease in success rate over length (Figure 3) is well-fit by an exponential model (R² x 0.80 when regressing model success rate against the logarithm of human time-to-complete). As expected, more recent models can complete longer tasks, with Claude 3.7 Sonnet and o3 completing some tasks that take human baseliners >4 hours (Figure 4).
3
Time horizon
3.1 Computing time horizon
To convert agent run data to task completion time horizon, we first convert the agent performance on each task to a binary value (success failure). Many tasks are naturally binary, including all SWAA tasks. Continuously scored tasks are binarized via a task-specific threshold chosen to represent human performance. For HCAST, the task-specific threshold is the same "target score" the human baseliner tries to achieve, which we also use to filter for successful runs. RE-Bench tasks have fixed time rating of hours, so the task-speeific threshold is the average score of 7-9-hour human runs.
Once we have agent success rates and time ratings for each task, fit time horizons by performing the following logistic regression.
Psuccess (agent, task) ((log Magent - log ( Pugent)
'This number is approximate, because small number of runs failed due to internal infrastructure issues. 'Compare Item Response Theory (IRT) [24]. Like IRT, we use logistic regression find the task difficulty at which the agent has a 50% chance of success, but unlike IRT, we use difficulty ratings directly based on human baseline time rather than ratings learned from agent performance. Further details and comparison to standard IRT methods provided in Appendix C.4.
5

---

Lerigth agents
complete autonsmousy
Chrude 37 Son
Claude 35 Sonnet Niww
Cloude So Joid1
Claude Opus
PTP
VITET
min
3111813 1u1r0 wu1!!
18855 1000 8011800 s11f1
11111
0P1-/106
CF D31
Gavino sprc
0012
38311830 JAStEfE8 33511f30 4852 Talk srrgen Jiumin (rm))) Tadle Ningth ths.mantin timdd
Task lingth Dhaman tird)
Task ungtn jimmantimn
Tap Ningtt Dhuman (ima))
Figure 4: Success rates of all models on the test suite, showing the computation of time horizon as predicted 50% success rate time. The logistic fit is fairly good, though there is a jump in success rate between minute SWAA tasks and > minute HCAST tasks.
where lask the geometric mean time of successful human baselines, and hapont and Bagont are learned parameters, with hagent representing the 50% time horizon.
3.2 Time horizon vs. release date
In Figure 1, we plot the time horizons of each model against their release thee-the date at which the lab first publicly announced the frontier model.? In addition, we linearly regress* log(time horizon) against release date, finding that time horizon has doubled every 207 days with 95% bootstrapped confidence interval 166-240 days (roughly 119).. Error bars are calculated via 10,000 samples from three-level hicrarchical bootstrap over task families, then tasks, then runs.
While there are wide error bars on each individual models' horizon lengths, these errors are highly correlated between models. This is because tasks at the same human time rating vary widely in difficulty for models, and sampling easy (or hard) tasks will result in higher (or lower) horizon estimate for all models. Therefore, we are more confident in the slope of the time horizon trend than in the time horizon of any particular model. The fit is not sensitive to various hyperparameters such as regularization, weighting of tasks, and WLS vs. OLS (see Figure 6).
Horizon length increases substantially over the entire time period from 2019 to early 2025. GPT-2 has 50% time horizon of only 2 seconds, while o3 has a 110-minute time horizon and succeeds several tasks over 4 hours. Also, o3 lies above the long-run trend (p- 0.006); this is robust to methodological ablations like using continuous scoring (Appendix H), which may imply the trend in 2024 and early 2025 is faster.
In most cases, the release date is for each frontier AI model is the same model the evalu- ated. However, GPT-3 (davinci) and GPT-3.5 (code-davinci-002 and text-davinci-002) are closed-source and no longer available through API access, but we use their release dates for the closest available models, which OpenAl advertises as having equivalent performance: GPT-3's release date for davinci-O0P and GPT-3.5's date for Sert3.--tubb-insttee
*Specifically, we perform Ordinary Least Squares regression on log(model horizon) = a 3 release date. In Appendix H we discuss other curve-titiing methods, and conclude that the fit is not sensitive to various hyperparameters such regularization, weighting of tasks, or WLS OLS.

---

Table 2: Categorization of 31 failed runs by GPT-4 1106 and 32 failed runs by ol (Section 3.3). Note that as ol suc- ceeds at more tasks, its failures corre- spond to more challenging tasks com- pared to GPT-4's failures.
Failure type
GPT-4 1106
ol
Poor planning/tool choice
Incorrect mental mathvreasoning
6
7
Premature task abandonment
8
16
Repeating failed actions
12
2
Other
Total
31
32
3.2.1 Time horizons at 50% % success rate vs 80% success rate
To check whether our choice of 50% success rate affects the long-run trend, we also compute the time horizon at which AI agents succeed at tasks with 80% success rate, shown in Figure 17. The doubling time in 80% time horizon (204 days) is similar to the doubling time of 50% time horizon (207 days), within margin of error. However, models' 80% time horizons are 4-6x shorter, suggest- ing that even models that sometimes succeed on difficult and diverse tasks cannot reliably perform tasks of moderate length. Due limited amount of tasks of sufficiently short length, we cannot confidently measure time horizons at very high success rates (e.g. 95%) see Section E.3 for more discussion.
3.3 Qualitative analysis
By examining transcripts from 31 randomly sampled unsuccessful GPT-4 1106 agent runs and 32 ol runs, we observe that models seem to have improved greatly in terms of tool use capabilities, demonstrate a markedly greater ability to adapt mistakes (as opposed to repeating unsuccessful actions), and perform much better at parts of tasks requiring logical reasoning or code generation. However, noticed that Al agents still seem to struggle in intuitively "messier" environments- specifically, environments without clear feedback loops, or where the agent needs to proactively seek out relevant information.
Table 3.3 shows the categorized failures by model. We provide definitions of these categories, as well as examples of qualitative improvements and continuing limitations in Appendix D.
4 External validity and robustness
We first check whether the 2023-2025 trend without the SWAA dataset retrodicts the trend since 2019, and find that the trends agree (Appendix F.1). We also perform three experiments to check external validity.
First, we replicate our methods on SWE-bench Verified. We find that the exponential trend still holds (Figure 4.1), albeit with an even shorter doubling time, possibly because SWEhbench Verified diffi- culty annotations, meant represent high-context maintainer time, may differentially underestimate how long human contractors take to perform easier SWE-bench tasks.
Second, we aim to capture some systematic differences between the tasks studied here and hard"real-wond tasks. Some examples include whether the task is resource limited, novel, involves dynamic environment (Section H.6). Controlling for task length, we find models perform worse on tasks that have higher messiness scores. Notably, trends in AT agent performance over time similar for the lower and higher messiness subsets of these tasks (see Figure 12). In particular, we find no evidence of plateaus performance trends specific the higher messiness subset.
the HCAST and RE-Bench tasks against list of 16 "messiness" factors that
Third, we measure AI agent performance on a small, uncontaminated set of our internal pull re- quests (PRs) (Section C.2). We find large differences in the speed which different human groups complete the internal PR tasks, with contractors taking 5-18x longer to fix issues than repo main- tainers. We also find that AI agent performance is worse than would be predicted by maintainer -ome-to-complet as a measure of task length, but is consistent with contractor time-to-complete,
In addition, following the original publication of our paper, we performed some preliminary analysis exploring the time horizon of models on other domains, including math competition, video QA, and autonomous driving (Appendix I). We found that while the specific estimates of the current time
7

---

Time Horizon for SWF-Ranch Verifle TAskS
Figure 5:
Cleadi Sernec (Now)
Performance of frontier AI models using reported SWE-bench Verified results (Section 4.1). We observe
Gludi
GPT-4
GPT4
similar exponential trend to Figure 1, albeit with steeper slope.
auccass
50%
cau leves Medal Doubling
Doubling o tamw 70 pays RP ou7
205
2004
205
Model release
horizons of models sometimes differ between domains, most of these domains also show exponential trends with similar doubling times (Figure 27).
SWE-bench Verified information is provided below. Details of the retrodiction and messiness anal- ysis can be found Appendix F, and details of the internal PRs experiment C.2. Details of our cross-domain preliminary analysis can be found in Appendix I.
4.1 SWE-bench Verified
To check whether observe similar performance trends other benchmarks, apply methodology to SWE-bench Verified, an industry standard benchmark for evaluating language model performance software engineering tasks [27]. All tasks in the SWE-bench Verified dataset were harvested from large open source repositories, like matplotlib or django, and then filtered to ensure they are automatically checkable and well-specified [28].
Model time horizon computed from SWE-bench Verified tasks seem to follow an exponential trend from late 2023 through 2024. However, while the doubling time predicted by HCAST SWAA + RE-bench using 2024 models is 143 days, the doubling time on WWE-bench Verified results is ahouter-around 70 days.
SWE-bench Verified time annotations were based on the expected time it would take an "engineer who has had few hours to familiarize themselves with the codebase" to solve the issue [28]. We found that annotator time estimates differentially underestimate how long our contract baseliners take to complete the easiest SWE-bench verified tasks. As result, our time horizon estimates for SWE-bench Verified (which use annotator times) are likely to underestimate the time horizon less capable models relative to contractor times, in turn shortening doubling times. For more details see Appendix H.5.
Extrapolation
Extrapolating towards une-month-horizon AI To forecast when Al systems may be capable of autonomously producing large economic value and potential catastrophic actions, we chose one month (approximately 167 working hours for fair comparison with humans, since humans not work 24/7) for two reasons. First, Ngo [21] writes that a 1-month AGI (defined as an Al that outperforms most knowledgeable humans who are given month of work hours, i.e. 167 hours, to perform the task) would necessarily exceed human performance at economically valuable endeavors like writing large software applications. founding startups, and making novel scientific discoveries." Second, one month is around the period when new hires at company begin to complete onboarding and generate economic value, and an AI with a horizon of month could be capable of quiring context like a human employee, allowing it to complete high- nigh-context as well as low-context
Note that our forecasts concern AI with 1-month horizon on software tasks, not 1-month AGI, because evaluate models only software and research tasks. Nevertheless, given past correlations in different areas Al performance, 1-month (167 hours) time horizon AI may significantly generally capable. "Obooarding "can last from few weeks to more than a year" [29], and employees often start generating economic value midway through the onboarding process.

---

Uncertainty extrapolated date of1 month Al (167 working hours)
Bootstrap [tasks]
Rootstrap [runs]
Bootstrap (models)
Weighting//egu/lliizttii
Baseline Noise
Overall (2019-2025 trenc]
Overall (2024-202) 2025-202520 trend)
2008
3027
2008
2009
20/00
Extrapolated month working hours)
Figure 6: A sensitivity analysis of the extrapolated date at which frontier AI systems will have horizon of I month. In each row, apply 10,000 random perturbations our data and find the distribution over the date of I-month AI implied by the perturbed data. Box endpoints represent the 25th and 75th percentiles, and whiskers the 10th and 90th percentiles, with outliers not displayed. Note that this plot does not account for future changes in the trend or external validity concerns, which are responsible for the majority of our uncertainty.
tasks. We cannot rule out that system capable of 1-month tasks, even 50% reliability, could be transformative for our society, potentially including the ability for catastrophic harm.
Sensitivity analysis Figure shows a sensitivity analysis of our results over various sources of noise. "Bootstrap (tasks)", "Bootstrap (runs)" and "Bootstrap (models)" represents the limited num- ber of tasks, run-to-run variance, and limited models respectively. "Wiighiingrregulaiizttoon repre- sents methodological choices; this perturbation includes 10 hyperparameter combinations with and without task diversity weighting, and with a logistic regression regularization parameter between 0.01 and 0.2. "IID Baseline Noise" multiplies every task duration by random factor based on the empirical distribution of baseline times. Note our confidence in the 2024-2025 only trend low because there are only seven frontier models in this time span.
Because our analysis puts low probability on the growth rate of time horizon on our tasks being much slower than one doubling every 8 months, the uncertainty in the extrapolated date of 1-month Al fairly small (80% CI width about years, central estimate mid-2029). future progress instead follows the 2024-2025 trend, I-month AI would arrive sooner, with half the probability in 2027 and even late 2026.
is possible that systematic biases and alternate methodologies have a greater impact on forecasts, and we discuss these in Appendix H. In addition, due the inherent difficulty of predicting the future, real forecasts will have larger emor than this naive extrapolation, which we discuss below.
Task distribution limitations We note several differences between this task suite and realistic tasks in Appendix B.2: in short, realistic tasks rarely have automatic scoring, and often require dealing with other agents, resource constraints, dynamic environments, and high reliability require- ments. Such differences cast doubt whether the rapid performance improvements task suite (and other benchmarks) will generalize. It may be that benchmarks like HCAST and SWE- Bench Verified are insufficient for forecasting real-world AI capabilities, and accurate forecasting will require deliberate construction of more realistic benchmarks without these limitations.
on this
Future changes in time horizon trends The time horizon trend may significantly change in the future due to factors including agency training, compute scaling, and automation of Al research and development. Agency training and eventual automation AT R&D are more likely to speed up the trend, whereas compute scaling could slow it (but be partially offset by algorithmic improvements);

---

future artificial general intelligence (AGI) could technically have an infinite time horizon. See Appendix E for details.
Discussion
Summary In this paper, proposed intuitive, quantitative metric for Al capabilities: the task completion time horizon, which relates Al performance on tasks to the typical length of time human experts require complete the tasks. We constructed dataset of 66 shorter SWAA tasks, combined these with existing benchmarks, and conducted baselines estimate their difficulty. To measure the trend in time horizon, benchmarked 11 frontier AI models released between 2019 and 2025 this dataset, and found (Section 3) that the 50% task completion time horizon on these tasks has been growing exponentially from 2019-2025 with doubling time of approximately seven months (Figure 1).
We explored the robustness and external validity of our results in several ways: we confirmed that similar trend also holds on the 80% time horizon (Figure 17). We performed preliminary analyses of 50% time horizons for other domains in Appendix I, and found that similar trends hold across other domains. That being said, we noted important limitations current systems, particularly their lower performance on less structured, "messier" tasks (Appendix F.2), complicating naive interpretation of our results.
Finally, we attempt to extrapolate the trend on these tasks to one-month (167 hours) Al (Section 5), finding that the trend continues and observed performance trends generalize to real-world tasks, an 80% confidence interval for the release date of AI that can complete 1-month long software tasks spans from mid-2028 to mid-2030 (Section 5)- or even as soon as early 2027 if the 2024-2025 trend continues.
Limitations and future work Even though we believe the exponential trend to be robust, we caution that time horizon is always measured relative to a domain, task distribution, and human baseliners' level skill and context, and can difficult to measure practice, especially for longer horizons or success rates near 100%. We discuss nuances with interpreting time horizon in Appendix E.3. Finally, in Appendix E.4, we discuss ways of extending or improving our results, such by studying multiple domains in detail, more models, better elicitation, inference scaling, or other statistical methods.
Acknowledgments and Disclosure of Funding
The authors thank the following reviewers for feedback on draft versions of this paper. Ryan Green- blatt, Aaron Scher, Romeo Dean, Mike Knoop, Jeff Wu, Steve Newman, Rohit Krishnan, Taren Hemebrickner-Kauffnas
JS Denain, Jacob Pfau, Seb Krier, Anton Troynikov, Max Henderson, Ajeya Cotra, Max Nadeau, Tamay Besiroglu, Nate Thomas. The authors thank Charles Foster and Michael Chen for their support with the publication.
We especially thank the following reviewers for substantial feedback: Sara Fish, David Duvenaud, Eli Lifland, Holden Karnofsky, and Rif A. Saurous. We also thank Stephanie He for her graphic design work on Figure 2, and Ryan Greenblatt for input on the scoring methodology.
10

---

References
[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[2] Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-Bench: Evaluating frontier Al R&D capabilities of language model agents against human experts. arXiv preprint arXiv:241..5114,225 2024.
[3] Mary Phuong. Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.13791, 2024.
[4] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800 2019.
[5] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. Humanity's last exam. arXiv preprint 1rXiv:2501.14244 2025.
[6] Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Bryn- jolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. Artificial intelligence index report 2024, 2024.
URL https:://axtiii wwwp//ii..../oiieoooo abs/ 19522.
[7] Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Ley- ton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench ver- ified. https:/ http://pppaa...e/ wttpop...om/inne........pooo com//nndexxintrrouccinng wwwprrdddcciinnssseeeeeeeeeeeeee cessed: 2025-02-26.
2024. Ac-
[8] David Rein, Joel Becker, Amy Deng, Seraphina Nix, Chris Canal, Daniel O' O'Connell, Pip Arnott, Ryan Bloom, Thomas Broadley, Katharyn Garcia, Brian Goodrich, Max Hasin, Sami Jawhar, Megan Kinniment, Thomas Kwa, Aron Lajko, Nate Rush, Lucas Jun Koba Sato, Syd- ney Von Arx, Ben West, Lawrence Chan, and Elizabeth Barnes. HCAST: Humann-alibbran Autonomy Software Tasks. Forthcoming. 2025.
[9] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai. Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agenthench: Evaluating llms as agents, arXiv preprint arXiv:2308 2308.03688, 2023. URI. https: /ttp///axvvvv/ve//////eeeeeaaav/ wwwpe/iiiooo//ooaeeeeeeeeee org/ /2308 03688.
[10] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Ttoblehoolleench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.
(11) Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. The Twelfth International Confer- ence Learning Representations, 2024. URL https:// http:///ppperrreeeeeeettorrrrdd fibxvahve3.
[12] Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking I1m agents on consequential real world tasks. arXiv preprint arXiv:2412. 14161, 2024.
[13] Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla cs. berkeley.
B__erreeeeyyeeeeeeeeeeeeeerreeee wiaeelloyyyucttioonnoom atw.erccccaaaallliaeeeeeeerrrrre www.ereoarddddeeee.o.o.......eae html, 2024.
11

---

[14] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: evaluating lan- guage agents on machine learning experimentation. Proceedings of the 41st International Conference Machine Learning, ICMI'24, JMLR.org. 2024.
[15] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench: Evaluating machine learning agents on machine learning engineering, 2024. URL https: //axxxxivv.///////////e////aavva waaxiv.org/ae prrg/abs.. aataaaa///41 07095.
[16] Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents to becoming data science experts? arXiv preprint arXii:24099 409.07703, 2024.
[17] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can fronticr Ilms carn million from real-world freelance software enginecring", 2025. URL https://arxie: org/ abs/2502. 12115.
[18] Malcolm Murray, Henry Papadatos, Otter Quarks, Pierre-Frraçois Gimenez, and Simeon Cam- pos. Mapping benchmark data to quantitative risk estimates through expert elicitation, arXiv preprint arXiv:2503.04 042999, 2025.
[19] David Owen. How predictable is language model benchmark performance?, 2024.
[20] Govind Pimpale, Axel Højmark, Jérémy Scheurer, and Marius Hobbhahn. Forecasting frontier language model agent capabilities, 2025. URL. hwtpt:ttttt///tttee wwwp:iiivvvrg//bbbe.beee wwwp///gg/b//....a/////////// 15850.
[21] Richard Ngo. Clarifying and predicting AGI.
1ew.wrong........oommom...oo www.swww...ooooooottt...... com/poats/
2023. Accessed: 2024-03-21.
[22] Joseph Carlsmith. How much computational power does take to match the hu- man brain?, 2020. URL https://wwv.c http://wwwwopppphllatorroooo org/research
how-much- potpod-ooopttatiinllppooooeee -otrr---------------------------
tooos-it-takketov mato--matcchtttt whw-ttttee-hhaaaiili.e
Open Philanthropy report, August 14, 2020.
[23] Ajeya Cotra. Draft report on Al timelines, 2020. URL org arg/posts/Kan www.oozpppppr/aarraaaaaaa timelines; technical report.
frrmm on- ai-timelines. Draft report on AI
[24] Frank B Baker. The basics of item response theory. ERIC, 2001.
[25] R. J. de Ayala. Handbook of Item Response Theory: Models, Applications, and Issues. Guilford Press, 2017.
[26] Fernando Martinez-Plumed, Ricardo B.C. Prudêncio, Adolfo Martínez- and José Hermández-Oralle Item response theory in Al: Analysing machine learning classifiers at the instance level. Artificial Intelligence, 271:18-42, 2019. ISSN 0004-3702. doi: https://
09.004. URL. URL.ttep:://ww..cceeeedeeee www.ereeeettttccteeeeeeeeeeeeeee com/sciem/et scinccience///// arricleepp1//00447000002222
[27] OpenAl. OpenAl o3-mini. cessed 18-03-2025].
2025, [Ac-
(28) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024. URL https:
// //openreview
[29] Dave Ziclinski. How to optimize onboarding, June 2019. URL https://wwat shrm. ceweiizi-oobbrrrdingg Accessed on
March 17, 2025.
[30] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bow- man. GLUE: A multi-task benchmark and analysis platform for natural language under- standing. International Conference on Learning Representations (ICLR), 2018. URL. com/.
12

---

[31] Alex Wang, Yada Pruksachatkun, Naman Nangia, Amanpreet Singh, Julian Michael, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language un- derstanding systems. In Advances in Neural Information Processing Systems (NeurIPS) Work- shop, 2019. URI
com/-
[32] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. CoRR, 0ts/20009033000 org wwrg/abb/20099 03300.
2020. URL
[33] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong. Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating large lan- guage models to master 16000+ real-world apis, 2023. URL https:/ tttps://arxiv. aarxiv.ivvv..... org/ / 2307.16789.
[34] Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vias Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, Vishaal Udandarao, Jingyi Lu, Shiyang Chen, Sam Purkis, Tianshuo Yan, Wenye Lin, Gyungin Shin, Qiaochu Yang. Anh Totti Nguyen, David I. Atkinson, Aaditya Baranwal, Alexandru Coca, Mikah Dang, Sebastian Dziadzio, Jakob D. Kunz, Kaiqu Liang, Alexander Lo, Brian Pulfer, Steven Walton, Charig Yang. Kai Han, and Samuel Albanie. ZeroBench: An impossible visual benchmark for contemporary large multimodal models, 2025. URL https:
//arxiv. org/ abs 2502. .96996.
[35] [Author] Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capa- bilities of language models. arXiv preprint arXiv:2206.04611 2022.
(36] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code, 2021. URL httts::///attiiii org/ abs/ 03374. arXiv preprint, arXiv:2107.03374.
[37] Jeremy Austin, Augustus Odena, et al. Program synthesis with large language models, 2021. 07732. arXiv preprint, arXiv:2108.07733
URL
[38] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He. Dawn Song, et al. Measuring coding challenge competence with apps. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Truck, 2021. URL httts:: pesttss:/arztie www.xxwwwwwgg..///oooeeo. httg///b/2100.0000
[39] Dario Amodei and Danny Hernandez. AI and compute, 2018. URL httpsss/:/oppe blog/ ailaaai-nddda computet../ OpenAl blog post, May 16, 2018.
[40] Epoch AI. Data on notable AI models, 2024. URL httpps:/eppch..i//ae
notable-ai-me ammdeelss.. Accessed: 2025-03-04.
[41] Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Vil- lalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2022.
[42] Andy Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.0892 08926, 2024.
[43] Hao Song and Peter Flach. Efficient and robust model benchmarks with item response theory and adaptive testing. International Journal of Interactive Multimedia and Artificial Intelli- gence, 2021.
[44] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAM- BADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 1525-1534, 2016.
13

---

[45] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4/68, 2021.
[46] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2023. URL https: //arxiv. arrg0
[47] Alex Kipnis, Konstantinos
Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench -a sparse benchmark of reasoning and knowledge in large language models, 2025. URL htwph:ttp///o. htt///aaxxv.oraaaaaeee/ /t407 12844.
[48]
Ege Erdil and Tamay Besiroglu. Algorithmic progress in computer vision, 2023. URL https: // aaaiv.voooo org/ abs//222..2221555
[49] Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress in language models, 2024. URL https: //axivv..//////////ae///ee orggabs//40/0 05812.
[50] Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, lan Kaplan, Deniss Ja- cenko, and Yuxiang Wu. AIDE: Ai-driven exploration in the space of code. arXiv preprint Stti:25002131335 2025.
[51] METR. Measuring automated kemel engineering.
https://metes http://eetrrrr/blgg///////
2025-11 02- hanr44ammarrrnnnggatttttttt--ere automated-kernel www.--eeeeeeeeenieniinnnoooooeo 02 2025.
[52] David Malin Roodman. On the probability distribution of long-term changes in the growth rate of the global economy: An outside view, 2020. URL https: //api. semanticscholar. org/Corpus ID: 221222222222 1419999.
[53] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: a graduate-level google-proof qa benchmark, 2023. URL https://arxivy http:/iv..rga/bb////ee/ 12022.
[54] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: / /tw.xxxxivvvo//////////////aa orgoooggg//o abs//103. 03874.
[55] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contami- nation free evaluation of large language models for code, 2024. URL https://arxiv.. org/
abs/2403. 07974.
[56) Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmark- ing multimodal agents for open-ended tasks real computer environments, 2024. URL https://arrae org/ absp/bb//.22///// 07972.
[57] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL https://arxi:: org/abs/ /bs// 2307. 13854.
[58] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-mode llms in video analysis, 2025. URL //axxi: wrg/ 2 21075.
14

---

A Expanded Related work
Below, we expand on our discussion of related work in Section 1.1.
A.1 Agent and capability benchmarks
The evaluation of AT capabilities has evolved significantly from single-task benchmarks complex, multi-step evaluations designed to assess agent-like behavior. While traditional benchmarks such as GLUE [30]. SuperGLUE [31], and MMLU [32] have provided valuable insights into language model performance, they primarily measure static knowledge rather than the dynamic problem- solving capabilities essential for real-world applications. Recent work has developed more complex agent benchmarks. AgentBench [9] evaluates agents across diverse environments including web browsing, coding, and game playing. MLAgentBench [14] and MLEBench [15] focus specifically on machine learning research tasks, while ToolBench [33; 10] assesses tool use capabilities. The recent ZeroBench [34] involve difficult reasoning. but the context of visual puzzles rather than economically valuable tasks. Other noteworthy benchmarks include GAIA [11], which evaluates reasoning across multiple modalities, and BIG-bench [35], which contains hundreds of diverse tasks including many requiring multi-step reasoning.
Software engineering has emerged as a particularly informative domain for evaluating Al capabil- ities. HumanEval [36] and MBPP [37] provide programming challenges of varying complexity, while more complex benchmarks like SWE-bench [28] and APPS [38] test more sophisticated pro- gramming abilities. We use SWE-bench Verified [7]'s human time estimates for task completion our work. RE-Bench [2], which we use this work, evaluates models on complex research engineering tasks that may require hours of human effort and compares Al performance to human machine learning engineers. SWF--Lancer [17] generates tasks
While these benchmarks provide valuable insights into specific capabilities, they often lack a unified metric that allows for tracking progress over time and comparing models of vastly different capabil- ities. Our time horizon approach aims to address this gap by providing a continuous metric that can be used measure progress across different capability levels.
A.2 Forecasting Al progress
Quantitative forecasting of AI progress has employed various approaches, often starting with the observation that the compute used in AI training has increased dramatically over time. Amodei and Hernandez [39] observed that Al training compute usage has been increasing exponentially, doubling approximately every 3.4 months between 2012 and 2018; Epoch Al [40] included more recent data well trends in training dataset size and energy usage.
Other work has studied how AI performance has increased over time, relating benchmark perfor- mance to release date, compute usage, and other inputs. Sevilla et al. [41] found that compute usage growth rate increased at the start of the "deep learning era" in 2010, coinciding with increases in performance. More recently, Owen [19] and Pimpale al. [20] use compute and other metrics to forecast future benchmark performance.
Several recent efforts have been made to contextualize AI benchmark performance. One such effort the annual AI Index Report [6], which tracks performance across various benchmarks, including the date at which models achieved human-level performance. Murray et al. [18] had cybersecurity experts relate AI performance on Cybench [42] to the ability to autonomously develop malware. Phuong al. [3] commissioned professional forecasters to predict whether Al ranks amongst top public concerns by 2030, conditioned on benchmark performance. These efforts generally lack a unified, quantitative metric for cross-benchmark comparison.
Carlsmith [22] and Cotra [23] developed the "bio-anchors" framework, in which they related the compute involved in training Al models to the "effective horizon length" of tasks required for AI to have transformative impacts. Ngo [21] proposed using the time horizon for which Al systems outperform most human experts at most tasks to measure general Al capabilities. In our work, we empirically evaluate the relationship between task duration and Al agent success rate, which we convert into quantitative metric of Al agent performance.
15

---

A.3 Psychometric methods and Item Response Theory
Our methodological approach draws inspiration from psychometric testing, particularly Item Re- sponse Theory (IRT) [24], which models the relationship between latent traits (such as ability) and observed responses test items. In traditional IRT. item difficulty is a parameter in logistic model predicting response correctness based on respondent ability. Our approach inverts this, using task completion time (a proxy for difficulty) to predict Al performance. Our methodology also relates to difficulty estimation techniques in educational testing [25], where multiple metrics including com- pletion time are used to estimate the difficulty of tasks. IRT has been applied to machine learning classifiers by Martinez-Plumed et al. [26]. and was used to design efficient benchmarks in Song and Flach [43].
B
Task suite details
As with most benchmarks, the three task suites we study were designed to isolate specific unit of work that can be reliably accomplished within a time limit. This usually means that the tasks require much less context than the average task in the middle of larger project." !1 We confirmed that all tasks were possible given the instructions provided by having humans successfully complete each task at least once.!?
B.1
Task subsuites
The HCAST and SWAA suites are divided into task families, which are groups of tasks that similar. For example, the "crossword" task family consists of tasks such as creating a 3x3 crossword puzzle, or 5x5 crossword puzzle, etc. We segment tasks into families because performance within families is correlated and we down-weight families with many tasks for diversity.
B.1.1 HCAST suite
We use 97 tasks from 46 task families in HCAST, diverse set of challenges in cybersecurity, machine learning, software engineering, and general reasoning.
Tasks in this suite range from easy tasks that take humans couple of minutes (e.g. looking up basic factual question on Wikipedia) to tasks that take expert humans multiple hours (c.g. writing CUDA kernels, or fixing a subtle bug in Py PoTocc)). Because modern frontier Al systems are relatively more proficient at text-based tasks, the majority of these tasks do not require visual/multimodal capabilities, and all tasks are solvable by text editing via bash shell.
Compared to many recent benchmarks, tasks in HCAST are not designed to be as difficult as possible for either human domain professionals or current AI systems. Instead, most tasks are designed to be realistic, such that doing well on task requires skills we expect to be economically useful. As result, we expect that most of these tasks are solvable by humans with a few years of professional experience in the relevant domain.
Tasks are defined by their instructions, starter resources, and an algorithmic scoring function. Task instructions are strings, typically between 1-2 sentences and few paragraphs, although they can refer to other sources of information included as starter resources, externally available via the internet. Starter resources typically consist of code, data, and documentation.
Each task is automatically scored between 0 and I, with higher scores indicating better performance. Many tasks only return scores of 0 or 1, but for tasks with continuous scoring, we manually define success threshold score, which we use in some of our analysis to binarize agent scores. 13
" We define context as information that experienced employees use to complete a task which is not explicitly in the task description possessed by most external experts. For example, when fixing bug in software package, the package's maintainer may their experience with past bugs in the same package to guess at the bug's cause, fluency with the codebase to find the bug. and knowledge their organization' priorities to decide whether to apply quick patch or a more thorough fix. We discuss possible effects of context in Section 4. done by in-house staff
12 As many of these attempts exclude these attempts from our human baseline numbers.
done using different methodology,
A small subset of these tasks is available on request. (We do not share the content of most tasks to reduce the likelihood of AI systems accidentally or intentionally being trained them.)
16

---

Distr butioo Task Difficuity
Task Soiurce HCAST
Bench
SWAA
Numbiarr of taskt (RSC4ed)
1980
4.500
sec
1mmlr
min
15
1hr
4hrs
15hre
Human task time
Figure 7: Stacked histogram of tasks by difficulty rating. HCAST mainly includes tasks longer than minutes, while we focused on tasks in the 2-second to 15-second range with SWAA in order to measure GPT-2 and GPT-3. There is gap between the two which limits our ability measure time horizons this range.
More details about HCAST tasks are provided in Rein et al. [8].
B.1.2 RE-Bench suite
RE-Bench consists of 7 challenging open-ended ML research engineering environments, each of which are intended to take human expert approximately 8 hours to complete. See Wijk al. [2] for more details.
B.1.3 Software atomic actions (SWAA) suite
HCAST is designed to be a diverse set of tasks, but the shortest tasks are around minute long, lim- iting both the representativeness and the achievable resolution in measuring Al agent performance on shorter tasks. To fill this gap, we observed that real-world intellectual labor consists part of measurable, single-step actions shorter than I minute. We created the SWAA task suite, which com- prises 66 small tasks corresponding to <I minute atomic actions commonly performed in software engineering work. The SWAA subset includes both multiple-choice and completion questions.
Example SWAA task (file selection)
Which file is most likely to have a password in it?
1. credentials txt txt
2. installation.noter txt
3. main.py launcher_win cicheerwinneee
In contrast to other simple benchmarks like LAMBADA [44] or GSM8K [45], which test skills not directly applicable software engineering. the SWAA set represents actions that are needed in both software engineering work and the longer tasks we study. SWAA consists of 5 task families, three representing common decisions, one for code completion, and one for math; see Appendix 1.1.3 for more details.
Development of SWAA tasks was blind to AI agent performance; that is, all tasks written before seeing AI attempts, and elicitation (development of the few-shot prompt used for evaluation) was carried out on separate development task suite.
SWAA consists five task families: three representing common decisions, one for code completion, and one for math.
Decisions (multiple choice)
17

---

Envirunment
Briel description
Scuring l'unction
Optimize runtime
Optinize LLM Foundry
Given a linetuning scripi, reduce
Log time laker. by the optimized
runtime much as passible
script to fineture the madel
without changing its behavior,
1000 dalapoints.
Optimize Kernel
custom kernel puting the prelix sum of l'une-
Log miker evaluate
prelix sum of the function on 1o randomly generated inputs.
tion GPU
Optimize loss
Fix Eanbedding
a cotrupted model permuted embeddings, recover much of its original OpenWec-- performance as pussible
log(loes (loes achioved by model on the Ween Text test set.
Scaling Experiment
Predict optimal tradooff between hidden size and number train- ing sleps for model trained flops while only ing much smaller training rans, with less thum le16 flops for experimentation
achioved by the hmiteed parameners on Open OpenWebTen WebTeat test, set and prediction emoor submitted loss prediction.
Restricted Architecture (Masked Language Model)
Build model text predica- tion out of limited set of Py- Torch primitives, not including divisson or exponeniation.
log Mloes achieved by model on the OpenWeb Text test set.
Optimize win-rate
Finetune GPT-2 for QA
Finetune GPT-2 (small) to be an effective chathot.
Average win perceniage, as eval- uated by Llama-3 8B, against GPT-2
base model (small) model linetured on the Stanford Alpaca dataset.
Scaffolding for Rust Codecontest
Prompt and scaffold GPT:3.5 to as possible compe-
Percentage of problems solved a beld-out dataset 175 Code Contest problems.
titin programming problems given in Rust.
Figure 8: The 7 original RE-Bench tasks.
File selection: Which file has certain property, or is appropriate to read in situation?
Alert triage: Which team at a company should investigate an alert?
Request routing: Which action needs to be taken in response to some request?
Fill-in-the-blank
Code complction: Complete a single word of code.
Math: Solve simple arithmetic problems, either standalone or software engincering-themed word problems.
When developing SWAA tasks, attempted to avoid biasing the results by blinding task authors to model performance, so the preliminary set of 71 tasks was written without running any models on them during development. These were filtered down to the final set of 66 by excluding tasks that more than one baseliner failed.
RE-Bench
See Figure 8 for a description of the RE-Bench tasks."
B.2 Limitations of the task suite
The tasks we use to benchmark AI capabilities are systematically different from real tasks. These differences could result in the trends we observe on these tasks not generalizing to real world tasks. For instance, all SWAA, HCAST, and RE-Bench differ from world tasks in the following ways:
14The Restricted MLX task involves ML engineering while prohibiting certain PyTorch methods; current models sometimes use these prohibited methods in an indirect way such that the cheating can't 't be automatically detected, but model cheating does meaningfully affect our results.
18

---

Automatic scoring All tasks we use are automatically scorable, meaning piece of code rumning in the task environment determines the final score. This imposes constraints on e.g. the format of solutions, that tend to reduce task open-endedness, and the need for sensible value judgments.
No interaction with other agents No tasks involve interacting with other autonomous agents. Coordinating with, or competing with, other agents seems likely to increase task difficulty. For instance, by increasing the importance of strategic decision-makee making, real- time coordination, and predicting the actions of other complex agents.
Lax resource constraints No SWAA tasks, and few HCAST tasks saliently involve making efficient use of limited resource-4 common constraint in real-world tasks.
Unpunishing Similarly, very few tasks are punishing of single mistakes. This is in part to reduce the expected cost of collecting human baselines. Real world tasks can often be more punishing, for instance, when they involve competing against other agents. For instance, single blunder in chess game can greatly reduce the chance of winning the game.
Static environments Tasks in this suite typically use environments that do not significantly change unless directly acted upon by the agent. In contrast, real tasks often occur in the context changing environment.
We attempted to measure how these systematic differences might affect Al agent performance in Section F.2, by including the above properties "messiness" factors. We found that though the absolute performance on "messier" tasks was lower, the trends in performance were similar to less messy tasks.
C Methodological details
C.1
Human baselines
C.1.1 HCAST tasks
We use existing baselines collected part of HCAST. These baselines are collected from domain professionals with relevant experience in software engineering, ML, and cybersecurity. Baseliners work in the same environment as agents, using Vivaria (an open source platform for language model agent evals) with their screens and audio recorded for manual review, to prevent cheating. They are incentivized with bonuses for successful completion and for completing tasks faster than other baseliners. After screening out attempts tasks other than subset of HCAST, failed attempts, and those with issues (such as using disallowed AI tools), we include 286 successful basclines from approximately 460 total attempts. Task durations are calculated using the geometric mean of successful baselines, with manual estimates for tasks lacking successful baselines." 16
For further information about HCAST baselines, see Rein et al. [8].
C.1.2 RE-Bench baselines
For RE-Bench, we used the baselines from [2]. As baseliners were instructed to achieve the best performance for each task, consider the task duration of each of these 6 tasks 8 hours, and instead use the mean score achieved by baseliners who spent between 7 and 9 hours to convert raw score into success threshold.
Baselines for RE-Bench were conducted with the same baseliner pool and incentives as with HCAST, but with a few differences. Rather than being told to achieve threshold score, base- liners were given a fixed time limit of 8 hours and instructed to maximize their score; the threshold was set based on average baseliner performance. RE-Bench baseliners had access to AI tools, so the human scores on these tasks may be biased upwards.
15 With the exception submitting an answer too early on HCAST tasks. Non baseline estimates were based on information including the length of QA runs that did not follow strict baseline conditions, and the length of similar tasks with successful baselines.
19

---

C.1.3 SWAA baselines
Unlike HCAST and RE-Bench, which were baselined by external contractors, SWAA is baselined by internal employees with relevant expertise using a custom webapp that enables more accurate timing. Because these tasks are intended to be single step and exclude context acquisition, the timer for SWAA tasks ends as soon as the user chooses a response. For decision based tasks, only one selection is allowed to avoid random guessing; for fill in the blank tasks, baseliners can try multiple times until getting the answer correctly or opting to skip. We baselined each decision- based task 4 times and each fill-nnthhe-blank style task 3 times.
Unlike the HCAST and RE-Bench sets, SWAA is baselined by internal employees with relevant ex- pertise using custom webapp that enables more accurate timing. Because these tasks are intended to be a single step and exclude context acquisition, the timer for SWAA tasks starts after reading general instructions and ends as soon as the user chooses a response, so that baseline time only in- cludes reading the question itself and choosing an answer. Baselines done with our usual setup have timing overhead of seconds to tens of seconds, which would be unacceptably high for single-step tasks with duration shorter than seconds.
C.1.4 Baseline success rate
We aggregate human baseline times into task length rating by taking the geometric mean time of successful baseline runs, which more predictive of model performance than mean time because baseline times are roughly log-normally distributed.
We chose to filter successful runs for two main reasons one practical and one principled. Firstly, collecting enough baselines to estimate the sime-versus-success curve for each task would be prac- tically difficult. Secondly, we wanted exclude cases where the baseline failed for reasons that are not applicable to models. A substantial fraction of human failures appeared to fall in this category including humans having insufficient expertise for the task, or giving up on a task for unclear reasons (possibly due to getting interrupted, or getting bored). In particular, because we were optimizing for obtaining successful baselines, our payment scheme incentivized contractors to make a quick guess or give up early, in order to move on to other tasks where their expected carnings were higher.
However, conditioning on success biases towards shorter task length ratings, thereby underestimat- ing model performance. This is especially the case if many of the human failures are due to problems that are also relevant for models for example, if some of the tasks require guessing. This bias is most significant on longer tasks, which often have a baseliner success rate below 50%; therefore, have lower confidence in the difficulty ratings of these longer tasks, and expect they may be underestimates. If this is the case, we may be underestimating the pace of model improvement.
Human time horizon An alternative approach would be to calculate the human time horizon using the same methodology as we do for models. One natural interpretation of time horizon would imply that the time horizon of "a human given hours" is hours. Since spending up to 8 hours per task, we would expect their time horizon to be around 8 hours. However, in practice it's much lower, at around 1.5 hours (which would imply that the best models will surpass humans in under 7 months). As discussed above, we think this is artificially low, given that many human failures seemed to be artifacts of our incentive scheme.
baseliners were paid for
Figure shows graph of baseliner success rate by task length.
C.1.5 Baseline time analysis
We manually watched and annotated the recordings of 50 baselines to get a sense of how our base- liners spent their time. Based on this small survey of baselines:
1. On short (5-15 minute) tasks, a substantial fraction of the time (between 25-75%) is spent reading instructions understanding how submit correctly. Particularly for short tasks, our time estimates may be hard to interpret as there is some fixed overhead for reading the instructions, independent of the difficulty of actually performing the task per se.
2. Our tasks are designed to require relatively low amounts of context but base liners still spend over 25% of their time reminding themselves of relevant information (e.g. how to do port forwarding).
20

---

Human baseliner performance on tasks human
10
Fried cure reteo Wi 201
Succpes Prooab lit
Timee Hig 201
0.-
02
4l
1o
lingth Jhuman timeto-compline)
Figure 9: Success rates and time horizon of human baseliners. Note that the time horizon is not directly comparable to the time horizon ML models (see Section C.1.4)
C.2 Internal PR tasks
Summary We ran GPT-4o, Claude 3.5 Sonnet (New), and o1l on five uncontaminated issues from an internal repository. Resolving these issues real work performed by internal staff, might expect results on these tasks to better represent performance on real economically valuable tasks than typical benchmark task.
We find that our contract baseliners take 5x-18x longer to resolve issues than repository maintainers. Additionally, Al agent performance these issues is not inconsistent with Al agent success rate curves derived from HCAST, SWAA, and RE-Bench performance contractor time-to-con to--lplet used to measure the tasks length. However, it takes much longer for contract baseliners to complete these tasks than repository maintainers. This suggests that time horizons may have better correspondence the labor of low-context human, rather than high-context human.
Methodology We collected five real, recent, uncontaminated issues from internal repository. We then ran GPT-4o, Claude 3.5 Sonnet (New), and ol on these issues. We recorded how long it took repository maintainers to solve these issues, and also had external baseliners attempt to fix these issues. Unlike SWE-bench Verified, we did no filtering to require these issues to be automati- cally verifiable. Instead, maintainers of the relevant repository scored model and baseliner solutions manually, as if they were reviewing PRs:
1. 0 if the PR was incorrect and required fundamental refactor.
0.25 there were minor changes that must made before merging.
3. 0.75 if there were nits but the PR could still be merged.
4. 1.0 if the PR could merged as is.
Example issues We include two example issues one easy (Issue 1), the other (Issue 8) more challenging. Issue 8 demonstrates how context on the codebase may provide a dramatic speedup in human time to complete between baseliners and repository maintainers. Results of these baselines can be seen in table 3.
21

---

Issue
Agent
Time taken
Score
Repository Maintainer
5 minutes
1.0
Baseliner
81 minutes
1.0
8
Repository Maintainer
20 minutes
1.0
Baseliner
113 minutes
0.25
Table 3: Results of baselines selected internal PRs
Issue
The stage plot_logiatic_ intiiiidaadd with the error message:
FileNotFoundErro::
[Errno 2] No such file directory: png
ERROR: failed to reproduce
failed to run: python - src plot. ioiiao --input- file http///magg//ggggaccrrrrrraaaaar
_ftr.
--output file plots plots//ogiatic. itciiiiiiivvviii http:stiiiiiiiiviuuaaaaaaaaaaaaa htg.aasic..aaiiadaaaannnnrsteete weight -1.01.
png
-pplot format png --lo---- -log-levee INFO exited with
.EC
Issue
Reweighting in bootatrapping
Currently the hierarchical bootstrapping does not reveight runs.
Task ID)
GPT-4o
Claude 3,5 Sonnet
Issue
0.35 (5)
0.45 (5)
0.875 (6)
Issue
(5)
0.0 (5)
(5)
Issue 9-1
0.0 (5)
0.0 (5)
(5)
Issue 9-2
0.0 (5)
1.0 1.0
0.85 (5)
Issue 10
0.0 (5)
0.0 (5)
0.0 (5)
Issue 0.02 (12)
0.0 (11)
0.0 (8)
Table 4: Internal PR Per-Task Average Scores (number of trials in parentheses). Note that we did
minimal processing Issue 9 to turn into issues, as in practice the issue description contained two entirely separate pieces of work.
Repo maintainer time baseliner time baseliners to complete issues, compared to how long it took repository maintainers to complete issues (Table C.2). In practice, repo maintainers were 5-18x faster completing the given issues.
There was dramatic difference in how long it took
Task ID
Maintainer Time (min)
Baseliner Time (min)
Slowdown
eval wwal-analysis-puble vvll-anayysis-uuliie
5
81 (score I)
16x
20
113 (score .25)
55.5x
aod-analysii-puublee 9
235
eval-analysis Sicpllssiisuuuli---eee
5
69 (score 1)*
14x
waaaaallssssuuiiiiee.. 10
20
5
93 (score .75)
18.6x
*Note: This was run on a slight variant
Table 5: Comparison of time fix issues by repo maintainers and baseliners
22

---

Model
Agent
Claude 3.5 Sonnet (Old)
modular-publle
Claude 3.5 Sonnet (New)
modular-public
Claude 3 Opus
modular-public
davinci-002
modular-public
gpt-3.5-turboiinsrruct
modular-public
GPT-4 0314
modular-public
GPT-4 0125
modular-pubil
GPT-4 1106
modular-publle
GPT-4 Turbo
modular-pube
GPT-4o
modular-pun public
ol
triframe
ol -previiww
duet
Table 6: Scaffolding used for or each model in this report
Manual scoring methods and results Maintainer, baseliner, and model solutions were manually scored as if they were pull requests. We experimented with both contractors scoring PRs, well as different repository maintainers scoring PRs. practice, repo maintainers scored results much more consistently than contractors. Contractor correlations were between 50-60%. Repo maintainer correlations were between 88-91%.
Time to score For models to be cost competitive real work, their total cost to complete PR must be less than the cost for a repository maintainer to implement this PR themselves (plus ancil- liary costs, c.g. the time of code reviewer). As such, kept track of the time that it took to score model solutions, to better understand the net cost of using a model to fix a real issue. Repo main- tainers scored submissions much faster than contractors, although the ratio time to complete- contractors took an average of minutes per run, while repo maintainers took complete-contraators
less dramatic than
average of 3.5 minutes per run. Different issues had different average times score, with easier issues generally being much quicker to score, and all issues being faster to score than to complete. In the future, we plan create an approximate cost metric that takes into account scoring effort, model cost, and model success rate. We expect that this will be a useful framing in understanding how the ratio of model / maintainer work may shift in the future, the cost of agent usage continues to fall.
Contract baseliners vs repository maintainers Model success rates were reasonably consistent with those predicted by contract baseliner times. In practice, baseliners also did much worse on casy tasks than experienced maintainers. Issue 11, for example, took a baseliner upwards of 1.5 hours to complete, and so we would not expect models to consistently complete this task.
Qualitative impressions Naively comparing to per-time bucket success rates on HCAST SWAA + REE-bench tasks, repository maintainer time-to- timeto-comppete is not good predictor of model perfor- mance on these tasks. For example, Issue 11 was the simplest task for maintainers. It requires writing simple comments across 10 Python files, and it took maintainers less than five minutes to complete. However, models did not successfully complete the task successfully once 30 runs. There were also tasks where models did succeed, as we would expect from per-time bucket success rates on other tasks. For example, models were consistently able to add a missing folder creation step to a single stage in data pipeline. While baseliners and models both struggle compared to repository maintainers, they struggle in different ways. Baseliners often lacked knowledge about the tools and techniques used in the codebase: e.g. "what is DVC?" "what is bootstrapping again?". Current AI models, on the other hand, consistently demonstrate knowledge of tools and techniques, but appear to struggle with the larger context required for working in real codebase.
C.3 How models run
All models were run using agent scaffolds using the Anthropic or OpenAl APIs.
23

---

We used the same agent scaffolds across the evaluation suite, with no task-specific prompting or scaffolding, except for the SWAA tasks, which used a simple prompting scaffold. All agents were provided with the same affordances provided to human baseliners.
GPT-2 is incompatible with scaffolding due low context length, so we imputed score of zero for GPT-2 on all tasks in RE-Bench and HCAST. We think this is reasonable because the far more capable davinci-002 (GPT-3) scores zero on this set. Removing these imputed GPT-2 zero scores has a negligible effect on all subsequent results in this paper.
C.3.1 Scaffold and platform details
Most Al models were evaluated with modular-public- basic agent scaffold. This scaffold provides the model with Python and Bash commands and some very simple context management to keep the input within the context window length of the LM. We used a slightly different scaffold for ol ol-preview and ol, because they seemed to struggle with tool use, responding to environmental feedback, and generally acting as an agent.
For both human baselines and Al agent runs, our experiments use the open-source platform Vivaria. CPU and GPU resources are provided inside the secure VMs based on the resource specifications of tasks. Table displays the scaffolding used for each model. Both modular- pallaar-puccccc and triframe/duet incorporate principles from the ReAct framework [46], which interleaves reasoning traces with ac- tions. These agents are developed on a held-out dev set of tasks, not included in HCAST, to reduce the likelihood of overfitting these tasks. See the respective repositories for implementation details about these scaffolds.
Both scaffolds allow agents to plan via ohain-of-thought reasoning, before calling selection of tools. The agents interact with their task environment primarily through running Python code and Bash commands.
In Modular, the model generates single command for execution, then the scaffold executes the function call, returning relevant information from the environment (e.g. STDOUT/STDERS This process repeats until either the model determines that the answer is ready for submission or the system reaches its predefined usage threshold.
In Triframe, to decide on a command execute, the model generates one suggested plan, then gen- erates three possible commands to execute based on the plan, well as three suggested commands that ignore the plan (to diversify the ideas it generates). Then, the model generates two scores between -2 and 2 for each of the six proposed actions, and the scaffold executes the top scoring function call averaged across the two scores.
C.3.2 Compute usage
The total amount of compute used included about 2,000 H100-hours for RE-Bench environments and 50,000 CPU hours for other environments on a combination of cloud and internal machines, plus roughly 50,000 H100-hour equivalents of compute used internally or from API providers for inference. This includes compute used for preliminary experiments. Compute used for data analysis is minimal.
C.4 Comparison to Item Response Theory
Item response theory (IRT) is collection of statistical techniques used to predict the probability that a person will answer a question correctly, based on their ability level. The standard IRT two- parameter logistic (2PL) model is (simplified from Cai et al.):
(1 (1egeett a((lageet (ttask) Brask)
Where (1 [eggnn)) indicates the probability that a person with ability level Negent answers task correctly. IRT is usually used regress on both the question difficulty and the ability level
"In the guessing the task. In the 2PL model and our method, "task = O. This is effectively true for most tasks, but some SWAA tasks are multiple-choice with a 25% chance of guessing correctly. This choice only affects the time horizon of GPT-2 and GPT-3 and does not meaningfully affect our results.
logistic model, a parameter Yousk added representing the probability of correctly
24

---

simultaneously: in contrast, we define the question difficulty as the logarithm of human baseline time, and therefore only regress on ability level. In particular, our model is follows.
(1 (agent) a ((log Magent log (tak)) -agent)
Note the following changes from the 2PL model:
1. Ottakk = log bask where bask is the geometric mean of successful human baseline times,
2. Magent can now be interpreted as log hagent where hagent the 50% time horizon of the agent rather than an abstract ability parameter. This is because when hagent tasks our model predicts a success probability of (0 Bagent) = o (0) 0.5.
3. Pagent is an agent-dependent learned parameter, rather than being task-dependent.
To fit equation we weight tasks by diversity, such that a task from family of size n gets weight 1/.n. We then use logistic regression to fit the time horizon Ragent and slope Bagent for each model. For example, following this methodology ol has a time horizon of about 39 minutes (see Figure 4).
These changes allow us to compute an interpretable time horizon without several complexities of IRT. First, IRT has an initial calibration/joint optimization stage in which the task difficulty is es- timated from all agent scores, so the ability score of one agent may depend on other agents' per- formance; in our method each agent's time horizon is computed individually. Second, IRT outputs "ability scores" which then need to be converted to a time horizon estimate; our method eliminates that step by estimating horizon directly. Third, the 2PL model uses one slope parameter per task, whereas we use one slope parameter per agent. In our setting with many fewer agents than tasks, this reduces the overall number of parameters.
In exchange, we forego the ability use IRT metrics designed for test design, such as Fisher infor- mation functions for individual tasks (used in e.g. Kipnis et al. [47]). Future improved methodolo- gies could include these metrics to design sample-efficient benchmarks and evaluations grounded in human data.
D Qualitative analysis
D.1 Failure categories
To better understand the differences between current and older Al agent failures, we separately sampled 31 unsuccessful agent runs from our GPT-4 1 1106 agent and 32 unsuccessful runs from our ol agent, and manually labeled them for the following exclusive categories of failures:
Poor planning and tool choice: the agent generates high level plan that seems unwork- able on its own merits, or picks tools for the plan that would not accomplish the desired purpose.
Incorrect mental math or reasoning: the agent performs incorrect mental math or logical reasoning crucial step, causing the to fail.
Premature task abandonment:
The agent abandons the task in the middle of the attempt and either submits a nonsensical answer or submits solution without checking for cor- rectness. These failures often result from the agent submitting their answer before looking at all the pieces of code or information required arrive at the correct solution.
Repeating failed actions: The agent repeats the same behavior that doesn't 't make progress toward the problem, such as running a command that leads to an error over and over again, without trying other approaches.
We report the results in Table 3.3. We find that over third of the GPT-4 failures resulted from repeating failed actions, compared to 2 out of 32 for ol, which we see as quantitative evidence for our claim that models seem to have improved failures resulted from abandoning the task prematurely, while only a quarter of the GPT-4 failures resulted from the same-this this may result from ol failures occurring on qualitatively more difficult tasks, or may reflect idiosyncrasies of ol.
their ability to adapt to mistakes. Half of the
25

---

D.2 Specific skills of newer models
To better contextualize the observed trend of improved model performance, we examined the tran- scripts for the tasks where earlier models (c.g. GPT-4) do substantially worse than current models. Specifically, we categorized all task families based on the type expertise required, and noticed that current models tend outperform earlier models for tasks that involved MI. training, reverse engineering compiled software binaries, and cybersecurity CTFs. In addition, we noticed that agents improved greatly on tasks requiring situational awareness of the Al agent's limitations or defeating an opposing strategy. This gave us five sets of task families to examine.
For each of these task family sets, then used contractors manually read through all runs from all models for all tasks each of these five task families, and identified possible explanations for the improvement in Al agent performance on those tasks, well as potential limitations.
We provide examples of both ways that models have improved, as well as their major limitations, as described in Section 3.3.
D.2.1 What do newer models do better?
We start by discussing some the ways in which current models have improved compared to pre- 2023 models. For some of our public tasks, we also provide links to the transcripts of the full runs.
Greater robustness to mistakes Models have become increasingly robust time, and espe- cially seem to be able to avoid repeating failed actions. Early models typically fail to correct even trivial mistakes, instead engaging in looping behavior introducing additional errors. However, starting with GPT-4o, models demonstrate improved ability to recover from mistakes and course- correct, an ability that is especially valuable the harder variants of the pico_ctf' tasks, or other coding related tasks that require iteration. Consider examples from a task where models need to debug small amount of Python code:
1. In one run 100-4-turbo-0024-00 backslash character in a Python file, and despite copious attempts is unable to understand fix the issue until it gives up.
introduced syntax errors related having misplaced
2. Claude 3.5 Sonnet (New) first adds code at the wrong place, and then attempts several times to add the code the correct file using the CLI tool sed. However, when doing so it encounters IndentationErroos and duplicated methods. Finally it successfully pivots to writing the entire file from scratch.
Agent output
I apologize for the confusion. It seems that our previous attempt didn 't fully remove the old 'save" method. Let's try fix the 'base_model, py file. We'll remove both 'save' methods and add the correct one
time
Agent output
I apologize for the persistent issue. It seems that our previous attempts didn't fully resolve the problem. Let's try one more time to fix the 'base_model.pp file by completely rewriting the content of the file:
Tool use Early models like davinci-002 (GPT-3) struggle with syntactically correct tool use, while later models improved with clear explanations and examples in their prompt. Current frontier models generally use the tools afforded our scaffolding without issue. For example, we noticed that when tasks required editing Python scripts, GPT-3 struggled with formatting even when provided explicit
26

---

instructions. and GPT-4o would sometimes write scripts iine-by-ling using echo commands in bash, while ol preview would consistently edit files efficiently.
1. Despite the reminder in the user message about the expected format, davinci-002 fails to use the correct syntax with angle brackets.
User
No function call included in the last message. Please include function call in the next message using the <[tool_name]> [args] </[ tool_name]> syntax.
Agent output
bash> touch test. txt python import time
python> time slmep (10)
python> exit()
submit> python main.py
2. GPT-4o sometimes struggles to write the Python script effectively. In run on the same task, it ineffectively writes each line of the script with separate command.
Agent ouptut
I will write the script content directly into the script file using echo commands, line at time.
bash: ("command":
"echo 'def read_nth_ line (file_path, n): >> podate_test_txt.pp
baah: {" {"command":
"echo with open (file_path, "r") as file
>>
3. ol ol-preview consistently edits files on this task without any issue.
Improved logical and programmatic problem solving Consistent with increasing performance on coding and reasoning benchmarks, we saw models' ability to write syntactically and semantically correct code gradually increase. This allowed models to more reliably interpret task instructions correctly and use code to find correct answers.
1. For example, the interpret data task, GPT-4o fails in pretation of the request to find "the tallest building in the world in 1972" as finding "the tallest building in the world built in 1972". as opposed the common sense interpretation that the task is to find the tallest building in the world in 1972. For example, it writes code like:
due to incorrect inter-
Agent Output
tallest_ cailest_building.t 1972 = data [data [
completion_year *] 1972] [ 'height_m '] max
tallest__uuiddng_.119
2. In contrast, op-preveew correctly interprets the request as finding the tallest building in the world built in before 1972.
27

---

D.2.2 What do they still do poorly?
Despite improvements, even current frontier models still struggle in substantial ways. In general, our observerations indicate that agents tend to do worse on tasks that are qualitatively "messier". We investigate this claim further in Section F.2.
AI agents struggle without clear feedback Without clear feedback or cheap ways of verification, such as unit tests or answer checking scripts, all models we tested often have difficulty understanding whether their solution is correct. For example, two of the tasks hardest for models in their respective time buckets are 'blackbox'' and 'symbolic regression'. Both of these tasks involve guessing hidden function under the constraint that probing for information costly.
AI agents often fail to proactively seek out relevant information We also observe that all mod- els still exhibit deficiencies in understanding their own limitations or proactively seeking helpful information; instead, models tend to assume that they already know how to complete the tasks and then only reevaluate after they fail do so.
For example, in a task about experimenting with an API interface the agents are informed that they can read more about the API in a locally present markdown doc. Even if agents succeed, they typically start off hallucinating or guessing the API endpoints, and only when they encounter an error from the task environment do they read the API.
In capture the flag task, agents have to open a file with unknown encoding in python, and often fail on the first try because they specify an incorrect encoding. Even the best models respond by trying out different encodings in the python script, which is inefficient and wastes tokens. We never saw an agent running the bash file command that would identify the necessary encoding directly.
E
Further discussion
F.
Constant factors versus slope changes
Although extrapolation is always imperfect, forecasting AI time horizons far into the future is much more sensitive to changes in doubling rate than to constant factors time horizon. For example, a naive extrapolation based on ol 's time horizon of 39 minutes and past doubling time of 218 days (3.2x/year) predicts that AIs will reach a 1-month 50% time horizon (roughly 8 doublings over ol) about 4.8 years after the release date of ol. A 2x increase in doubling time would delay the month point by a further 4.8 years, but a 2x constant factor decrease in time horizon would only delay by 0.6 years.
E.2 Important factorr that could change time horizon slope
Agency training Horizon growth since 2024, which may be faster than the long-term trend, could be explained by researchers post-training models to be more agentic (that is, capable of taking many sequential actions towards completing a task) using outcome-based RL. Research into making models capable and agentic is likely to continue. Future agency training could be faster than the long-run trend (since post-training may be more compute-efficien than pretraining at increasing horizon length). But 2024-2025 agency training could also be a one-time boost from picking low- hanging fruit, in which case horizon growth will slow once these gains are exhausted. Overall, we think agency training is more likely to increase the time horizon growth rate compared to the 2019-2024 trend.
Compute scaling Between the release of GPT-2 and today, the compute used to train the most impressive frontier language models has increased by at least factor of 10,000x [40], with training compute usage doubling every 6-10 months [41]. More recently, models like ol and o3 have began to use more compute at inference time. It is unclear whether there is sufficient capacity to expand either training or inference compute by many more orders of magnitude in the next 5 years. However, algorithmic improvements, which have historically decreased the compute requirements for fixed performance level [48] [49], can substitute for compute limitations. We think that limits to compute scaling will slow the growth of Al agent time horizons somewhat, but be partially compensated by more investment into algorithmic improvement.
28

---

Automation of AI R&D The main inputs to Al research and development are compute and re- searcher time. If future Al systems are capable of substituting for human research engineers and/or increasing the compute-efficencc of training, the rate of Al progress will increase. We think it is likely that there will be substantial AT R&D automation once frontier Al time horizon reaches tens of hours, shortening the time from then until nne-month-horizon AL.
AGI will have "infinite" time horizon An infinite time horizon does not mean an arbitrarily capable AI, merely the ability to complete tasks that take humans an arbitrarily long length of time. an artificial general intelligence (AGI) is capable of completing all tasks expert humans can with a success rate of at least X%, its X% time horizon will necessarily be infinite. Therefore, if such systems are ever developed. the long-term trend in time horizon will be faster than exponential, with an asymptote the date of AGI deployment.
E.3 Interpreting time horizon
Although time horizon is an intuitive measure of Al agent capability, measuring requires large dataset annotated with human time, and time horizon is always measured relative to a task distribu- tion and baseliners' levels of context and skill.
Context and skill effects At real companies, junior software engineer hires often take weeks of onboarding to begin contributing economic value. The human baseliners that determine the length of most tasks have much less context than average employees, potentially increasing measured task length. Our tasks are designed to require minimal context, which somewhat mitigates this problem; our internal PRs (Section C.2) were not designed this way, and so baseliners took many times longer than employees. However, highly skilled baseliners can also complete tasks far faster than average employees. Our expert baseliners are likely much more skilled than the average software engineer, potentially decreasing our measured task length.
Task distribution effects Figure 3 shows that Al agent success rate is imperfectly predicted by hu- man time-to compleeee meaning that other factors also substantially influence the difficulty of tasks. When models are measured in different domains of intellectual labor like research mathematics, computational biology, or law, we expect their time horizons to differ.
Measuring extreme time horizons Accurately measuring that the X%-time horizon of an AI agent is about t minutes requires many tasks of human length t that the Al agent completes with a success rate of about X%. This has two implications. First, accurately measuring very long time horizons requires a dataset of difficult tasks with long human baseline runs, which can be impractical construct, especially because success criteria for realistic difficult tasks are often complex enough to require manual grading. Second, measuring time horizons at extremely high success levels 95%, 98%, or higher requires very large task datasets containing many easy tasks, with near-zero label noise that cover the population of tasks they are meant to represent.
Human time horizon measurements In theory, one could also measure the time horizon of human or population of humans. However, there are both theoretical and practical difficulties to doing We discuss this more in Appendix 1.1.4
E.4 Limitations and future work
We believe that there are several ways in which our work could be improved.
More models with better elicitation In general, we find that properly eliciting models can make a very large difference in their performance. We have put a limited amount of effort into eliciting models to get good performance some models may have somewhat greater capabilities than we demonstrate. The most work was done to elicit ol and the original Claude 3.5 Sonnet, each of which had around 2-3 engineer weeks of iterative development. All other models use the same scaffolding with at most minor changes.
tasks, while our results are reasonable lower bound,
"hiis is a common obervation; see e.g. the improvements to software development capabilities from AIDE [50] recent work on KerneBeench: [51]
29

---

Future work could replicate our results with more effort spent on eliciting the full capabilities of frontier models.
More rigorous human baselining Our per-task human time estimates are likely noisy due to rel- atively small sample size, and potentially also systematically skewed in various ways. Most notably, we select only successful completions of task, and encourage baseliners to give up on tasks they may not complete in reasonable amount of time. Our baseliners' skills also vary significantly, and a wide variety of skills are relevant to these tasks. Though we attempt to match baseliners with appropriate tasks, this process is unlikely to be perfect. From manual reviews of baseline attempts, we also observe that humans sometimes simply give up even when the task seems within their capa- bilities, and it is unclear what selection effects are produced on the distribution of success times as result. See Appendix C.1.4 for further discussion of how bias in human baseline times could affect our results. Future work could replicate our results with more rigorous human baseliner selection or explore how sensitive the results are to methodological choices around human baselining.
More natural, varied tasks There are reasons to believe that our task distribution is systematically different from the distribution of economically valuable work (and perhaps systematically different than the distribution of risk-model relevant tasks). We explored some of these reasons in Sections 4, but there remain many differences that we did not explore. For example, the modality of interaction in these tasks is also relatively narrow- for example, no tasks require the use of a mouse. No tasks require cooperating or competing with humans other agents," 19 while real software engineering or ML research involves communicating and coordinating with managers and other engineers or researchers. Many real-world tasks require very high reliability, and these are underrepresented in our dataset due to the difficulty of measuring models on these tasks. Most importantly, the tasks we study are heavily skewed toward software engineering and ML research. Future work could explore how the capabilities of Al agents are progressing in other domains.
More use of inference compute Our scaffolds made relatively limited use of inference-time com- pute. When assuming that the human expert is compensated at $143.6 144..6//hour (average L4 Engineer salary at Google divided by 2,000 hours), more than 80% of successful runs cost less than 10% of what would cost for a human to perform the same task. (Figure 14). This implies that if inference- time computation could be used to improve performance, there is substantial room to do so while still remaining economically competitive with human experts. Previous research has found that tech- niques such as best-of-k can substantially improve performance on a subset of these t2sks, and better use of inference-time compute may lead substantially different scores.
Data analysis The 2024-2025 trend appears faster than the 2019-2025 trend, but due to the small number of models is unclear if this is noise. Future work should include hypothesis testing to detect possible 2024 slope change, and more sophisticated statistical methods to create credible in- tervals for an overall forecast. We also lose some information in the multiple stages of estimation- conversion of baseline data to task difficulty ratings, time horizon computation, and linear regression to find the trend-anm end-to-end methods could be more data-efficient.
F External validity details
F.1 Retrodiction from 2023-2025 data
As part of exploratory work for this paper, we measured the time horizon of 9 frontier and near- frontier models released in 2023 and 2024, using only our HCAST and RE-Bench suites. This trend (with Claude 3.7 Sonnet and o3 added) is shown in Figure 10: time horizon doubles about every six months. Since we only had two 2023 models (GPT-4 0314 and GPT-4 1106) and small data range (release date spanning 2 years and time horizon spanning 5 doublings), error bars were very wide. In addition, restricting our data further to 2024-only models produced different trend with time horizon doubling about every three months, so any extrapolation into the future would not be robust.
See Xu et al. [12] for an example of a recent benchmark that requires multi- agent interaction in a relatively realistic setting.
30

---

To address these issues, we collected more data to extend the trendline into the past, developing the Software Atomic Actions (SWAA) suite decrease the minimum human time of the task suite from minute to under 2 seconds, and enabling us to measure GPT-2, davinci-002 (GPT-3) and tur-o---urro-iinstuuet on the combined suite.
The 2023-2025 trend may be faster than the longer-term trend (Figure 11). The measured doubling time over the whole year period 2019-2025 inclusive was 207 days, which less steep than the 172-day trend based on data from non-SWAA tasks and 2023-2025 models. However, this difference is still within margin of error.
F.2 Messiness factors
Real-world intellectual labor often involves messy details that benchmarks usually don't include, such being under-specified or poorly scoped, having unclear feedback loops success criteria, or requiring coordination between multiple streams of work in real-time. We generally observed that agents struggle more tasks that have these "messy" details (Section 3.3). A natural question is therefore whether agents showed similar rates of improvement on "less messy" and "more messy" tasks.
We rated HCAST and RE-Bench tasks on 16 properties that we expected to be 1) representative of how real world tasks might be systematically harder than tasks study and 2) relevant to AT agent performance. Some example factors include whether the task involved a novel situation, was constrained by a finite resource, involved real-time coordination, context. We labeled RE-bench and HCAST tasks on the presence or absence of these 16 messiness factors, then summed these to obtain a "messiness score" ranging from 0 to 16. Factor definitions can be found in Appendix H.6.
sourced from real-world
The mean messiness a messiness score above 8/16. For comparison, a task like 'write a good research paper' would score between 9/16 and 15/16, depending the specifies of the task.
amongst HCAST and RE-Bench tasks is 3.2/16. None of these tasks have
On HCAST tasks, Al agents do perform worse on messier tasks than would be predicted from the task's length alone (b=-0.081, R = 0.251), see Figure 13. An increase in task messiness by 1 point reduces mean success rates by roughly 8. 20
However, trends in AI agent performance over time are similar for lower and higher messiness subsets of the task suite. For example, on sub hour tasks, success rates increased by 40 percentage points between Jan. 2023 and May 2025 in both high and low messiness splits (Figure 12). In particular, we find no evidence of either much slower performance trends, or plateau, specific to our higher messiness subset.
G Miscellaneous tables and figures
The following are several figures not included the main body, from external validity and other sections.
H More ablations and robustness checks
There are many ways to measure "time to complete" a "well-defined task", so our analysis involved many somewhat arbitrary choices. To ensure our results are robust, we've reproduced our results with alterations to our methodology and find the results are largely robust to these changes.
Because used these analyses to inform methodology choices, these ablations relative to a slightly different version of the pipeline than the one in the final report. In particular, they include slightly different set of baselines and different filtering for task success.
performed
1. A Alteraatvve curve-fits (H.1)
2. Re-nommalizing the task suite to other distributions than log-uriform
A linear approximation of this relationship is used for the purpose of roughly quantifying the size of this effect in an intuitive way.
31

---

50% Time Horizon for HCAST RE-Bench Tasks
Took tirne (for hummans) that model completes with
2hrs
Claude 377 Sonne:
Claude 35 Sonner. (New)
Sonnet (Old)
30 min
preview
GPT-10
50% success
GPT-4 TIDE
4 min
Doubling time: '72 days 01 R 0.9
1mmn
Model release date
Figure 10: Time horizons on HCAST + RE-bench, for models starting with GPT-4 0314.
50% Time Horizon Retrodicted from 2023-2025 Data
Timk (for humnans) that model completes with
Claude 3.7 Sonnec
2hm
(New)
Claude 3.5 Sonnet (Old)
80 min *
ol-preview
min
GPT 4o
2
GPT-4 0314
gpt 3.5 turbo instruct
1mn
devinci-OC2 (GPI-5)
se
GPT-2
15
Fit on nan-SWAA tasks, 2023-2025 models
Doubling time: '72 days
4.ec
2023-01-01+
on all data Doubling time: 207 days
1sec
2019-01-01+ data
2020
2022
2025
2074
Model release date
Figure 11: The full time series for the time horizon of models, by release date. We plot in blue the regression from only 2023+ data on HCAST + RE-Bench tasks, extended into the past, and in gray the regression with all tasks (including SWAA) on the whole year period. Points on the graph are models' time horizons on all data including SWAA.
32

---

Table 7: Average Success Rate by Model and Task Source
Task Source
HCAST
RE-Bench
SWAA
Claude Opus
0.25
0.00
0.98
Claude 3,5 Sonnet (New)
0,46
0.11
0.99
Claude 3,5 Sonnet (Old)
0.38
0.05
1.00
Claude 3.7 Sonnet
0.58
0.04
1.00
GPT-2
0.40
GPT-4 0314
0.23
0.00
0.98
GPT-4 1106
0.30
0.02
0.97
GPT-4 Turbo
0.23
0.00
1.00
GPT-40
0.30
0.00
0.98
davinci-002 (GPT-3)
0.00
0.00
0.65
gpt-3.5-turbo-insrric
0.01
0.00
0.95
o1
0.54
0.07
1.00
ol-previey
0.44
0.02
1.00
o3
0.65
0.40
04-mini
0.61
0.27
Model Success Rates over Time, by Task Length and Messiness (Weighted by Task Diversity!
Task Length 50% least Messy Tasks [Messinese score + 233, N tasio
Task Length 50% least Messy Tasiis
10
[Messiness score 3.0. tooks 18)
Claude Sommer
Chocde Senvee thent
mi
Waighted Ratt
CaT do
06
0C
Instruct
80
20214107
201
2038-25-13
ag
2024
2004
107
201t
Tack Length: hour
Teek Lengch hours
50% Tesks
(Messinesc most Messy
10
|Messiness
tasks
05-
02-
Welgittd
0.4
34
20241-0
20to of
aocee (oud)
2026-25-77
300 d.
2022
Mode Heleose Late
Model Me edse Date
Figure 12: Performance trends over time for HCAST and RE-Bench tasks by length and messiness
(Section F.2). The data spans only 2023-2025 as pre-2023 models score 0 on non-SWAA tasks. Whilst our messier tasks have lower average success rates, trends in model performance improve- ments are not obviously slower on the high messiness split.
33

---

Excess Success Rate vs Task Messiness Score
Residual (Observed Predicted Success Rate)
R==0.283
02
-02
XXX
x
303
-06
Task Messiness Score
Figure 13: We plot the excess success rate (the observed empirical task success rate, minus success rate we would predict using the task's length, see Section 3.1) against messiness score for each task. As discussed in Section F.2, there is a negative relationship between excess success rates and messiness.
Cost Ratio vs Task Langth n=1460 (10 outside visiblé range)
10"
1011
Cost Rabio (Modee Cost/ Human Cost)
10-
Xy*
1MMC
15
sec
Triin
Zmii
Binee
15 min 30
1he
2 kms
10
Task Langen
Figure 14: Cost of successful run using an LLM agent as a fraction of the cost of the salary of human expert performing the same task.
34

---

Longth tasks agents have been able complete autonomously for 189 software engineering, cybersecurity, general reasoning, and ML tasks
4 hrs 2hr
Task time humanns model completes
1hr
50% 50% ratss
4mii
2mh
Hyperbolic 2019-01-0i data
1im
0.69
50c
2019-01-00 data
R 0:36
Expomential FiL Douhling time: 2C7 days
2019-01-01
1990
Cate model released
Figure 15: Linear, hyperbolic, and exponential fits for model time horizon since 2019.
3. Alternative means of estimating task difficulty
4. Sensitivity to baseliner ability:
Baseliners whose abilities we are subjectively very confident in
Restricting the task suite tasks with least baselines and using the best baseline time on each task as our difficulty estimate
Restricting the task suite to tasks with at least 2 baselines and using the worst baseline time on each task as our difficulty estimate
Adding noise to baseline times (included in Figure 6)
5. Task choice: Removing RE-Bench tasks
6. Different weightings of task families:
vs uniform (included Figure 6); also
familyy size
family size
7. Estimated training date vs estimates release date
8. Continuous scoring: Figure 16
H.1 Alternative curve fits
In our main result, fit an exponential curve (linear with log y axis) because the fit very good (R²² > 0.96, depending on the exact data and methodology). Linear and hyperbolic curves have poor fits (Figure 15). Because there are only frontier models in the time span we studied, and the exponential fit has such high R² with only two parameters, we think applying fits with more parameters would be more likely overfit than to give accurate predictions. particular, considered:
A double exponential function log(horizon) ~ + bexp(e- (release -aateedteeaeea + d)) is strictly more expressive than an exponential (because exp(x) =s x for small æ, they are equivalent when i small)
Likewise, the initial part of any saturating logistic function horizon ~ a (relcasc dato + d) looks very similar off (on our metric), it is essentially impossible to predict when or it will plateau, or for how long.
exponential, and without any evidence that AI horizon is leveling
Sophisticated stochastic models that asymptote to infinity often have large uncertainties; see Roodman [52], which applied superexponential diffusion model economic data. As we discuss in Section 5, large uncertainty may be appropriate when constructing prediction interval for a long-term forecast, quantitatively modeling the potential impact of the factors we mentioned is out of scope of this paper, we prefer to discuss these qualitatively.
35

---

Longth tasks agents have been complete autonomously
Continuous Scoring
4 hrx
Task time humanns that model completes
2hr
1hr
50% 505
4mii
Claude 5. Sonnet (New)
2mh
Claude 35 Sonnet (OId)
1im
50
preview
GPT-44
GPT-4 0316
3Ph Setrin/Srruuroooootoot
1980
Madel release date
Figure 16: Time horizon with continuous (non-binarized) scoring. Claude 3.7 Sonnet has 50% time horizon of nearly hours. We think this methodology captures more signal from 8-hour RE- Bench tasks, but overstates the time horizon of recent models, since it is easier to achieve an average score of 0.5 on most tasks than to match human performance 50% of the time. The slope is also likely an overestimate, because longer tasks tend to be continuously scored.
Length tasks agents have been able complete autonomously
80% success
Tasi time (for humans)) that modell completes with
Claude 3,7 Sonnet
2hrz
Cloude 3.5 Sonnet (New)
Claude
o3
15
oleprev
0 min
CPT-4o
GPT-4 T106
80% success
GPT-4 0314
1min
devinci-000 (GPT-3)
30
4.0c
Doubling time: 204 days 2020-01-01+ data R.: 0.96
258c
Model release date
Figure 17: Trend in 80% success rate time horizon. The doubling time is similar to the 50% plot, but horizons are substantially lower. 50% horizon trend shown in grey.
H.2 Horizons at 80 % success rate
In Figure 17, we show the trend for the 80% time horizon for models over time. The doubling time similar the 50% plot, but horizons are substantially lower.
H.3 Success rate versus inference cost
In Figure 18, we plot the success rate of models across our tasks as a function of token costs. All of the models studied perform better with larger token budgets, the majority of models show clear evidence of plateaus far below the maximum number of tokens allocated to each model.
36

---

Agent Performance on HCAST RE-Bench by Cost (Weighted by lask Diversityl
Numoon
best human for ench cask
08
Claude Ooos
Clludo 15 Sonnet plow
got- Bof--Sotttrborretrra
weighed
04
CRTT 0125
CRT-2
sonnnntt
Cluudo 3.5 Sonnut 04 min
00
Figure 18: Success rate by cost. Models were given sufficiently high token limits to reach plateau in success rate. Cost information not included for o3 and o4-mini.
2024 2025 Trendlines in Horizon for 189 software engineering, eybersecurity, general reasoning, and ML tasks
2019
Task timee hurnans) that model completes with
5 hrx
Couds 377
2hr
Caude (Nired)
Caude 85 Sonnet (C14)
preveeww
GPT-40
50% 50%.
4mmi
1106
2mh
opt-ss--trbb-iinnruut
1imin
awwinci-002 (CPT-3)
s0
Doubling 109 days
Doubling
1960
Model release date
Figure 19: 2024-2025 and 2019-2025 exponential fits for 50% time horizon.
H.4 2024-2025 Horizon growth trend
the 2024-200 2025 horizon growth trend continues to exceed the 2019-20200 slope, future work should apply change point analysis to determine whether the difference in slope is statistically significant.
H.S SWE-bench Verified
Data Collection We collected per-model, per-task results from the official SWE-bench evaluation results repository for frontier models: Claude 3 Opus, Claude 3.5 Sonnet (Old), Claude 3.5 Sonnet (New), GPT-4 1106, GPT-4o, and ol. Note that this is only 6 of the 11 models we include in the main result.
Time estimates SWE-bench Verified contains time estimates created by contractors that split tasks into four buckets: 15 min fix, 15 minutes-1 hour, I hour-4 hours, or > 4 hours. These estimates were based on the expected time it would take an "engineer who has had a few hours to familiarize themselves with the codebase" to solve the issue.
We verified annotator time buckets by running seven baselines across 6 SWE-bench Verified tasks. We baselined four tasks in the "< 15 min fix" bucket, and found that they took our baseliners 8, 26, 67, and 84 minutes respectively. We baselined two tasks in the "I hour-4 hours" bucket, and both baseliners took between 2-3 hours.
37

---

Task Time Bucket
Task Time Estimate
Average Baseline Time
< 15 min fix
3.9 min
32.9 min 131.6 min
15 min-1 hour
30.0 min
1-4 hours
120.0 min
> 4 hours
480.0 min
Table 8: We convert the SWE-bench Verified time annotations into task estimates, by taking the geometric mean of the time annotation. We caution that this likely underestimates the time each issue takes to resolve by a human baseliner without context-notablle we observe that the geometric mean of baseliner time for four randomly sampled tasks in the "< 15 minute fix" bucket is 32.9 minutes.
Model Time
Model Time
Model Time Horizon Ratio
Model
Horizon
Horizon
(Our Tasks)
(SWE-bench Verified)
Claude 3 Opus
6.42 min
0.83 min
7.8x
GPT-4 1106
8.56 min
1.18 min
7.2x
GPT-4o
9.17 min
5,96 min
1,5x
Claude 3.5 Sonnet (Old)
18.22 min
5.91 min
3.1x
Claude 3.5 Sonnet (New)
28.98 min
16.88 min
1.7x
o1
39.21 min
51.21 min
0.8x
Table 9: The time horizon of less capable models is substantially longer on our tasks than on SWE- bench Verified.
Analysis methodology We converted annotator time range estimates into per-task time estimates. We do this by taking the geometric mean of the starting and ending times of the time buckets. We select 16 hours the upper limit for SWE-bench Verified tasks, but there are only 3 tasks in this time bucket, and they do meaningfully affect the results. We then applied the same methods as used in Section 3 to convert per-model, per-task results and task time estimates into a model time horizon. To reduce the influence of similar tasks, we consider tasks that involve issues belonging to the same repository as part of the same task family, and down- weight the contribution of tasks by square-root of the number of tasks the same family.
Annotator vs contract baseliner times As shown in Table 8, we found that annotator time esti- mates are likely more inaccurate on the "<15 min fix" time bucket, as practice baseliners often take much longer than this to complete tasks. This results in easier tasks' difficulty being under- estimated, meaning that the SWE-bench Verified time horizon estimates likely underestimate true model performance for less capable models, which in turn leads to steeper exponential trend.
H.6 Messiness factors
Messiness factor labels were collected from internal contractors and one researcher.
The full list of messiness factors and definitions used are shown in Tables 10 and 11.
In Figure 20, we plot the messiness scores of HCAST tasks against log human time-to-complete. We find that messiness is correlated with task length, and that we have very few short tasks with a high messiness score. We also color each point by the performance of the model. Tasks with high messiness and which take humans a long time to complete tend to have low success rates, and vice versa for shorter, lower messiness tasks.
limitation of our messiness factors that they are least somewhat adversarially selected against current models. This is due to the combination of selecting for perceiving the factor to be relevant to model performance, and of these factors being expected to make tasks more difficult. As a result, we might expect model performance to be more negatively correlated with our task messiness measure than ought to be-and for predictive adjustments based on messiness to work less well for models released long before, or after, the time of this analysis.
38

---

Table 10: Messiness Factor Definitions 1-8
Factor
Definition
Real life source
Is this task: EITHER 1. Intended to be representative of some real world end-to-end task (if uncertain, is more likely than not) OR 2. Sourced from real-life problem
AND
3. Has NOT been intentionally simplified to an easier and smaller subtask.
Resource limited
Does this task:
Require the of a limited and one-time consumable resource where BOTH
Using the resource wisely is more relevant to task failure/ success than using the default resource limits wiscly (time, tokens, cost)
2. The resource is not casily replenishable within the task (c.g. limited number of queries to verifier, a permanent counter on guesses made that counts against the agent's score)
Not casily reset- table
difficult for the agent to reset the (task success relevant) state back the initial starting state desired, given no very obvious poor actions were taken? (c.g. mm -rf ing the env is too obviously poor) To be easily resettable, this should be able to be done without prior planning, but can involve back tracking based on the history of actions up to that point.
Irreversible
Do you think it would somewhat likely (>20%) that a 20th percentile human (from the distribution of people with background in the domain of the task but not prior context on this specific task) would make mistake during the first 50% of their time spent on the task that would permanently and irreversibly reduce the score they could achieve?
mistake
availability
Dynamic
Does the task involve interacting with a changing environment success, even if the agent takes no actions? This is true if this is the case for significant part of the task, even if it isn't true over the whole duration of the task.
ways sigmificant to task
environment
Difficult
Are there significant success-relevant parts of the task where is difficult for the agent to figure out whether a given effect should attributed its own actions (or lack of actions). or to something else?
counterfactuals
An example might be a task where the agent was optimizing a distributed computing system's 's performance. In this task the system occasionally experiences cascading slowdown and failures but this isn't mentioned in the task prompt. If the agent making some seemingly unrelated optimization it might be difficult to determine if the cause of these failures is downstream of the agent's 's direct changes, differences in load over time, random variation e.t.c
Another example might be case where adding debugging behavior like logging would change the behavior of the system being logged, which would make it harder to observe the system and tease apart the causal effects of different interventions.
Not purely auto- matic scoring
Does this task NOT have an entirely automatic scoring function? Specifically, for tasks implemented in the task standard:
1. There manual scoring instructions in those cases AND 3. There are no clauses about obeying rules that are not actually checked in the automatic scoring. (c.g A family which had the instruction "you may not attempt to reverse engineer X"" but this is not checked in the automatic scoring function)
branches which the scoring function returns None AND 2. The task provides
Implicit
it saliently possible to satisfy the stated success conditions in the prompt in such a way that the spirit of task is not actually fulfilled, or the product would not be uscful for the implicd purpose?
generalizability
required
For example, task asking the agent to train good small model may have various explicitly stated conditions for scoring well, like loss achieved, accuracy. But meeting the word of these requirements may not actually be enough for solution to useful for the implied purpose (in this case, something like 'make a good model). For example, model that takes a minute to produce single token doesn't 't seem very useful. This could be thought of as an implicit requirement that the solution generalizes to the implied use case.
The spirit of this column basically 'does the agent have to make sensible judgement calls on soft constraints that are not necessarily explicitly stated by the task prompt in order to produce an output that would he useful for the implied purpose of the task'? For toy tasks where there is not strong reason to think that there is implied purpose, this is false.
AND
Are these implicit generalizability requirements checked during scoring, either by the scoring function or by manual judgement/ review? (For example, some tasks have additional manual review stage check for exploit solutions, this counts. Other tasks use manual scorer judgement to check that the output is useful.)
39

---

Table 11: Messiness Factor Definitions 9-16
Factor
Definition
Non explicit
Both must true:
scoring
Firstly: a. The constraints on which the agent is scored are NOT all explicitly stated OR b. the scoring is described in a way such that there remains significant hidden information in the scoring)
description
AND secondly:
This lack of information about scoring produces task-relevant uncertainty about the best course of action within the task. (For example, managing tradcoffs between two different factors being optimize, but whose exact weightings in scoring are uncertain) For example, task that just says to return the number of days there were in March 2022 would not count having non explicit scoring. This is because there is task-relevant uncertainty introduced by this lack of information.
Is suboptimal
Does this task involve competing against other agents or programs that will actively exploit weakness suboptimal behavior of the agent (or outputs of the agent)? Tasks which have some adversary type program behavior are typical examples. The task docs not provide a way for the agent to check its answer or score midrun. Requires the agent to do some non con-trivial amount of work to check its answer.
behavior
exploited
No verification
provided
mechanisms
Real-time
Does this task require real-time coordination with other agents, or services, or require compe- tently orchestrating multiple real-time processes in parallel? For this, monitoring an ongoing training run would NOT count unless it was expected required by the task that the agent per- form multiple training runs (or similar) in parallel.
coordination
Self
Does this task require the agent to modify a portion of itself, or the code that is being used to interact with the environment, or to create a new tool for itself, or to create data that could be used to do this?
modification
required
Self
Does this task require the agent to improve itself in some that permanently increase its per- formance on a wide distribution of tasks? For example, finetuning data that works to improve the agents own general agency performance would sat- isfy this. However, ereating finetuning data that improves performance on some narrow task would not satisfy this (and would instead satisfy self modification).
improvement
agents, this could be creating
required
Information
The task requires gathering information that the agent would not expected to know ahead of time. For example, probing the behavior of a novel system.
secking required Novel situation
The task has some kind of unusual constraint, or umusual property, without which the task would be significantly easier or more rote, and which is significant source of the tasks diffi- culty.
H.7 Randomizing file names
To better understand the effect of messiness on agent and human performance, we modified an existing HCAST task, which involved looking through local LaTeX and PDF files find a specific piece of information, by randomizing file names, folder names, and providing less information on what to search for.
H.7.1 Methods
The specific HCAST task modified is called Local Research TeX. The unmodified version of this task initializes the agent environment to have a collection of nested folders and content, and then prompts the user to find some specific information, for example, with the prompt "According to the AR report, how many tasks did the best- pest--erformig agent complete?" Agents (or humans) must then search the local folders and content to find the specific answer to this question (in this case, "four"). We modified this task in a variety of ways, but the most notable variant involved:
1. Modifying the prompt from "According to the AR report" to "Find the answer to the fol- lowing question in one of the resources in thome/agen/resourcess"""
2. Modifying the all folder names to random integers, and all file names to meaningless names like "final" or "myl"ile."
We then ran agents for about a hundred runs across the original variant of this task and the modified version of this task, and compared the respective success rates. We also calculated the human time to complete using our standard bascline methodology.
40

---

Observed Model Success Rates
by Human Time-to-Complete and Messiness
3.0
0.8
0.7
Task Length for Human
Observed Success Rate
0.6
[lagi0 minutes)
2.0
0.5
1.5
1.0
0.3
0.2
0.5
0.1
0.0
0.0
2
6
7
Task Messiness Score
Figure 20: Messier tasks tend to be longer.
H.7.2 Results
Surprisingly, ol's performance is average better the messier variant of the report, and human baseliners performed much worse. Qualitatively, ol appears to do much better on the scrambled-
Task Variant
Success Rate
Baseliner Time
local_resestt www.eereeeehettte report
34%
24 minutes
local research tex/ar report scrambled files
50%
53 minutes
filename variant of this task because it not given the term "AR report" its prompt. In practice, ol often decides grep for the term "AR report" and fails to find it (as the actual report is named "ARA report"), and then makes a guess. On the other hand, when told to look in one of the resources, performs
general search, and finds the report it's looking for (more often). While this is only one limited example, we believe this further illustrates the challenges of quantifying and predicting how aspects of any task may effect agent performance- indeed, factors that make humans worse some tasks may indeed improve agent performance.
H.8 Correlation in performance between agents
In Figure 22, report the correlation matrix of per-task success rates between each pair of models.
Excess success rates Excess success rates Sabaerued Spredieted are metric for how much better S Spredicted
(or worse) an AI agent performed compared to what we would expect, given task's 's length and that model's s ability (expected success can be seen in Figure 4).
In Figure 23, we report the correlation matrix for the excess success rate (Sabserwed - Spredictea) Spredicted (where Spredtcttd is the success rate predicted from model's time horizon). While the correlations for excess success rate are lower than the correlation of raw task success rate (0.38 instead of 0.71), the fact that positive suggests additional factors that explain model success rates across tasks which are common across models.
41

---

Model Success Rates over Time, Time,split by Messiness (Weighted by Task Diversity)
50% loost Messy Tasks
(Messiness 2.3, N tasks 55)
10
Eleude Sunnet
Claude Sonnet
Cloude Sonnet
Weighted Success Rate
previww
4
201-U5 S.5---- bo-nuruutt
2022-01-m
2022-05-01
2022 mi
2033-05-o
20d do
2024-0--1 dl
202k-tt-ot df
2015-01-
2025
50% most Messy Tasks
(Messiness scofe x- 3,0 tasks 59
Weighted Success Rate Rate
2022-00-1
2022550
8222099 m
2013-0t01
2033-5-07
2033-08-01
2036-0-11 dl
32
20g 2024-09-t dt
201s O1-01
2015-05
Model Release Date
Figure 21: Model success rates on HCAST RE-Bench tasks, split by task messiness rating. Models have higher success rates the less messy tasks, but the rate of improvement over time is similar for both subsets. Both davinci-002 and gpt-3.5-ture t-rrbo instruct score 0 on the subset of HCAST RE-Bench with higher messiness.
I Time Horizons of Benchmarks in Other Domains
In Figure 27 we show derived time horizons on nine other benchmarks across a wide range of domains, including software, GUI usage, driving, and video understanding. When subset-level per- formance data were available we used maximum likelihood estimation to fit the 50% time horizon and the logistic slope parameter. When only overall benchmark scores are available, we assume slope parameter value is 0.6 (the same value METR-HRS).
Time horizons on these benchmarks show similar exponential trends the trends on the three datasets studied in this work (METR-HRS), but absolute performance varies greatly. Software and reasoning domains like scientitic QA (GPQA [53]), math contests (MATH [54], Mock AIME), semi-realistic software (METR-HRS) and competitive programming (LiveCodeBench [55]) all have 50-200+ minute horizons which are currently doubling every 2-6 months, similar to METR-HRE Visual computer use (OSWorld [56], WebArena [57]) time horizons are 40-100x shorter than the software and reasoning cluster but increase similar rates. Self-driving (Tesla FSD) is improving more slowly at around 0.6 doublings year. In video understanding (VideoMME [58]), models are capable of answering questions on hour videos with >50% success rates. However, since video understanding benchmarks show weak correlation between length and difficulty-for-modde (see
42

---

Comelation Observed Success Rates Between Agents
on
gen-3 Seuree mettcct
UAZ
SC4
OPfi4 106
6.
CFT-4o
Caudo 53 Sormes 10
877
6M
OUT
CoN
6M
CAR
02c
S
Paeee 10w.oooo 1or N
CPP o/t
GP"A 19G-
Cf Sommen
Claute Eornet
Figure 22: Correlation matrix of observed success rates across all models and tasks.
Correlation of Fractional Excees Succcas Rates Setween Agents
c912
4043
cm
634
SDM
6:77
025
C001
0.54
0:033
0.
678
3066
08
6/4
6.00
0.5
67
674
625
04
audde Sooner 10ic)
500
5006
0207
6/3
GOOD
6/4
0.5
Cheace
Newt
Zonk
Crle
0277
AM
mm
mum
2/4
1009
CATC3
EIT 4002
CPT44 TOC
digrener
Cluute Serve (Mees
Chaese 37 Soreet
Figure 23: Correlation matrix of excess success rates (defined by Sabserued SerrrraddSppeeteee Spredieted across all models and tasks.
Spradicted
43

---

Length tasks Al agents have been able to complete autonomously for 169 software engineering. cyoersecurity, general ressoning, and ML tasks
Task time (for humanng) th model completes with
1hr min
50% 50% success rate
hr, mi
1h
30 min
e1 presiew
GPT PPT Nov 23
GPT
GPT
GP
0.500
Model release date
Figure 24: Change time horizon of frontier models over time. Note: the data displayed is the same as in Figure 1, but with a linear axis.
Length tasks agents have been to complete autonomously for 169 software e engineering, cybersecurity, general reasoning, ML tasks
Task time (for hurnans) that model completes with 50%
hrs
Claude Sonnet
2hre
Cloude Ss Sonnet (New)
3.5 Sonnet (Old]
Claude Opus
30 min
Q3
15
8miin
al
allppev
GPT-4a
2min
GPT-4 Turbo
1min
GPT-4 0125
GPT 4 TD06
15 sec
GPT-4
tut-s Instruct
8sec
davinal-002 (DT-3)
450c
2.ec
Doubling days
2019 01 01+ data R": 0.95
1480
Model release date
Figure 25: Time horizon of all models we measured, including non-frontier models.
44

---

Length of tasks Al agents have been able to complete autonomously for 169 software engineering. cybersecurity, general reasoning, and ML tasks
Task time (for humens) that model completes with
4 hra
36 mm
8 min
50% success ratee
mmn
GPT 40
min
CPT-6
50%5
GPT35
990
GPT3
MMC
2.00
GPT2
Doubling time: 207 days 2018-09-034 data
R% 0.97
2019
2020
2021
2022
2023
2024
2025
2006
2027
Model release detee
Figure 26: Length in human expert clock-time of tasks that frontier models can perform competently over time. See Section 3 for details on time horizon length calculation. The line represents the linear regression fit, with confidence region calculated via hierarchical bootstrapping. In this plot, davinci-002 and gpt-3.5-turoo-nstrue are placed at the release dates of GPT-3 and GPT-3.5 respectively, and GPT-2's score is imputed as zero for longer tasks for which our scaffolds are incompatible. Note: this is the same as Figure but presented differently.
hrs
GPOA Diamond
Testa FSD
LiveCodeBench
SWE-bence Verified
MATH
4
1 min
OSWorld
15 59c
WebArena
RLBench
(Original Time Horizons)
2020
2022 Model release date
2024
CC-BY
metr.org
Figure 27: 50% time horizon trends across benchmarks over many domains. Most other domains show similar exponential trends to the results from this work (METR-HRS).
45

---

Benchmarks vary in how strongly model performance correlates to task length
(each point is a model)
54,60x
4.00
Model success strongly related to task length
Fallure rate Increase per task length
5.35x
1.66
2.96x
1.09
(odds ratio)
(log scalo
2,02x
1.58x
0.46
1.34x
0.29
1.21x
0.19
Model
1.13x
weakly related
0.12
task length
1.08x
0.08
METR.RRS (Original)
SWE banch verdfed
Video
GPGA Diamand
LiveCodeBanch
Hock
AIME
Figure 28: Correlations between task time-to-compleds and model success rates. Video MME shows only weak correlation between length and difficulty, which may make time horizon inappropri- ate metric for tracking Video MME performance.
Figure 28, time horizon may not be an appropriate metric for these benchmarks. No domain we examined displayed trends were clearly sub-exponential.
For more details, see our blog post: https://metr htt.:///woo......oooooroaraaaaar
J
Code
The code to reproduce the results in the main body of the paper can found at: https://github httMM////eeaaaaaalllalaaaaaaaaaa
K
Author contributions
Thomas Kwa led the development SWAA tasks, wrote most of the data analysis code, and wrote the plurality of the final draft including generating about half of the figures. Ben West performed the initial analysis, and co-led the project. Joel Becker led human baselining processes and data collection for RE-Bench and HCAST tasks, and contributed to early data analysis code. Amy Deng created most of the SWAA tasks, collected human baselines and Al agent results on the SWAA tasks. She also investigated agent and task failures and assisted with obtaining agent results on HCAST tasks. Katharyn Garcia wrote evaluation and data analysis code. Max Hasin led the qualitative impressions work and wrote key sections, gathered internal PRs, and contributed to data analysis code and baselining collection. Sami Jawhar contributed to the task-running and data analysis infrastructure. Megan Kinniment performed the messiness experiments, excess success rate and initial reliability analyses, oversaw the creation many HCAST tasks, and contributed to figures and writing. Nate Rush led the SWE-Bench and Internal PR experiments and drafted those sections of the paper. Sydney Von Arx oversaw internal validity analysis.
Brian Goodrich contributed to the development of the modular- public agent, and created the flock and duet agents. Nikola Jurkovic baselined SWAA and RE-Bench tasks and tested RE-Bench tasks. Seraphina Nix contributed to writing and revising the paper and figures. David Rein managed the development and collection of HCAST tasks and oversaw the human baselining process for those tasks. Lucas Jun Koba Sato led the development of the AI agents and collection of agent results
46

---

on HCAST tasks. Chris Painter suggested substantial framing and presentation changes. Neev Parikh contributed running external validity experiments.
Elizabeth Barnes helped develop the basic concept, suggested experiments/analyses, gave feed- back, and assisted with writing/early task developmen//analysi code. Lawrence Chan co-led this project, including setting the overall direction, helping decide what experiments to run, and con- tributing much of the writing for our paper.
47

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize all important claims, assumptions, results, and limitations, except some minor results in the appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See the conclusion for a short summary of our limitations, appendices E.4, B.2, and E.3 for more details, and 4 for results that partially address these limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech- speech-to-text system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
48

---

Answer: [NA]
Justification:
This paper is mainly empirical and does not prove theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The models and scaffolds used are described, models are all publicly available (except those which have been deprecated), and raw data and analysis code are provided in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
49

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Although we provide basic analysis code sufficient to reproduce the main re- sults from data, there are several challenges sharing all code used; it is split across several repositories, difficult to anonymize, and some benchmarks (both internal and exter- nal) are private prevent data contamination. We believe the analysis notebook provides better usability for reviewers and sufficient reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nip)) cc/
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cttccc//uuu hetllllcggudee//oooo
simmmisssii onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: For details of our methodology, see and various other sections.
Guidelines:
The answer means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Ys]]
Justification:
error bars generated using hierarchical bootstrap. which counts for covariance between similar tasks and thus gives wider, more conservative error bars. For figure we clearly describe the boxplot, which is 80% rather than 95%.
See 3;
Guidelines:
The answer NA means that the paper does not include experiments.
50

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See section C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpeerrppppp.ppppppeeeetteeiiee
Answer: [Yes]
Justification: All workers were fairly and legally paid and consented to this experiment. Data collected from humans is limited to success/failure and time taken on tasks, and is anonymized to the extent practical.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
51

---

Justification: We discuss the impacts of capabilities forecasting in several places in the introduction, conclusion, and appendix, including economic value created by future pow- erful AI, possible catastrophic risks, and the application to determining appropriate risk mitigation measures. This paper does not develop specific technologies vulnerable di- rect negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The only datasets we use are either original or previously published.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all benchmarks and data used; the only models or other assets we use are the OpenAl and Anthropic models we evaluate through their APIs.
Guidelines:
52

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: The full text of instructions would be impractical to share, as it includes around 170 separate task instructions for private internal and external tasks, many which reveal the details of their tasks. As for compensation, SWAA baselines were collected on internal employees rather than contractors or crowdworkers.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to NeurIPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
53

---

Answer: [NA]
Justification: As above, all original human data was collected on internal employees who consented.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were used for exploratory work and editing but not core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https:/ http://eeeruicoeeeeeeeeeee /ttp:///aaeeeeene/////////////// for what should or should not be described.
54


Reviews:
[{"review_id": "W6KIEMxMde", "reviewer": "Reviewer_m8QB", "summary": "The paper \"Measuring AI Ability to Complete Long Software Tasks\" proposes a new metric called the 50% task completion time horizon, which quantifies how long a task takes a human to complete that an AI model can solve with 50% reliability. By benchmarking 13 frontier AI models from 2019 to 2025 on 170 software and research tasks, the authors find that this time horizon has been doubling approximately every seven months, indicating rapid progress in AI capabilities. The study uses datasets like HCAST, RE-Bench, and a new SWAA suite to evaluate tasks ranging from seconds to hours, comparing AI performance to skilled human baselines. Key drivers of improvement include better logical reasoning, tool use, and adaptability. However, AI agents still struggle with “messier” tasks that lack structure or clear feedback. Authors extrapolate the results to say that by 2028–2030, AI systems may autonomously complete software tasks that currently take humans a month.", "questions": "I think that answering the next questions could help to improve the paper:\n\nHow do you account for the potential exponential growth in task complexity over time, and could this affect the interpretation of the 50% completion time horizon metric?\n\nWhat measures have you taken to ensure that the models evaluated were not exposed to benchmark data during training, and how do you mitigate potential contamination effects?\n\nCould you analyze or report how the gap between 50% and 100% task completion evolves, especially for longer or more complex tasks?\n\nHave you considered evaluating model performance on unseen or adversarial variants of the benchmarks to better assess generalization?\n\nCan you provide evidence or case studies showing how well the 50% time horizon metric correlates with real-world software development performance?", "limitations": "I would recommend the authors to read the paper: Larger and more instructable language models become less reliable (https://www.nature.com/articles/s41586-024-07930-y) as maybe the models work better and better with complex problems but forget the simple and routine ones (much more useful) because the large models have become too focused on solving increasingly complex problems (leaving aside the comments of possible contamination).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n*.- Novel Metric for Long-Horizon Tasks: The introduction of the 50% task completion time horizon offers a fresh, intuitive way to measure AI progress on complex, real-world tasks, bridging the gap between toy benchmarks and practical capabilities.\n*.- Empirical Breadth and Historical Scope: The study evaluates 13 models across six years (2019–2025) and 170 tasks, providing a rich longitudinal view of AI progress, especially in software engineering and research domains.\n*.- Insightful Trend Analysis: The finding that AI capability (as measured by the 50% horizon) doubles every ~7 months is a compelling and quantifiable insight into the pace of AI advancement.\n\nWeaknesses\n*.- Neglect of Task Complexity Growth: The paper does not analyze whether the complexity of tasks increases over time. If task difficulty grows exponentially, the gap between 50% and 100% completion could also grow non-linearly, potentially biasing the metric and overstating progress.\n*.- Potential Benchmark Contamination: There’s a risk that models have been exposed to benchmark data during training, especially given the popularity of certain datasets. This could inflate performance and obscure whether improvements stem from genuine generalization or memorization.\n*.- Limited External Validity and Real-World Testing: While the benchmarks are diverse, the paper lacks a clear discussion of how well these results translate to real-world software development environments, where ambiguity, collaboration, and evolving requirements are common.", "was_revised": true, "final_justification": "I think that the paper should be accepted if the other reviewers agree.", "rebuttal": "Thanks for the detailed review! We’re glad that you find our core time horizon metric novel and intuitive, and that you appreciate the breadth of tasks and models used in our empirical evaluations. We respond to each of your questions below. \n\n1. **Task complexity growth for longer task lengths:** We find that a logistic curve against log-task-length space is the best way to model an agent’s success, and that the slopes of such logistic curves are fairly similar for each agent. If longer tasks were exponentially more complex, we would see steeper and steeper slopes, with agents able to consistently handle (say) 1-hour tasks but almost never 2-hour tasks. Instead we find that the 50% time horizons are roughly 5x longer than 80% time horizons (Section 3.2.1), at least on our data. This can perhaps be reconciled by a model where exponentially longer tasks have exponentially more steps, but the failure rate of agents is also decreasing exponentially over time.\n2. **Data contamination concerns:** Since RE-Bench was released in late 2024, HCAST in 2025, and we developed SWAA privately, the problems are either private or past the training cutoff date of all models we evaluated (o3’s cutoff is June 2024). As for potential future models, some of the tasks we evaluate on are now public. HCAST authors claim 35/91 of the solutions in a subset of the 97 we used are potentially memorizable. RE-Bench has a public repo, but due to the small number of tasks we think it’s unlikely that a lab would train on it specifically. So overall, we think data contamination is not a concern on the models in this paper but may become relevant in the future. In the camera-ready, we will also include time horizons on LiveCodeBench, which is continuously updated to prevent contamination.\n3. **Gap between 50% and higher% task horizons:** It is difficult to measure time horizons at reliability close to 100%, since one would need hundreds of independent samples and <<1% label noise to distinguish between 98% and 99% accuracy. However, we observe that 80% horizons are improving at similar rates (Figure 17), and 80% horizons are roughly 5x shorter than 50% horizons on all models, with the curve perhaps slightly steeper on more capable models (Figure 4).\n4. **Unseen/adversarial datasets and benchmarks:** As discussed in Q2, all of the datasets are likely unseen to models. About other benchmarks, data were designed to be realistic rather than adversarial. However, we have looked at other benchmarks like SWE-bench Verified (Section 4.1), as well as GPQA Diamond, LiveCodeBench, and AIME (will be in camera ready). As discussed in the paper, some models have much shorter time horizons on SWE-bench Verified; on GPQA, 1-2x longer; this paper’s tasks are roughly in the middle.\n5. **Generalization to real-world software engineering performance:** We perform several external validity checks (Section 4): measuring performance on SWE-bench Verified (4.1), subsets of HCAST that are more realistic (F.2), and on our internal PRs, which we manually reviewed for code quality as well as correctness (C.2). It also seems to be empirically true that models with higher time horizons also perform better on other SWE benchmarks, and anecdotally are more helpful for software engineers. To more definitively answer this question, in the future we’re hoping to study the software engineering performance of models at a higher level of realism, using real-world in-situ software engineering tasks done by professional software engineers. \n\n**Limitations:**\nThanks for referencing this work, we'll discuss and cite it! The referenced work is consistent with our qualitative analysis of failures (Table 2) where we observe that many failures of less capable agents are obvious, like repeated failed actions, whereas o1 makes mistakes that might be harder to supervise, like submitting answers without checking. We do observe that recent models like Claude 3.7 and o3 get near-perfect accuracy on tasks shorter than ~4 minutes (again see Figure 4), but we can’t rule out that performance of more capable models is lower on a different distribution of simple tasks."}, {"review_id": "UaqETAG94w", "reviewer": "Reviewer_irnC", "summary": "- This paper proposes a new metric to evaluate AI abilities, the 50% task-completion time horizon, which is the time humans typically take to complete tasks that AI models can complete with a 50% success rate.\n- The authors evaluate a dozen or so models on three datasets and compute the task-completion time horizon metric.\n- The authors find that the task-completion time horizon has been growing exponentially in the past few years.", "questions": "- Can the authors discuss why this particular metric was selected and how to mitigate concerns about the population of human baseliners that are used?\n- Can the authors discuss how benchmarking more agentic frameworks might affect their results?\n- Can the authors explain what the primary takeaways from this work are beyond the exponential trends observed in existing data?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality:\n- I agree with the author's reasoning about why existing metrics are flawed. However, they fail to provide a sufficient justification for why X% task completion time horizon is the “right” metric that the community should be thinking about.\n- The reliance of the metric on human baseliners means the results might depend a lot on the population of humans that is recruited. Can the authors comment on this?\n\nClarity:\n- Since the authors focus on software tasks, why do you only evaluate LLMs on their own? Many software tasks (e.g., SWE-Bench) cannot be solved with one LLM call and rather need more agentic frameworks.\n- While the authors claim to focus on realistic tasks, the collection seems like a somewhat adhoc combination of software-related tasks. More discussion on how the set of tasks covers software engineering development and life cycle would be helpful.\n- I noticed there is consistent writing. For example, L47 states that 13 models were evaluated, L111 says 12 frontier and 3 “near frontier” models, and then L256 states 11 models. The inconsistencies throughout this paper are concerning.\n\nSignificance:\n- I am finding it difficult to understand what the significant takeaways or contributions are of this work and how it implicates future work. The primary contribution of the work seems to be evaluating models on existing datasets and baseliner results with relatively simple methods. \n\nOriginality:\n- This type of metric seems fairly novel in the literature. Most other evaluations do not have an axis of human task completion time.", "was_revised": true, "final_justification": "Per my review, there are multiple presentation aspects that can be improved in the next version.", "rebuttal": "Thanks for your review! \n\nWe’d like to start by clarifying that our work is studying the ability of AI agents to perform agentic tasks – both RE-Bench and HCAST tasks require models to take multiple actions in series, and many of the agent runs use millions of tokens. We discuss these tasks in more detail in Section 3.1 and the ReAct-based agent scaffolds we used in Sections 3.3 and C.3. We agree that we could’ve been more clear that we’re studying AI performance on agentic tasks in the introduction and abstract; we’ll revise the terminology there to be more clear and consistent with the rest of the paper. \n\nWe also apologize for the typos on the number of models. In total, we evaluated 16 models (Figure 25), of which 12 advanced the state-of-the-art in terms of time horizon and 4 did not (and thus were ‘near-frontier’). We’ll edit the text to clarify this; thanks for pointing out these errors. \n\nTo address some other questions throughout the review: \n1. **Population of human baseliners:** We agree that the task lengths measured on different human populations will differ substantially, and we do some robustness checks related to this in Section 4. On SWE-bench Verified (Section 4.1), whose task lengths are estimates of high-context maintainer time, models have shorter time horizons. We also measure the ratio of time taken by contractors vs maintainers on our internal PRs (Section C.2). In our main analysis, the population of human baseliners is people from top-100 universities or equivalent aptitude, with several years of experience in their field (Section 2.2); this is consistent between HCAST, RE-Bench, and SWAA, with details of each in C.1.\n2. **Task distribution vis-a-vis SWE development cycle:** The HCAST (Rein et al) and RE-Bench (Wijk et al) papers have a breakdown of the categories each task falls under. HCAST is 30% MLE, 20% SWE, 20% cybersecurity, and 30% general CS-relevant reasoning, with subcategories given in Table 1 of the HCAST paper. RE-Bench is 7 long ML research engineering settings, each of which inherently require subskills like algorithmic thinking and debugging. As for SWAA, a breakdown between math, code completion, and other decisions is in appendix B.1.3. Overall, we have coverage of several SWE subfields and process steps, but lack the ability to measure some elements of such work that are inherently difficult to benchmark, like project management and code quality.\n3. **Choice of time horizon metric:** We chose the time horizon metric due to its relevance to the role of AI in the future economy. When companies choose whether to automate a task, two important factors are the amount of human labor the AI can replace with minimal supervision, and the reliability of the AI. The time horizon metric combines these into a measure of performance that should relate to which tasks must still be done by human engineers vs. can be automated. Time horizon also has good explanatory power for the reliability of agents at different task lengths (Figure 4); the logistic fit is reasonably good and we find that agents’ 80% time horizons are roughly 5x shorter than their 50% horizons (Figure 17).\n4. **Other takeaways:** Using time horizon, we observe that agent capability is more consistent with contractor time than experienced maintainer time (4.1, C.2), and that agents perform under their time horizon on more realistic tasks (H.2). By using human baselined time, we also quantify the degree to which agents can do impressive tasks but are unreliable on longer tasks (Figure 4). In the camera ready, we will also have an analysis of time horizons across 9 benchmarks in other domains like GPQA (scientific QA), AIME (competition math), RLBench (robotics), and OSWorld (graphical computer use). Models’ time horizons in other domains differ from their software time horizons, though we find similar rates of increase, and time horizon trends allow better forecasting of when automation in these other domains will be practical."}, {"review_id": "fZv25QHZQh", "reviewer": "Reviewer_woeK", "summary": "This paper introduces a novel measurement for assessing AI capabilities: the 50% task-completion time horizon, i.e. the amount of time it takes a human to complete tasks that the AI model achieves 50% success rate. The authors tested over 100 tasks across three datasets. Based on the results, they conducted backtesting and provided forward-looking analyses, such as estimating when AI systems might be able to complete one-month-long software tasks.", "questions": "Considering the high cost of collecting human baseline data, do you have any insights on reducing variance or bias in the measurement process to obtain a more robust result.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed 50%-task-completion time horizon is an interesting and reasonable metric for evaluating AI model capabilities. Unlike traditional success rate or performance-based metrics, it better captures model behavior in more realistic and automated settings.\n\n2. The authors validate this metric across 170 diverse tasks and 12 frontier models. The results are reasonable and informative.\n\n3. The authors further apply the metric to perform verification and exploration, including replicating their measurement on SWE-bench Verified and extrapolating toward one-month-horizon AI. These analysis contributes to both a robust measurement approach and valuable insights.\n\nOverall, using time horizons as a metric provides a fresh perspective on evaluating large language models. Although similar ideas were mentioned in the GPT-4o technical report, this paper stands out by offering a more systematic and comprehensive treatment of the concept.\n\nWeaknesses:\n\n1. The human baseline sample size is relatively small, which could bring significant bias into the overall evaluation.\n\n2. The paper may be better suited for a datasets and benchmarks track.\n\n3. The paper lacks discussion of recent works that also explores LLM capability evaluation beyond traditional success-based performance metrics. For example: Miserendino, Samuel, et al. \"SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?.\" arXiv preprint arXiv:2502.12115 (2025).", "was_revised": true, "final_justification": "I have no further concerns after reading the author's rebuttal, and decide to maintain my positive score of the paper.", "rebuttal": "Thanks for the detailed review of our work! We’re glad that you think our work provides a fresh perspective on evaluating LLMs and that you find our results informative and insightful.\n\n- **Human baseline sample size/population variance:**  We agree that the task lengths measured on different human populations will differ substantially, and we do some robustness checks related to this in Section 4. On SWE-bench Verified (Section 4.1), whose task lengths are estimates of high-context maintainer time, models have shorter time horizons. In our main analysis, the population of human baseliners is people from top-100 universities or equivalent aptitude, with several years of experience in their field (Section 2.2); this is consistent between HCAST, RE-Bench, and SWAA, with details of each in C.1.\n- **SWE-Lancer:** Thanks for bringing up the SWE-Lancer paper! We’ll cite and discuss it in our work. \n- **Reducing variance or bias in measurement data:** We have several ideas for how future work might increase measurement precision:\n  - As we mention in Section 3.2, errors between time horizons are correlated, so relative comparisons have smaller error bars than absolute ones, e.g. a bootstrap test shows o3 is significantly above Claude 3.7 and above trend. We are also somewhat conservative with our CIs as they do 3-level hierarchical bootstrap, which some literature suggests overestimates variance.\n  - Correcting for developer-specific effects (some humans are faster than others): not included because we had too few tasks with multiple human baselines to calibrate them accurately. The baseliner pool has different specialties for different tasks, so the tasks done by different baseliners were often disjoint.\n  - Tasks could have subtasks like Cybench-- not done in this paper because the long tasks were taken from Rein et al. and Wijk et al., and the shorter SWAA tasks are only one step.\n  - Tasks could be stratified by difficulty. This would require a new benchmark constructed to have task variants that are easier vs harder for models at the same length for humans, which would be difficult to keep realistic.\n  - Item response theory methods could be applied to place more importance on more informative tasks. We rejected this for simplicity but it could be worthwhile in other contexts.\n  - Overall we think there isn't enough statistical juice to be worth the complexity; as for future work we think it's best to collect human baseline data from real software development in the wild, which will have much lower marginal cost."}, {"review_id": "EWYyKIRkQc", "reviewer": "Reviewer_npLT", "summary": "The paper evaluates the performance on agentic tasks (including e.g. SWE and ML tasks) of 12 models released over the past 6 years, and analyses how models get better over time. To do that, it uses RE-Bench, HCAST and a newly created set of short tasks called Software Atomic Actions, and draws on extensive human baseline data comprising hundreds of human baseline trajectories taking in total >2,500 hours by skilled SWEs and ML researchers. Based on this data, each task is assigned a task length (how long humans took on average to solve it). The paper’s main contributions are: a) The definition of a model’s time horizon as the (estimated) length of tasks which the model can do with 50% reliability. This is an intuitive measure of a model’s agentic capabilities. b) The discovery of an exponential trend in models’ time horizons: since 2019, models’ time horizons have doubled approximately every 7 months. The trend approximately holds up under various ablations and can be used to forecast the arrival of future highly agentic models.", "questions": "- IIUC, the human baseline methodology is slightly different between the different datasets (SWAA, HCAST, RE-Bench). How worried are the authors that these differences systematically bias the fitted trend?\n- How far do the authors think the trend will continue? Does it seem possible that e.g. week-long or month-long tasks will tend to be more messy, and therefore the trend fitted on non-messy tasks won’t hold up?\n- (Nit) Table 2 would be more informative if the analysis was done on the same set of tasks", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This is a highly significant and original model evaluation paper. Most evaluation papers focus on a narrow capability and are quickly saturated, as a result of which they cannot be used for tracking and forecasting model capabilities across multiple years and model generations. The discovered trend is simple to understand and highly consequential for timelines to AGI and beyond, and any efforts aimed at preparing society for that transition.\n- The human benchmarking effort that went into this paper (as well as HCAST and RE-Bench), is immense, especially in light of the fact that all the baseliners were experienced SWEs / ML researchers / cybersecurity experts. This makes the time-horizon metric grounded and highly interpretable.\n- The paper is very rigorously done. It has many ablations, e.g. accounting for task messiness, a sensitivity analysis for the final forecast, a follow-up experiment on real-life pull-requests, … Many limitations or follow-up questions I had while reading the paper were discussed in detail in the appendix.\n- The writing is clear, high information-density, and the figures / tables are informative.\n\nWeaknesses (minor)\n- While the logarithmic axis makes sense for longer time horizons, I’m not sure that the distinction between a 1s, 2s, 4s, 8s, 15s, or 30s task is that meaningful or reliably measurable, and so I’m not sure how seriously to take the results for the early models.\n- One interesting observation is that models’ competence at different levels of reliability spans quite a few orders of magnitude in terms of time horizons. E.g. tasks the model can do 90% of the time can be ~60 times shorter than tasks the model can do 10% of the time. An analysis of this in more detail could improve the paper.", "was_revised": true, "final_justification": "Very strong paper and thorough rebuttal.", "rebuttal": "Thank you for your detailed review of our work! We appreciate that you found our work highly significant and original, and that you found our writing clear and methodology rigorous. \n\n- **Distinction between very short tasks:** We agree that short tasks are difficult to measure, and we attempted to mitigate this by designing the SWAA baselining app for minimal friction. The median SWAA task has a stdev of a factor of 1.43 in its human baseliner time (i.e. median_{t \\in tasks} exp(stdev_{all baseline times on task t}(log(human time))) = 1.43), much lower than the range of SWAA task lengths, which is about a factor of 20. Note a limitation of SWAA is that many short segments of intellectual work are not capturable in standalone tasks, so SWAA is still far from a representative distribution of 2-second or 15-second tasks.\n- **Differences between time horizons of different reliability:** We agree that it’s interesting that the tasks model can complete very reliably are so much shorter than tasks they can complete 50% or 20% of the time, though we’ll note that this seems consistent with anecdotal evidence about how current frontier models can be competent on average but are not reliably so. Interestingly, we do observe that 80% horizons are improving at similar rates to 50% horizons (Figure 17), and we find this exponential increase more convincing than the exact time horizon figures.  We’ll edit the paper to include more discussion about these observations. \n\n**Questions:**\n\n- **Differences between SWAA and other baseliners:** The baseline methodologies generally match the ways that humans interface with tasks in the real world but are not perfect. For SWAA, humans immediately type an answer, much like a short segment of SWE work. On HCAST and RE-Bench, the baseliner instructions are somewhat different-- \"time to reach target score\" vs \"maximize score after 8 hours\"-- so we wouldn't be surprised if the effective times differed by 25% or so. This error is swamped by variance derived from the limited number of long tasks, but a consistent methodology would be important for more precise time horizon estimates. We also do robustness checks, including fitting a trendline without SWAA (Figure 10) which is consistent with the main trend. In the camera ready, we’ll also have additional SWE-bench Verified data and analysis of GPQA and LiveCodeBench, which use different human baseline or time estimate methodologies.\nWe also note that when forecasting, the trendline slope only weakly depends on the bias of human baseline data since the y axis spans orders of magnitude. A 2x bias in SWAA tasks will only affect the projected date of 1-month software AI by a few months.\n\n- **Speculation on future trend:** Predicting the future is always difficult, but we can speculate on the four factors in section E.2. The effectiveness of agency training could push the trend in either direction at any time, but the prevalence of reward hacking behavior suggests it might cause faster progress on verifiable tasks than nonverifiable ones. Compute scaling will probably continue for between 2-5 more years before it slows down the trend. AI R&D automation will probably depend on the ML time horizon reaching some unknown threshold. As for AGI generalizing better than current AI systems, we still think this could happen but have little idea when. Our overall guess is that the future trend on verifiable tasks will be around every 3-5 months, and on realistic tasks about every 4-8 months, with large uncertainties. This will continue until either compute limitations, AI R&D automation, or AGI happen.\n\n  As for the messiness of week+ long tasks, we think the low messiness of our dataset is an important limitation at all task durations. Most week-long tasks will have at least some messy segments, and the time horizon may be much lower on those subtasks leading to higher overall failure rates. It is rare for a software engineer to go a week without encountering poor specifications or resource limits. In some analysis which will be in the camera ready, OSWorld and WebArena (agentic computer use) time horizons are as low as 2-4 minutes, but tooling has improved faster than we thought with systems like ChatGPT Agent, so some types of messiness may have less of an impact."}]

Author rebuttal:

Meta-review: The paper presents a perspective of 50% task completion time horizon and empirical analysis. The review identifies its value to the relevant problems and evaluation. The paper would be expected to clarify and enhance its final version by addressing the concerns about various aspects, in particular, the significance and originality of the work to the main track, the assumption, evaluation settings and robustness analysis of the metric, as well as other relevant factors and aspects such as complexity, benchmarking, and presentation issues.
Decision: Accept (poster)