# How 2025 AI Forecasts Fared So Far

At the end of 2024, there was a lot of discussion about whether AI scaling was hitting a wall and whether we would reach AGI soon.

We created [a forecasting survey](https://ai2025.org/) that would track key markers of AI progress in what might turn out to be a pivotal year. After filtering the data, we had 421 unique respondents. All questions were optional.

The survey was open from Nov 30th 2024 to Jan 20th 2025 – during which OpenAI o3 was announced, so we can compare forecasts before and after the announcement.

In this post, we summarise respondents' forecasts, and look at how they're holding up so far. At the end of the year, we'll resolve all the forecasts and write up the results (you can join the mailing list [here](https://theaidigest.org) to get notified about that).

**Note**: I'd recommend [reading this post on AI Digest](https://theaidigest.org/ai2025-analysis-may), where you'll find expandable sections for background info, and how responses varied based on forecaster timelines and P(doom).

**AI Research: RE-Bench**

### **1) What will be the best normalized score achieved on the original 7 RE-Bench tasks by December 31st 2025?**

[RE-Bench](https://metr.org/AI_R_D_Evaluation_Report.pdf) consists of 7 challenging, open-ended **Machine Learning optimization problems** created by METR to highlight "automation of AI research and development (R&D) by AI agents".

How much AI will speed up AI R&D is a typical point of disagreement on whether AI progress will speed up or slow down. Research Engineer benchmark (RE-bench) is one of our best current indicators.

![Distribution of responses for AI Research: RE-Bench](https://theaidigest.org/ai2025-analysis/plots/distributions/ai-r-and-d:-re-bench.svg)

Forecasters were bullish – they expected most of the gap between 2024 AI performance and the best observed human performance to be closed in 2025.

Since December, the median human expert score has already been beaten, as [METR reports](https://metr.github.io/autonomy-evals-guide/openai-o3-report/#summary-of-results:~:text=When%20given%20a%20total%20time%20budget%20of%2032%20hours%2C%20o4%2Dmini%20exceeds%20the%2050th%20percentile%20of%20human%20performance%20averaged%20across%20the%20five%20tasks.):

> *When given a total time budget of 32 hours, o4-mini exceeds the 50th percentile of human performance averaged across the five tasks.*

The maximum possible score is around 2.0.

Note that the listed state of the art (SOTA) performance from o4-mini is approximate (it's on 5 of the 7 questions, and it's read off [this graph visually](https://metr.github.io/autonomy-evals-guide/openai-o3-report/#summary-of-results:~:text=On%20the%20subset%20of%20RE%2DBench%20tested%20here%2C%20o3%20performed%20worse%20than%20other%20public%20models%20tested.%20o4%2Dmini%20achieved%20the%20highest%20performance%20recorded%2C%20primarily%20owing%20to%20exceptional%20performance%20on%20one%20of%20the%20five%20environments%20used%20in%20this%20evaluation.)).

### Pre/Post o3 Comparison

Part-way through the forecasting survey, OpenAI o3 was announced with strong reported performance. Forecasts submitted after this announcement were generally more bullish.

![Pre/Post o3 comparison for AI Research: RE-Bench](https://theaidigest.org/ai2025-analysis/plots/o3-comparisons/ai-r-and-d:-re-bench.svg)

**Software Engineering: SWE-bench Verified**

### **2) What will be the best performance on SWE-bench Verified by December 31st 2025?**

The [SWE-bench Verified](https://www.swebench.com/) benchmark consists of **500 software engineering problems** drawn from real GitHub issues and verified by human annotators to be non-problematic.

![Distribution of responses for Software Engineering: SWE-bench Verified](https://theaidigest.org/ai2025-analysis/plots/distributions/swe-bench.svg)

Forecasters expected progress on the popular SWE-bench Verified benchmark to continue apace, likely saturating by the end of 2025. In December 2024, performance rose from 55% to 62%, and since then it has risen to 70%.

Note that we don't count pre-release o3's reported 71.7% score because [that was PASS@5](https://arxiv.org/abs/2502.06807) (best of 5 attempts). We only accept PASS@1 (best of 1 attempt) because human software engineers in the real world generally do not have access to the scoring metric (ground truth unit tests) present in the SWE-Bench Verified dataset when resolving a GitHub issue.

**Cybersecurity: Cybench**

### **3) What will be the best score on Cybench by December 31st 2025?**

[Cybench](https://cybench.github.io/) consists of **40 professional-level** [**Capture the Flag**](https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity)) **(CTF) tasks** from 4 distinct CTF competitions, chosen to be recent, meaningful, and spanning a wide range of difficulties.

![Distribution of responses for Cybersecurity: Cybench](https://theaidigest.org/ai2025-analysis/plots/distributions/cybench.svg)

o3 hasn't been evaluated publicly on Cybench yet. On OpenAI's internal CTF benchmark, Deep Research is the leading system, scoring 70% on their professional CTF suite, while o3 scores 58% – up from o1's score of 23% ([source](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf#page=16)). So we expect o3 and Deep Research to outperform o1 on Cybench by a considerable margin.

Note that when the survey was running, we weren't aware of that US AISI evaluated o1 and it scored 45% on Cybench. In our survey materials, we therefore told respondents that the SOTA was 35%, which is what Claude Sonnet 3.5 (New) scored.

**Computer Use: OSWorld**

### **4) What will be the best performance on OSWorld by December 31st 2025?**

[OSWorld](https://os-world.github.io/) is a benchmark measuring AI Agent performance on **369 computer tasks** involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. OSWorld has 4 possible input data formats, which vary by using computer vision or accessibility interfaces.

Computer use is an emerging capability of AI systems, which has gained more attention since Anthropic released a [computer use API](https://www.anthropic.com/news/3-5-models-and-computer-use) for developers in October 2024.

Current products like OpenAI's [Operator](https://openai.com/index/introducing-operator/) are unreliable. But competent computer-use might change that – and is a step towards drop-in remote workers.

![Distribution of responses for Computer Use: OSWorld](https://theaidigest.org/ai2025-analysis/plots/distributions/osworld.svg)

Forecasters expected most of the gap between 2024 AI and human performance to close by the end of 2025. By April, we're already seeing strong progress on OSWorld, about halfway to forecasters' median estimate. See [Agent Village](https://theaidigest.org/village) to get a sense of current computer use capabilities.

**Mathematics: FrontierMath**

### **5) What will be the best performance on FrontierMath by December 31st 2025?**

[FrontierMath](https://epochai.org/frontiermath/the-benchmark) is a benchmark with hundreds of original, expert-crafted mathematics problems spanning major branches of modern mathematics, which typically require hours or days for expert mathematicians to solve. They are also designed to be “guessproof” (less than a 1% chance of guessing correctly without the mathematical work).

Pre-o3, the state of the art performance was 2%.

![Distribution of responses for Mathematics: FrontierMath](https://theaidigest.org/ai2025-analysis/plots/distributions/frontiermath.svg)

Note that [OpenAI commissioned Epoch AI to create this benchmark](https://epoch.ai/blog/openai-and-frontiermath), and has access to the problem statements and solutions, aside from a 50 question holdout set.

Pre/Post o3 Comparison

The median pre-o3 forecast for end of 2025 capabilities had already been exceeded by the end of 2024 – o3 scored 24%! We therefore see much higher forecasts after o3's strong reported performance on Frontier Math.

![Pre/Post o3 comparison for Mathematics: FrontierMath](https://theaidigest.org/ai2025-analysis/plots/o3-comparisons/frontiermath.svg)

### **6) Will pre-mitigation preparedness scores exceed those reported for o1-preview by December 31st 2025?**

The above questions were on benchmarks measuring capabilities. We also included some other indicators of progress in the survey. First up is the OpenAI preparedness scorecard.

OpenAI measured four categories of risk in [their original preparedness scorecard levels](https://cdn.openai.com/openai-preparedness-framework-beta.pdf). o1-preview [scored](https://openai.com/index/openai-o1-system-card/) **Low** on Model Autonomy and CBRN (Chemical, Biological, Radiological, and Nuclear), and **Medium** on Cybersecurity and Persuasion (scores range from Low to Critical). Note that these are **pre-mitigation** scores – if pre-mitigation scores are High or Critical, released models should have mitigations attempting to reduce the risk.

We asked respondents whether each of these risk scores would be exceeded by the end of 2025.

OpenAI released an [updated version of their Preparedness framework](https://openai.com/index/updating-our-preparedness-framework/) in April 2025 that replaces the original version. There are now three tracked categories: Biological and Chemical capabilities, Cybersecurity capabilities, and AI Self-improvement capabilities. We therefore won't be able to resolve all parts of this question.

However, before OpenAI switched to the new framework, Deep Research had hit the Medium Cybersecurity threshold, and o3-mini had hit the Medium Model Autonomy threshold, so these questions already resolve YES.

Meanwhile, the CBRN and Persuasion questions will resolve AMBIGUOUS: CBRN is tracked differently in the new framework, and Persuasion is not tracked at all:

> *Persuasion risks will be handled outside the Preparedness Framework, including via our Model Spec, restricting the use of our tools for political campaigning or lobbying, and our ongoing investigations into misuse of our products (including detecting and disrupting influence operations⁠).*

**CBRN**

### **6A) Will an AI system be reported by OpenAI as of December 31st 2025 as having a pre-mitigation score of High or higher on CBRN?**

CBRN is focused on risks related to model-assisted creation of chemical, biological, radiological, and/or nuclear threats.

![Distribution of responses for CBRN](https://theaidigest.org/ai2025-analysis/plots/distributions/openai-preparedness:-cbrn.svg)

CBRN is now tracked differently in OpenAI's [updated Preparedness Framework](https://openai.com/index/updating-our-preparedness-framework/), so we won't be able to resolve this question.

On the new framework, o3 and o4-mini [don't meet the High threshold](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf#page=12) for their updated Biological and Chemical capabilities category:

> *OpenAI o3 and o4-mini do not reach the High threshold in any of our three Tracked Categories: Biological and Chemical Capability, Cybersecurity, and AI Self-improvement*

> *Our evaluations found that OpenAI o3 and o4-mini can help experts with the operational planning of reproducing a known biological threat. Because such experts already have significant domain expertise, this risk is limited, but the capability may provide a leading indicator of future developments.*

However, before the release of o3 and o4-mini, [in the Deep Research system card](https://cdn.openai.com/deep-research-system-card.pdf#page=17), OpenAI reported that:

> *Several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold. We expect current trends of rapidly increasing capability to continue, and for models to cross this threshold in the near future.*

**Cybersecurity**

### **6B) Will an AI system be reported by OpenAI as of December 31st 2025 as having a pre-mitigation score of Medium or higher on Cybersecurity?**

Cybersecurity is focused on risks related to use of the model for cyber-exploitation to disrupt confidentiality, integrity, and/or availability of computer systems.

Resolved: YES

![Distribution of responses for Cybersecurity](https://theaidigest.org/ai2025-analysis/plots/distributions/openai-preparedness:-cybersecurity.svg)

[Deep Research hit the Medium Cybersecurity threshold](https://openai.com/index/deep-research-system-card/), so this question has already resolved YES.

**Model Autonomy**

### **6C) Will an AI system be reported by OpenAI as of December 31st 2025 as having a pre-mitigation score of Medium or higher on Model Autonomy?**

Model autonomy enables actors to run scaled misuse that can adapt to environmental changes and evade attempts to mitigate or shut down operations.

Resolved: YES

![Distribution of responses for Model Autonomy](https://theaidigest.org/ai2025-analysis/plots/distributions/openai-preparedness:-model-autonomy.svg)

[o3-mini hit the Medium Model Autonomy threshold](https://openai.com/index/o3-mini-system-card/), so this question has already resolved YES.

**Persuasion**

### **6D) Will an AI system be reported by OpenAI as of December 31st 2025 as having a pre-mitigation score of High or higher on Persuasion?**

Persuasion is focused on risks related to convincing people to change their beliefs (or act on) both static and interactive model-generated content.

![Distribution of responses for Persuasion](https://theaidigest.org/ai2025-analysis/plots/distributions/openai-preparedness:-persuasion.svg)

As we mentioned above, Persuasion risks aren't in the new Preparedness Framework, so we won't be able to resolve this question. OpenAI [reports](https://openai.com/index/updating-our-preparedness-framework/):

> *Persuasion risks will be handled outside the Preparedness Framework, including via our Model Spec, restricting the use of our tools for political campaigning or lobbying, and our ongoing investigations into misuse of our products (including detecting and disrupting influence operations⁠).*

**Sum of OpenAI, Anthropic, xAI revenues**

### **7) What will be the sum of OpenAI, Anthropic, and xAI's publicly reported annualized revenues by December 31st 2025, in USD billions?**

**Revenue numbers as of the end of 2024:**

OpenAI’s annualized revenue (monthly revenue * 12) was reported as [$3.6B](https://www.nytimes.com/2024/09/27/technology/openai-chatgpt-investors-funding.html) in August 2024, [$3.4B](https://www.theinformation.com/articles/openais-annualized-revenue-doubles-to-3-4-billion-since-late-2023) in June 2024, [$1.6B](https://www.reuters.com/technology/openai-annualized-revenue-tops-16-billion-information-2023-12-30/) in December 2023, and [$1.3B](https://www.reuters.com/technology/openai-annualized-revenue-tops-16-billion-information-2023-12-30/) in October 2023.

Anthropic's annualized revenue was reportedly projected to reach [$1B](https://www.theinformation.com/briefings/anthropic-reportedly-projected-to-reach-1-billion-in-annualized-revenue-this-year) by the end of 2024, and was reportedly [$100M](https://www.theinformation.com/articles/anthropic-projects-at-least-850-million-in-annualized-revenue-rate-next-year) in September of 2023.

xAI's annualized revenue was reportedly [$100M](https://www.wsj.com/tech/ai/elon-musks-startup-xai-valued-at-50-billion-in-new-funding-round-7e3669dc) as of November 2024.

Note that in September 2024,OpenAI made a recent projection that they would generate [$11.6B in annual 2025 revenue](https://www.nytimes.com/2024/09/27/technology/openai-chatgpt-investors-funding.html), which implies an annualized revenue greater than $11.6B by the end of 2025.

Since creating this question, xAI has acquired X, which will increase its revenue unrelated to its AI products. When resolving, we'll therefore estimate xAI's AI-related revenue.

![Distribution of responses for Sum of OpenAI, Anthropic, xAI revenues](https://theaidigest.org/ai2025-analysis/plots/distributions/ai-lab-annualized-revenues.svg)

**Public attention**

### **8) What percentage of Americans will identify computers/technology advancement as the US's most important problem by December 31st 2025?**

Gallup releases the results of [monthly polling](https://news.gallup.com/poll/1675/most-important-problem.aspx) on the question “**What do you think is the most important problem facing the country today?**”

Note that multiple mentions are considered, so people's responses can count for more than one category.

In recent months, answers classified as “advancement of computers/technology” have started to surface **around or below 0.5%**.

![Distribution of responses for Public attention](https://theaidigest.org/ai2025-analysis/plots/distributions/public-attention.svg)

The median forecast expects concerns to more than quadruple from under 0.5% to 2% by the end of 2025.

For reference, as of April 2025 here are some other topics that are a top concern for 2% of [Gallup respondents](https://news.gallup.com/poll/1675/most-important-problem.aspx): Race relations/Racism, Wars/War (nonspecific)/Fear of war, Education, Crime/Violence, Foreign policy/Foreign aid/Focus overseas.

And for comparison, here are some topics at the 1% level: Abortion, Terrorism, Wage Issues, Taxes, Lack of respect for each other.

### **Survey Demographics**

We promoted the survey through our Twitter, mailing list, and through our networks. Some notable AI researchers like [Ajeya Cotra](https://x.com/ajeya_cotra/status/1867813307073409333), [Ryan Greenblatt](https://x.com/RyanPGreenblatt/status/1870205360470135278), and [Peter Wildeford](https://x.com/peterwildeford/status/1873760001464189437) shared their forecasts – so some of their followers may have also taken it.

We also asked respondents some optional background questions to get a sense of their views on AI. This will also let us understand who forecasted our 2025 indicators best among different groups (short vs long timelines, AI experts vs non-experts, high vs low risk).

### **Timelines**

By what year do you think there's a 50% probability that there will be High Level Machine Intelligence which can accomplish every cognitive task better and more cheaply than human workers?

![Timelines](https://theaidigest.org/ai2025-analysis/plots/demographics/hlmi_years-binned.svg)

### **Risk**

How likely do you think it is that overall, the impact of High Level Machine Intelligence will be extremely negative for humanity in the long run?

![Risk](https://theaidigest.org/ai2025-analysis/plots/demographics/hlmi_risk.svg)

### **AI years of experience**

How many years of professional or academic experience do you have in the field of AI?

![AI years of experience](https://theaidigest.org/ai2025-analysis/plots/demographics/experience-binned.svg)

### **Correlations**

Finally, here's a plot showing how responses on all questions correlate with each other. Most indicators are positively correlated.

![Survey response correlations](https://theaidigest.org/ai2025-analysis/plots/correlations/correlation-matrix.svg)

At the end of the year, we'll resolve all the forecasts and write up the results (you can get notified when we do so [here](https://theaidigest.org)). We're looking forward to this glimpse into how unexpected the rate of progress has been.