# o3

*See* [*livestream*](https://www.youtube.com/watch?v=SKBG1sqdyIU)*,* [*site*](https://openai.com/12-days/?day=12)*,* [*OpenAI thread*](https://x.com/OpenAI/status/1870186518230511844)*,* [*Nat McAleese thread*](https://x.com/__nmca__/status/1870170098989674833).

OpenAI announced (but isn't yet releasing) o3 and o3-mini ([skipping](https://www.theinformation.com/briefings/openai-preps-o3-reasoning-model) o2 because of telecom company O2's trademark). ["We plan to deploy these models early next year."](https://x.com/OpenAI/status/1870186524182229014) ["o3 is powered by further scaling up RL beyond o1"](https://x.com/__nmca__/status/1870170101091008860); I don't know whether it's a new base model.

o3 gets 25% on [FrontierMath](https://epoch.ai/frontiermath), smashing the previous SoTA. (These are really hard math problems.[^uc50blxq7pe]) Wow. (The dark blue bar, about 7%, is presumably one-attempt and most comparable to the old SoTA; unfortunately OpenAI didn't say what the light blue bar is, but I think it doesn't really matter and the 25% is for real.[^bzt1de550x])

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ea4c2d487a479252fe19e6d84db195074138120d31346b02.png)

o3 also is easily SoTA on SWE-bench Verified and Codeforces.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/320ef1c5038ad2c8b5d4992a37f53e431ee04e1cf172d1b0.png)

It's also easily SoTA on [ARC-AGI](https://arcprize.org/blog/oai-o3-pub-breakthrough), ~~after doing RL on the public ARC-AGI problems~~[^vykbmdkdadf] + when spending $4,000 per task on inference (!).[^yb42bs6a75k] (And at less inference cost.)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/21b12842949a3cac3103248d9f995b87ddb9826092e35408.png)

ARC Prize [says](https://arcprize.org/blog/oai-o3-pub-breakthrough):

> At OpenAI's direction, we tested at two levels of compute with variable **sample sizes**: 6 (high-efficiency) and 1024 (low-efficiency, 172x compute).

OpenAI has a ["new alignment strategy."](https://openai.com/index/deliberative-alignment/) (Just about the "modern LLMs still comply with malicious prompts, overrefuse benign queries, and fall victim to jailbreak attacks" problem.) It looks like RLAIF/[Constitutional AI](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback). See [Lawrence Chan's thread](https://x.com/justanotherlaw/status/1870250651214823689).[^9qg1g6smyn9]

OpenAI [says](https://openai.com/index/early-access-for-safety-testing/) "We're offering safety and security researchers early access to our next frontier models"; yay.

o3-mini will be able to use a low, medium, or high amount of inference compute, depending on the task and the user's preferences. o3-mini (medium) outperforms o1 (at least on Codeforces and the 2024 AIME) with less inference cost.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e8e0667ae488d91d2f33b59c386ff36b678e7c8a92a72122.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f379c5744ba459369a73803df14047f9271b6dbfeef5d5cd.png)

GPQA Diamond:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a008f5686c6d0cf39e8d85c65b26fc42ba7017a40be0fb7e.png)

[^uc50blxq7pe]: Update: most of them are not as hard as I thought:There are 3 tiers of difficulty within FrontierMath: 25% T1 = IMO/undergrad style problems, 50% T2 = grad/qualifying exam style [problems], 25% T3 = early researcher problems. 

[^bzt1de550x]: My guess is it's consensus@128 or something (i.e. write 128 answers and submit the most common one). Even if it's pass@n (i.e. submit n tries) rather than consensus@n, that's likely reasonable because I heard FrontierMath is designed to have easier-to-verify numerical-ish answers.Update: it's not pass@n. 

[^vykbmdkdadf]: Correction: no RL! See comment.Correction to correction: nevermind, I'm confused. 

[^yb42bs6a75k]: It's not clear how they can leverage so much inference compute; they must be doing more than consensus@n. See Vladimir_Nesov's comment. 

[^9qg1g6smyn9]: Update: see also disagreement from one of the authors.