Title: Scaling Laws For Scalable Oversight
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scalable Oversight

AI Alignment

AI safety
Abstract: Scalable oversight, the process by which weaker AI systems supervise stronger ones, has been proposed as a key strategy to control future superintelligent systems. However, it is still unclear how scalable oversight itself scales.
To address this gap, we propose a framework that quantifies the probability of successful oversight as a function of the capabilities of the overseer and the system being overseen.
Specifically, our framework models oversight as a game between capability-mismatched players; the players have oversight-specific Elo scores that are a piecewise-linear function of their general intelligence, with two plateaus corresponding to task incompetence and task saturation. We validate our framework with a modified version of the game Nim and then apply it to four oversight games: Mafia,  Debate, Backdoor Code and Wargames. For each game, we find scaling laws that approximate how domain performance depends on general AI system capability. We then build on our findings in a theoretical study of Nested Scalable Oversight (NSO), a process in which trusted models oversee untrusted stronger models, which then become the trusted models in the next step. We identify conditions under which NSO succeeds and derive numerically (and in some cases analytically) the optimal number of oversight levels to maximize the probability of oversight success. We also apply our theory to our four oversight games, where we find that NSO success rates at a general Elo gap of 400 are 13.5\% for Mafia, 51.7\% for Debate, 10.0\% for Backdoor Code, and 9.4\% for Wargames; these rates decline further when overseeing stronger systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tests scaling laws for scalable oversight: how effective are weak supervisors at supervising strong models on hard tasks, and how does this effectiveness scale with the overall strength of the models (both supervisor and policy model)? This is evaluated on several game domains using general Elo scores to represent overall model strength.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Applying the scaling laws lens to scalable oversight is a great idea and I think this is a fruitful direction to explore\n- The Elo modeling approach as a general model for capabilities and deriving scaling laws is simple and sensible, if somewhat reductive\n- The paper does come out with interesting insights and raises interesting concerns about how scalable oversight will scale\n\nWeaknesses:\n- The data here is mostly toy/synthetic and I think that makes it hard to extrapolate to real use cases for scalable oversight. The challenge with scalable oversight right now is really on the data side—how do we test it, and how do we implement it in practice with AI assisting humans, etc.? Using synthetic data avoids these concerns which can be good for getting simple/clear results but risks losing external validity. TBF, the debate experiments use more realistic data, but unless I'm mistaken, these are mostly redundant with the Khan et al and Kenton et al debate results, which also demonstrated scaling trends. The backdoor experiments also are a tad more realistic, if not entirely.\n- More concretely, I think where the rubber meets the road in scalable oversight is around models' actual skill profile in validation and reward modeling. Scalable oversight can work if we can come up with a way to decompose and reframe problems so that a reward model that is good enough at specific skills related to verification can very robustly evaluate a policy for performing a task. That's what will determine scaling laws in practice and it's hard to get at that in toy settings.\n- Minor but \"scalable oversight\" is usually more generally defined as involving methods for weak or resource-scarce supervisors to provide good feedback signal to models performing hard tasks. This includes eg developing methods for humans to supervise powerful AI. \"weaker AI systems supervising stronger ones\" is a sub-case of scalable oversight though.\n- I think the NSO theory is great but more needs to be said about how 'weak models supervising strong models' is reductive in collapsing a complex skill profile into a single measure of intelligence. It's reasonable when applied to a single task though.\n\nAll that said, I think applying the scaling laws lens here is fruitful and these experiments are a nice step.\n\nQUESTIONS:\n- Can you say something about how these results might vary when you have models that have stronger e.g. generative/discriminative gaps, and how we might aim to bring that to be the case?\n- You have some recommendations in the paper, e.g., about intercept and slope of the scaling trends that should be ideal for a scalable oversight method. Can you say more in the paper about how that might be done, and perhaps provide a consolidated set of recommendations for future work at the end of the paper?\n\nLIMITATIONS:\nyes (besides anything I'm bringing up in weaknesses)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSimilar to Sudhir et al (2025), Khan et al (2024) and other preceding works, the paper studies /Elo curves/ for LLMs on scalable oversight tasks --- i.e. it looks at how task performance scales with general model intelligence. The paper's approach makes the following key contributions:\n\n1) While previous works investigate the scaling behavior of scalable oversight protocols by comparing the scaling behavior of the agents' (\"Houdinis'\") performance when simulating \"aligned\" vs \"unaligned\" behavior (e.g. giving the right vs wrong answer on a question answering task), this paper looks at the Elo curves for /both the Houdinis and the Guards (the models supervising the Houdinis)/[*]. A good scalable oversight protocol should show Guards outperforming Houdinis as intelligence scales.\n2) The paper introduces and motivates a \"double-ReLU model\" for modelling the ELO curves.\n3) They use this perspective to study /Nested Scalable Oversight protocols/, where weaker models are used to align stronger models, which are then used to align further stronger models and so on. They model this as a ladder up the Elo curves from the original starting Guard to the target Houdini. At each step one may compute the win probability based on the Elo difference, which can then be multiplied to calculate the overall win probability of the nested scalable oversight protocol. Thus they compute the optimal number of recursive steps, and highlight their observations.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n1) The application to Nested Scalable Oversight protocols is particularly elegant. It is the first empirical study of /nested/ scalable oversight protocols I am aware of.\n2) [*] Another strength of this paper is that this approach directly considers the \"worst-case\" win probabilities where the Houdini is always motivated to fool the guard --- Sudhir et al (2025) had noted this to be a limitation of existing approaches.\n3) Overall the paper makes a valuable stride towards studying AI risk quantitatively, and should be accepted.\n\n*Weaknesses*\nSee \"Questions\". I am not yet sure if these are weaknesses of the paper or something I missed. In any case none of them are significant enough to warrant a rejection.\n\nQUESTIONS:\n1) The counter-intuitive result \"More generally intelligent Houdinis make oversight easier\" is interesting, but could be explained better: it's because \"more generally intelligent Houdinis holding domain intelligence fixed\" actually means a smaller slope for the Houdini curve, which means the intermediate Houdinis also have higher general intelligence, and smart Houdinis become smart Guards (and since the Elo curve remains fixed, this corresponds to greater Guard Domain intelligence). Right?\n2) I didn't understand why numerical experiments were needed --- isn't the problem of finding the optimal number of steps quite tractable in itself?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nScalable oversight is the process of using weaker AI systems to supervise stronger systems. It is one of the potential approaches to controlling strong AI systems. The paper attempts at producing scaling laws for scalable oversight. The paper begins by first defining oversight as a multi-player game between agents with mismatched capabilities. The experiments in the paper further validate this framework and the developed scaling laws across five games: Count-to-21, Mafia, Debate, Backdoor code and Wargames. The paper also presents a theoretical framework for Nested Scalable Oversight (NSO) to estimate the number of oversight steps.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written and the ideas are clearly presented.\n- Presents a general linear model between General and Domain Elo.\n- Evaluates the presented model in a number of multi-agent games such as Count-to-21, Mafia, Debate, Backdoor Code and Wargames.\n- Presents limitations about the experimental results.\n- To the best of my knowledge, I believe the work is original and has strong significance.\n\nWeaknesses:\n- Details about the evaluation pseudocode are missing.\n- Results could be presented more sharply.\n\nQUESTIONS:\n- The paper presents a simple linear modeling between general and domain elo. It would be useful to the community if the authors could discuss other potential choices for modeling such a relationship. \n- In the relationship between general and domain-specific Elo (Section 2.2), the modeling is independent of the number of guards and houdinis. What would the modeling look like if it accounted for varying numbers of guards and houdinis? Also would the modeling change if the agents were of varying strengths?\n- How many evaluation runs went into generating the results? Would the results significantly change if we re-evaluated the approach on the same set of games?\n- Why does Mafia and Wargames scale with more oversight steps (Figure 8), whereas there is not much change in Backdoor Code and in Debate?\n- Comment: The clarity of the paper could be improved significantly by presenting the pseudocode for evaluating the approach across the various multi-agent tasks, either in the main text or in Appendix.\n\nI am happy with the current set of results in the paper and the above questions are aimed at further improving the clarity of the paper.\n\nLIMITATIONS:\nYes, authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of deriving quantitative safety guarantees for Nested Scalable Oversight (NSO), a technique for overseeing stronger untrusted systems with weaker trusted ones, akin to quantitative safety guarantees required by standards in other fields of engineering.\n\nThe paper models NSO as an asymmetric game played multiple times by a sequence of Guard (trusted overseer) and Houdini (untrusted stronger agent) players. The fundamental analogy is one where the Guard is used to align the Houdini (e.g. by RLAIF followed by some verification of alignment). As a proxy for this game, the paper uses four games: Mafia, Debate, Backdoor Code, and Wargames (the latter is an original contribution). All four involve Houdini engaging in deceptive behavior and the Guard winning if and only if it catches the deception.\n\nFor these games, the paper models the relationship between general intelligence (Chatbot arena Elo is used as a proxy in experiments) and domain-specific intelligence for each game as a Double ReLU curve: below a certain intelligence threshold, the curve is flat, then increases linearly until a higher threshold, after which it is flat again. This is justified by analogy to games like chess and tic-tac-toe: a very young child roughly never wins, and above a certain threshold, the game is solved and further intelligence improvements are irrelevant. It is further validated by fitting the Double ReLU curve to data from the oversight games. Importantly, since the games are asymmetric, each game admits two distinct such curves, one for the Guard and one for the Houdini.\n\nThe paper proceeds to parameterize the NSO scenario with four parameters:\n- $m_G$ and $m_H$, the slopes of the linear portions of the two Double ReLU curves;\n- $\\Delta_\\text{general}$, the gap between our initial trusted Guard's general intelligence and the general intelligence of the agent we ultimately wish to align and deploy;\n- $\\Delta_\\text{domain}$, the gap in domain-specific intelligence between the same two agents, obtained by reference to the Double ReLU curves at the above general intelligence values.\n\nBecause intelligence is scored using Elo, the model admits a win rate for any given pair of domain-specific intelligence values. Overall success of NSO requires the Guard winning at each step of increasing intelligence. Assuming we choose a number of steps $n$ and evenly divide the total $\\Delta_\\text{general}$ by $n$, playing the game once at each step, this admits a closed form for the overall success probability. The paper evaluates this for $n \\in [1, \\ldots, 20]$ across many settings of the four parameters to find intuitively-meaningful patterns in their relationships with $n^*$, the optimal number of steps, and validate some of these findings theoretically.\n\nFinally, using the above analysis on the Double ReLU curves fit to the four deception games, the paper derives $n^*$ and the total success probability $p^*_\\text{win}$ against $\\Delta_\\text{general}$ for each game.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe theoretical analysis is strong and clear, and offers substantial progress toward real-world safety guarantees under certain assumptions.\n\nThe paper is very well-written, progressing with very clear pedagogy, moving from motivation to formalism to small examples to large examples in a way that makes for a compelling and easy read. In particular, Section 2.3 with the Counting-to-21 toy example kept me firmly grounded in understanding the work during my first linear pass through the paper.\n\nThe games are clearly described and motivated for an overseer catching deceptive behavior. The last two in particular are nice models for specific deceptive behavior we're concerned about misaligned models engaging in in the real world.\n\nThe paper is very upfront about its limitations, with an explicit \"**Limitations**\" paragraph at the end of many subsections.\n\n## Weaknesses\n### Assumptions\nSome assumptions seem like they deserve more justification or discussion.\n- Why Double ReLU instead of a sigmoid curve? The latter is a more standard answer to the existence of the three phases described at the top of Section 2.2.\n- Will we really be able to procure models with general intelligences that are evenly spaced in general intelligence over $\\Delta_\\text{general}$? Probably not. Does this have any important implications for applying the theory to a real ASI alignment scenario?\n\n### Catching deception vs. successful alignment\nThe major issue I encountered was a confusion about the way in which the four deception games are used as a proxy for actually aligning a strong agent. These games are all about detecting deception, and the Guard is considered to \"win\" if it catches the deception, which is always occurring. In the paper's analysis, we proceed through the steps of progressively smarter Guards by the assumption that if the Guard wins, the Houdini against which it won can now be used as a more intelligent trusted Guard. But in all these examples, the Houdini is *always* deceptive, and the Guard winning is still a scenario in which the Houdini behaved in an untrustworthy way. I was quite confused by this at first.\n\nEventually I came to the understanding I described in my summary, that these are being used as a proxy for an alignment game. Specifically, this pair of sentences ultimately got me there:\n- Section 2: \"we analyze two-player games between unmatched players with different intelligences as a **tractable proxy for real world weak-to-strong oversight**.\"\n- Section 4: \"trusted weak AIs are used to **align untrusted stronger AIs**, which are then used to align even stronger AIs recursively\"\n\nSo the real-world game is one where the Guard *aligns* the Houdini. Winning the game is successfully aligning the Houdini and creating and validating a smarter trustworthy agent. Losing the game is declaring the Houdini trustworthy when it is in fact not. **Actually catching deceptive behavior is neither winning nor losing, because in that case one would simply keep trying to align the Houdini**. Eventually I understood this (I think; correct me if I'm wrong), but ironically, the fact that these games are all toy scenarios modeling LLM deception lands for me as a significant weakness in the clarity of the paper. I see how the theory developed here, against these four games, could also apply to the real-world alignment game, but LLM deception is a component of that real-world game in which catching deception is *not* a win for the guard, and so e.g. Figure 6 showing the bouncing ball plot was very confusing for me because it appears at first glance to be resting on the statement \"once we catch an agent being deceptive, we are ready to promote it to trustworthy guard.\"\n\nQUESTIONS:\n1. Why Double ReLU instead of a sigmoid curve? The latter is a more standard answer to the existence of the three phases described at the top of Section 2.2.\n2. Will we really be able to procure models with general intelligences that are evenly spaced in general intelligence over $\\Delta_\\text{general}$? Probably not. Does this have any important implications for applying the theory to a real ASI alignment scenario?\n3. Please see above for my comments about how the games all being about catching deception led to confusion for me. I had initially marked \"4: excellent\" for the clarity of the paper on its pedagogical strength before that confusion hit me, and I'd happily move that score back up and increase my overall score if this point were very clearly addressed in the paper. I would most like to see a toy scenario added to the list that is structurally more like the game of *aligning* stronger agents. Barring that, adding one or many games here that are not just about catching deception would probably help. I should also mention that I'm not deeply familiar with the scalable oversight literature, and perhaps this is all very standard. If that's the case, I'd appreciate the authors' take on my confusion and a citation to somewhere in the literature that clarifies this, as well as some explanation added to the paper to preempt others being confused in the same way I was.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}