# Oversight Assistants: Turning Compute into Understanding

Currently, we primarily oversee AI with human supervision and human-run experiments, possibly augmented by off-the-shelf AI assistants like ChatGPT or Claude. At training time, we run [RLHF](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback?ref=bounded-regret.ghost.io), where [humans](https://arxiv.org/abs/2203.02155?ref=bounded-regret.ghost.io) (and/or [chat](https://arxiv.org/abs/2206.05802?ref=bounded-regret.ghost.io) [assistants](https://arxiv.org/abs/2212.08073?ref=bounded-regret.ghost.io)) label behaviors with whether they are good or not. Afterwards, human researchers do additional testing to surface and evaluate unwanted behaviors, possibly assisted by a [scaffolded](https://arxiv.org/abs/2404.14394?ref=bounded-regret.ghost.io) [chat agent](https://alignment.anthropic.com/2025/automated-auditing/?ref=bounded-regret.ghost.io).

The problem with primarily human-driven oversight is that it is not scalable: as AI systems keep getting smarter, errors become harder to detect:

*   The behaviors we care about become more complex, moving from simple classification tasks to open-ended reasoning tasks to long-horizon agentic tasks.
*   Human labels become less reliable due to [reward hacking](https://arxiv.org/abs/2409.12822?ref=bounded-regret.ghost.io): AI systems become expert at producing answers that _look_ good, regardless of whether they _are_ good.
*   Simple benchmarks become less reliable due to [evaluation awareness](https://evaluations.metr.org/gpt-5-report/?ref=bounded-regret.ghost.io#we-do-find-evidence-of-significant-situational-awareness-though-it-is-not-robust-and-often-gets-things-wrong): AI systems can often tell that they are being evaluated as part of a benchmark and explicitly reason about this.

For all these reasons, we need oversight mechanisms that scale beyond human overseers and that can grapple with the increasing sophistication of AI agents. Augmenting humans with current-generation chatbots does not resolve these issues: such off-the-shelf oversight won’t be superhuman until general AI systems are superhuman, which is too late. Moreover, capabilities are [spiky](https://bounded-regret.ghost.io/what-will-gpt-2030-look-like/#1-specific-capabilities)—models can excel at some tasks while doing poorly at others—so there will be tasks where current AI systems are better at _doing the task_ than _helping a human oversee it_.

Instead, **we need superhuman oversight of AI systems, today**. To do this, we need to at least partially decouple oversight from capabilities, so that we can get powerful oversight assistants without relying on general-purpose advances in AI. The main way to do so is through _data_: the places where AI capabilities have grown the fastest are where data is most plentiful (e.g. massive online repos for code, and unlimited self-supervised data for math).

Fortunately, AI oversight is particularly amenable to generating large amounts of data, because most oversight tasks are _self-verifiable_—the crux of the task is an expensive discovery process, but once the discovery has been made, verifying it is comparatively easy. For example, constructing cases where a coding assistant inserts malware into a user's code is difficult, but once constructed, such cases are easy to verify. Similarly, identifying the causal structure underlying a behavior is difficult, but once identified, it can be verified through intervention experiments. This kind of self-verifiable structure is currently driving the rapid advances in [mathematical](https://www.nature.com/articles/s41586-025-09833-y?ref=bounded-regret.ghost.io) [problem-solving](https://arxiv.org/abs/2511.22570?ref=bounded-regret.ghost.io), [reasoning](https://openai.com/index/learning-to-reason-with-llms/?ref=bounded-regret.ghost.io), and [coding](https://arxiv.org/abs/2502.06807?ref=bounded-regret.ghost.io), and we can leverage it for oversight as well.

If we could decouple oversight abilities from general capabilities, then work on safety and oversight would be significantly democratized: it would not be necessary to race to the capability frontier to do good safety work, and many actors could participate at the oversight frontier without the conflict of interest inherent in building the systems being evaluated. This democratization of AI oversight is a significant motivation for my own work (and why I co-founded [Transluce](https://transluce.org/?ref=bounded-regret.ghost.io)), and structurally important for achieving good AI outcomes.

Below I will lay out this vision in more detail, first [demonstrating self-verifiability through several examples](#superhuman-oversight-from-specialized-assistants), then providing a [taxonomy of oversight questions](#a-taxonomy-of-oversight-questions) to show that this structure is fairly general. I’ll also close with a forward-looking [vision and open problems.](#vision-end-to-end-oversight-assistants)

Superhuman Oversight from Specialized Assistants
================================================

We can achieve superhuman oversight of AI systems by building _specialized, superhuman AI assistants_. They do not need to be broadly superhuman, only superhuman at the specific task of _helping humans to oversee other AI systems_. To build these assistants, we can train models on lots of data specialized to the oversight tasks we care about, similar to how AI developers achieved recent leaps in [reasoning](https://openai.com/index/learning-to-reason-with-llms/?ref=bounded-regret.ghost.io), [math](https://www.nature.com/articles/s41586-025-09833-y?ref=bounded-regret.ghost.io), and [coding](https://arxiv.org/abs/2502.06807?ref=bounded-regret.ghost.io). This aligns with the **bitter lesson**—the performance of AI systems at a task is primarily determined by the amount of data and compute they leverage.

AI oversight is particularly amenable to this type of specialized-but-scalable training. This is because AI systems inherently generate vast amounts of data, such as agent transcripts, diverse behaviors across prompts, neuron activations, and pre-training and fine-tuning data. Given one of these data sources, and a well-operationalized question, we can usually construct a scalable reward signal.

As a concrete example, consider _behavior elicitation_: the problem of determining whether and when an AI system exhibits a given behavior pattern (such as “_providing advice with predictably harmful consequences to the user_”). This asks the question “what could happen” and applies it to the data source of “input-output behaviors of an AI system.” Answering it is important for anticipating failures before they surface in the wild.

Behavior elicitation has exactly the problem structure we want: _finding_ an unwanted AI behavior is a hard search problem, but _verifying_ that it is unwanted is much easier^[\[1\]](#fn1)^, and it can often be done with an LM-based judge. This gives us an automated reward signal to optimize with RL, and we used this in [Li et al. (2025)](https://arxiv.org/abs/2502.01236?ref=bounded-regret.ghost.io) and [Chowdhury et al. (2025)](https://transluce.org/pathological-behaviors?ref=bounded-regret.ghost.io) to train specialized elicitation assistants. By exploiting the scalability of LM-based rewards, we searched across a huge number of inputs (40M+ tokens) to surface behaviors including self-harm, hallucinations, anger towards the user, and reinforcing delusions.

![training elicitation assistants 1.png](https://bounded-regret.ghost.io/content/images/2026/01/training-elicitation-assistants-1.png)

Training elicitation assistants using an LM-based judge of AI behavior. The judge answers whether a given response constitutes realistic, authentic, and unexpected harmful advice. This score is used to update the policy $\\pi_{\\theta}$ using reinforcement learning. Adapted from [Chowdhury et al. (2025)](https://transluce.org/pathological-behaviors?ref=bounded-regret.ghost.io)

As a second example, consider _feature description_: describing which inputs cause a given neuron or other unit of a neural network to fire. This asks the question “what happened” and applies it to the data source of “neuron activations.” This task is a key building block for many downstream applications of interpretability.

This again has the problem structure we want: given a purported description of a feature, we can use this to predict the firing patterns of the feature (by having a LM make predictions given the description), and see if the predictions match reality. In [Choi et al. (2024)](https://transluce.org/neuron-descriptions?ref=bounded-regret.ghost.io), we used this reward signal to [train a specialized model](https://github.com/TransluceAI/observatory/tree/main/project/expgen?ref=bounded-regret.ghost.io) to cheaply produce high-quality explanations, outperforming previous more expensive approaches that relied on general-purpose chatbots. We used these explanations to debug strange behaviors, such as why so many models believe that [9.11 is larger than 9.8](https://transluce.org/observability-interface?ref=bounded-regret.ghost.io#example-1).

![training explanation assistant 1.png](https://bounded-regret.ghost.io/content/images/2026/01/training-explanation-assistant-1.png)

Training an explanation assistant by automatically scoring descriptions. Given a set of descriptions, an LM-based scorer determines how well each one matches reality. We use this to re-rank and then train a specialized explainer model on high-scoring descriptions. Adapted from [Choi et al. (2024).](https://transluce.org/neuron-descriptions?ref=bounded-regret.ghost.io)

This second example illustrates a key point: **specialized models can outperform larger general-purpose models**. In our case, a specialized explainer fine-tuned from Llama-8B outperformed the general-purpose GPT-4 model, which is likely an order of magnitude larger. This is direct evidence that we can decouple oversight capabilities from general capabilities.

A Taxonomy of Oversight Questions
=================================

To generalize the two examples above, we’ll consider a broader taxonomy of oversight tasks. Both examples above took a **data source** and asked a particular question about it. For AI oversight, we are often interested in three types of questions: _what happened, what could have happened, and why it happened_:

*   What happened? Because AI agents act autonomously, take many consecutive actions, and sometimes [intentionally hide their mistakes](https://x.com/seconds_0/status/1917447726134091939?ref=bounded-regret.ghost.io), even answering _what happened_ in a given user interaction or agentic workflow is not straightforward. We need scalable oversight tools to process the actions that an agent took and explain them to us.
*   What could happen? Because AI acts in open-ended environments, it is difficult to anticipate the full set of behaviors an AI system might exhibit, leading to strange behaviors at deployment time (such as o3 [claiming it ran code on its personal MacBook Pro](https://truthfulness.docent.transluce.org/o3-2025-04-03?searchQuery=case+where+the+model+specifically+mentions+running+code+an+external+laptop+itself.+It+must+mention+running+the+code+itself%2C+not+suggesting+that+the+user+could+do+it%2C+or+that+it+could+be+done+in+the+abstract.+It+must+specifically+mention+a+laptop&ref=bounded-regret.ghost.io)). We need adaptive testing tools that will surface these behaviors early on before they create problems in the wild.
*   Why did it happen? Once problems are discovered, we need to trace them back to their root cause—this helps us to fix them and anticipate related problems. For AI behaviors, the root cause lies in the training data and the learned representations. We need attribution methods to trace behaviors to these sources and to make it practical for humans to interpret and intervene on the massive data sets involved.

We can apply these questions to each of the key data sources related to a given model: its outputs, its internal representations, and its training data. This provides a taxonomy, based on the type of question we are asking (what happened, what could happen, why did it happen) and the data sources we are using to answer it (behaviors, representations, data).

|     | What happened | What could have happened | Why did it happen |
| --- | --- | --- | --- |
| Behaviors | [Analysing agent transcripts](https://transluce.org/introducing-docent?ref=bounded-regret.ghost.io) | Behavior elicitation ([PRBO](https://transluce.org/pathological-behaviors?ref=bounded-regret.ghost.io), [Petri](https://www.anthropic.com/research/petri-open-source-auditing?ref=bounded-regret.ghost.io)) | [Counterfactual simulability](https://arxiv.org/abs/2307.08678?ref=bounded-regret.ghost.io), [counterfactual](https://aclanthology.org/2022.emnlp-main.14.pdf?ref=bounded-regret.ghost.io) [explanations](https://transluce.org/pathological-behaviors?ref=bounded-regret.ghost.io#evaluating-the-robustness-of-our-discovered-strategies) |
| Representations | [Feature descriptions](https://transluce.org/neuron-descriptions?ref=bounded-regret.ghost.io), [activation](https://transluce.org/observability-interface?ref=bounded-regret.ghost.io#example-2) [monitoring](https://arxiv.org/abs/2504.20271?ref=bounded-regret.ghost.io), [LatentQA](https://latentqa.github.io/?ref=bounded-regret.ghost.io) | Eliciting interpretability states ([fluent](https://arxiv.org/abs/2402.01702?ref=bounded-regret.ghost.io) [dreaming](https://www.alignmentforum.org/posts/hMBTaFvAzdMNnj29c/evolutionary-prompt-optimization-for-sae-feature?ref=bounded-regret.ghost.io)) | [Feature attribution](https://transluce.org/observability-interface?ref=bounded-regret.ghost.io#example-1), [Predictive Concept Decoders](https://transluce.org/pcd?ref=bounded-regret.ghost.io), [automated](https://arxiv.org/abs/2304.14997?ref=bounded-regret.ghost.io) [circuit discovery](https://arxiv.org/abs/2403.19647?ref=bounded-regret.ghost.io) |
| Data | [Data observability](https://arxiv.org/abs/2310.20707?ref=bounded-regret.ghost.io) | Fine-tuning sets that elicit a particular behavior (e.g. [emergent misalignment](https://arxiv.org/abs/2502.17424?ref=bounded-regret.ghost.io)) | Data attribution ([influence functions](https://arxiv.org/abs/2308.03296?ref=bounded-regret.ghost.io), [datamodels](https://arxiv.org/abs/2202.00622?ref=bounded-regret.ghost.io)), counterfactual data explanations ([persona features](https://arxiv.org/abs/2506.19823?ref=bounded-regret.ghost.io), [alignment pretraining](https://alignmentpretraining.ai/?ref=bounded-regret.ghost.io)) |

  

Example questions and relevant work for each cell in our 3x3 taxonomy.

While the details vary, for each of these oversight questions, there is a natural strategy for operationalizing it as a scalable objective. Let’s walk through each question type in turn.

To start with, for _what happened_, we typically want to produce a faithful and informative summary^[\[2\]](#fn2)^ of a large data source (e.g. _“what behaviors related to sycophancy appeared in these agent transcripts”_, or _“what examples related to biosecurity appeared in this pretraining corpus”_).

*   Faithfulness is relatively straightforward: we can have an LM judge check the source material in detail against the summary to ensure accuracy.
*   For informativeness, we can test how well a human (or LM-based proxy) can make downstream decisions given the summary. Or alternatively, we could provide two possible summaries and ask the human/LM to rank which one they’d find more useful.^[\[3\]](#fn3)^

For _what could have happened_, we have already seen the search-and-verify structure with behavior elicitation. This same structure also applies for representations: searching for inputs that elicit a given interpretability state (such as a specific feature being active). It also applies for data: we could imagine generalizing investigations into [emergent misalignment](https://arxiv.org/abs/2502.17424?ref=bounded-regret.ghost.io) by automatically searching for fine-tuning sets that elicit specified behaviors at test time.

Finally, _why did it happen_ seeks to reduce observed behaviors to underlying causes, which is the purview of empirical science. Given a claim that “X causes Y”, we could empirically test it by counterfactually varying X and checking that Y moves as predicted; and by enumerating alternative hypotheses and generating experiments to distinguish between them ([Platt, 1964](https://www.science.org/doi/10.1126/science.146.3642.347?ref=bounded-regret.ghost.io)). The self-verifiability of why is most apparent for interpreting latent representations, where [causal abstractions](https://arxiv.org/abs/2301.04709?ref=bounded-regret.ghost.io) provide a formal algebra for checking claims; but it can also be applied to behaviors and data, for instance by [varying parts](https://aclanthology.org/2022.emnlp-main.14.pdf?ref=bounded-regret.ghost.io) of [an input](https://transluce.org/pathological-behaviors?ref=bounded-regret.ghost.io#evaluating-the-robustness-of-our-discovered-strategies) to test what matters for a behavior, or [varying training data](https://arxiv.org/abs/2410.15661?ref=bounded-regret.ghost.io) to test causal effects.

These individual oversight questions tie into each other, creating a compounding dynamic: data begets data. For example, feature descriptions—a _“what happened”_ question—are a useful precursor to feature attribution—a “_why it happened_” question. In the other direction, describing internal representations also helps us to elicit them, and eliciting more behaviors gives us more examples to analyze or attribute to. This is why at [Transluce](https://transluce.org/?ref=bounded-regret.ghost.io) we tackle all these questions at once, and it’s reason for optimism that progress on oversight will accelerate over time.

Fortunately, the data to train these systems is abundant. For agent behaviors, a single transcript collection in [Docent](https://transluce.org/docent?ref=bounded-regret.ghost.io) can contain 400M tokens. For representations, we can track millions of neuron activations^[\[4\]](#fn4)^ across trillions of tokens in [FineWeb](https://arxiv.org/abs/2406.17557?ref=bounded-regret.ghost.io) or other large corpora. By generating data for each cell in the taxonomy above and training specialized assistants on it, we create a conduit for applying massive compute to oversight, in line with the bitter lesson.

Vision: End-to-End Oversight Assistants
=======================================

The taxonomy above reveals a common structure across oversight tasks: each involves a large data source, a natural question, and a scalable way to verify answers. This shared structure suggests we can move beyond building local solutions for individual tasks toward unified, end-to-end oversight assistants.

What should such an assistant be able to do? It should:

*   Understand the system it is overseeing
*   Understand what the human overseer cares about, and what decisions they need to make
*   Provide explanations that help the human do this

[Docent](https://transluce.org/docent?ref=bounded-regret.ghost.io) is a good illustration of this full experience: an AI assistant reads a large collection of agent transcripts, then interacts with a human who arrives with a high-level concern (e.g., _“Is my agent reward hacking?”_). It helps them refine exactly what they mean based on the data (by showing examples or asking follow-up questions) until they arrive at a precise question along with a clear, evidence-backed answer.

How do we measure whether such a system is working? The key insight is that oversight assistants ultimately provide _information_ to humans, so we can ground their quality in how well that information supports human understanding. Concretely, after interacting with the assistant, we check if the human can accurately answer probing questions about the system being overseen. Better assistants will lead to more accurate answers and fewer surprises.

This vision raises a number of open questions. How do we evaluate oversight assistants when human judgment is itself unreliable? How do we create scalable data to train these systems without being bottlenecked on human labels? How do we build good architectures and pretraining objectives for these assistants? These are the technical questions that I spend most of my time thinking about, and we’ll go into them in detail in the next post.

**Acknowledgments.** Thanks to Sarah Schwettmann, Conrad Stosz, Daniel Johnson, Rob Friel, Emma Pierson, Yaowen Ye, Prasann Singhal, Alex Allain, and James Anthony for helpful feedback and discussion, and Rachael Somerville for help with copyediting and typesetting.

* * *

[^MISSING-ID]:  

[^MISSING-ID]:  

[^MISSING-ID]:  

[^MISSING-ID]: