Combining Cost Constrained Runtime Monitors for AI Safety
Authors: Tim Tian Hua
James Baskerville
Henri Lemoine
Mia Hopman
Aryan Bhatt
Tyler Tracy
Venue: NeurIPS 2025 poster ()
Keywords: AI Safety
Agents
Constrained Maximization
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Optimally constructing monitoring protocols with multiple monitors of varying costs and performances.

Abstract: Monitoring AIs at runtime can help us detect and stop harmful actions. In this paper, we study how to efficiently combine multiple runtime monitors into a single monitoring protocol. The protocol's objective is to maximize the probability of applying a safety intervention on misaligned outputs (i.e., maximize recall). Since running monitors and applying safety interventions are costly, the protocol also needs to adhere to an average-case budget constraint. Taking the monitors' performance and cost as given, we develop an algorithm to find the best protocol. The algorithm exhaustively searches over when and which monitors to call, and allocates safety interventions based on the Neyman-Pearson lemma. By focusing on likelihood ratios and strategically trading off spending on monitors against spending on interventions, we more than double our recall rate compared to a naive baseline in a code review setting. We also show that combining two monitors can Pareto dominate using either monitor alone. Our framework provides a principled methodology for combining existing monitors to detect undesirable behavior in cost-sensitive settings.

Combining Cost-Constrained Runtime Monitors for
AI Safety
Tim Tian Hua" MARS cimhua0@gmail.com
James Baskerville MARS
Henri Lemoine MARS
Mia Hopman' MARS
Aryan Bhatt
Tyler Tracy MARS & Redwood Research tyllertracyrddusscomm com
Redwood Research aryanSrdwrs.. com
Abstract
Monitoring AIs at runtime can help us detect and stop harmful actions. this paper, study how to efficiently combine multiple runtime monitors into single monitoring protocol. The protocol's objective is to maximize the probability of applying safety intervention misaligned outputs (i.c., maximize recall). Since running monitors and applying safety interventions are costly, the protocol also needs to adhere to average-case budget constraint. Taking the monitors' performance and cost as given, we develop an algorithm to find the best protocol. when and which monitors call, and
The algorithm exhaustively searches allocates safety interventions based on the likelihood ratios and strategically trading off spending on monitors against spending on interventions, we more than double our recall rate compared to naive baseline code review setting. We also show that combining two monitors can Pareto dominate using either monitor alone. Our framework provides a principled methodology for combining existing monitors to detect undesirable behavior cost-sensitive settings.
Pearson lemma. By focusing
Introduction
As Al systems with greater autonomy and capability permeate world, their potential for harm scales lockstep. One straightforward safety intervention is to monitor Al outputs and block harmful actions. Indeed, monitors form cornerstone of proposed safeguards for advanced AI systems (Korbak et al.] 2025b Grosse 2024 Shah et al.) 2025) and are central to present-day content moderation (Inan et 2023 However, monitoring is imperfect and costly. Given budget, how can we spend it in the most cost-efficient manner to stop as many harmful actions as possible?
Developers often have access multiple monitors and wish to combine them into single monitoring protocol. For example, OpenAl uses two monitors to mitigate the misuse of ChatGPT Agent for creating biological and chemical weapon:
detect if the topie related to biology, and second
one to detect if the response violates their policies (OpenAl 2025 pg. 35). [Cunningham et al. (2025) from Anthropic propose "multi-stage classifiers" by first using cheap modellintermals-baan monitors then calling a larger monitor if the cheap monitor found the output to suspicious. There also academic studies that combine probe monitors and black box monitors for various safety- relevant tasks (McKenzie et al. 2025 Parrack et al. 2025). However, these existing monitoring
"Corresponding author: cimhuaO@gmall.omm i 'qqual contribution, author order randomized.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

systems rely on heuristics to determine when to call second monitor and how to weight the two monitors' outputs (e.g., Mc...iie et al. (2025) calls an additional monitor when the first monitor's score is close to its median and average the two monitors' scores when they're both called).
We solve for the optimal way construct monitoring protocols with multiple monitors of varying cost and performance profiles. An example monitoring protocol with two monitors could be described as "always call monitor one; call monitor intervention if monitor two's score is above 7, otherwise let the oulput through." We model AI outputs as either aligned misaligned, and our objective is to maximize the probability of performing safety intervention when the output is misaligned (i.e., maximizing recall). Both running monitors and false positive applications of the safety intervention on aligned outputs-aa such as blocking benign outputs from reaching the user user-incurs costs. Thus. we also aim to adhere to an average-case budget constraint.
if monitor one's score above 3, and apply the safety
Our key insight is that we can decompose the policy optimization into two parts: deciding when and which monitors call, and determining which outputs to flag for safety interventions. Given decision procedure on when to call monitors, we can apply the Neyman-Pearse safety interventions monitor scores with the highest misaligned/aligned likelihood ratios. We study settings with a limited set of available monitors-whic is realistic in many practical scenarios- and exhaustively search over defined class of monitor invocation strategies, picking the one with the highest recall rate for our final monitoring protocol.
lemma and target
We demonstrate approach in AI control setting (Greenblatt al. 2024). We monitors detect backdoored solutions to BigCode problems (Zhuo et al. (025). Our experiments reveal that: (1) wwwthhoood-rttt-bbassssee policies outperform naive baselines; (2) stochastic monitor calls can significantly outperform always-on monitoring by strategically trading off spending on monitors against spending on the safety intervention; and (3) combining monitors (Llama 3.1 70B and 8B) yields Pareto improvements over any single-monitor policy, as shown in Figure
Safety V. Cost pareto frontier one/two monitors Audit costs 20 times the Llama 70B monitor, which is normalized to cost $1
100%
Safety on I.I.d. test set
80%
60%
20%
83.00
$2.50
$2.00
$1.50 Total budget
$1.00
$0.50
Liama 705 monitor only
Liama 8B monitor only
Using both monitors
Lama BB costs $0.21, which is based on the real world relative cost to 706. SE from 39 repeased train tost splits.
Figure 1: Combining both monitors can achieve higher safety the same cost. See Figure 4b for an illustration of how we use two monitors and Appendix Section A for standard errors on relative performance as opposed to absolute performance.
To summarize, our contributions are:
A formal mathematical framework for the monitoring problem where we maximize recall subject to cost constraints on monitor calls and safety interventions.
A two-stage policy optimization procedure that combines exhaustive search over monitor calls with likelihood ratio-based intervention allocation.
Empirical demonstrations of significant performance gains compared to baselines.
The remainder of this paper structured as follows. Section overviews related work. Section 3 we introduce our mathematical framework and the procedure for finding optimal monitoring protocol. We then present our empirical setting in Section 4 and apply our framework first single-monitor case (Section and then two-monitor case (Section demonstrating that

---

combining monitors can yield Pareto improvements. Finally, Section 7 discusses our findings, limitations, and directions for future work. We also provide a practitioner's guide that walks through how to combine brsde-based monitor and a black box monitor using our framework on the Alignment Forum linked here, and our code can be found this Github link: github.com/tim-hua s//combbiinggmooniie monitorsspubllle
2 Background and related work
Monitoring for AI safety Runtime AI monitoring is an essential component of safety protocols,
such as those preventing jailbreaks (Xie al. 2024, [hhn et al., (024) and bio and chemical weapons related misuse (OpenAl] 2025| (Sharma et al. (2025). Beyond examining outputs, researchers have also explored chain of thought monitoring and activation probing (Chen et al. 2025 Goldowsky-
et al.] (025). For example, chain of thought monitors can uncover reward hacking during RL training (Baker et al. 2025). A particularly notable contribution is Guerin et al. (002), which developed a mathematical framework for safety monitoring focused on three metries: safety gain, availability cost, and residual hazard- an approach distinct from our cost-constrained optimization framework.
Related statistical theory Our approach of maximizing recall under budget constraints is conceptu- ally similar to AUROC maximization (rather than minimizing a standard classification loss). Previous work on combining classifiers to obtain optimal ROC curves has focused on estimating likelihood ratios and applying the Neyman- Pearson lemma (Barreno et al. [007 Clémençon and Vayatis (2008), which we leverage as well. The key distinction between our setting and AUROC maximization is that costs are incurred not only for false positives (i.e., applying the safety intervention aligned outputs) but also for monitor invocations themselves. Other work in costly classification uses approximations
like RL or costly trees (Janisch et al.) 2020;; Lomax and Vadera (013)). However, we use exhaustive search, which is feasible and near-optimal for the practical case of systems with few monitors, avoiding the need for such approximations.
AI control AI control is a research agenda aimed at preventing powerful Als from causing harm, even if those systems are egregiously misaligned and are actively attempting subvert safety measures (Greenblatt et al. 2024) Korbak et al.) (205a)). Monitor is essentially in Al Control, as it allows us target disruptive safety interventions such as human audits that they are only applied to the most suspicious outputs. Recent papers have also explored the cost of using different language models monitors (Terekhov et al. 2025).
Mathematical framework
3.1
Setup
Safety interventions and objective Als could perform a variety of undesirable actions, such as outputting copyrighted material, giving instructions for chemical weapons, even engaging in active acts of sabotage as misaligned agents (Benton et al.) 20024). By monitoring AI outputs, we can apply safety interventions on suspicious model outputs. A safety intervention could take many forms, such as replacing the output with a standard refusal, editing the output using another model, or calling human auditor review the action. For brevity, we will refer to our safety intervention as an "audit"- common intervention in AI control settings throughout this paper. These interventions incur costs if performed on an aligned output: customers are dissatisfied with over-refusals, edited actions might not work as well as the unedited ones, and human auditors cost money.
Formally, we model the AI deployment as series of i.i.d. draws from an untrusted Al model. The outputs can either be aligned (B = 0) or misaligned (B = 1), where B stands for "bad" outputs. Our primary objective is maximize safety, defined the probability of auditing (A 1) misaligned outputs P(A = B = 1) (i.e., recall). Auditing an aligned output incurs a cost Ca.
assess whether an output warrants an audit, we have access to k distinct monitors indexed by i. Calling monitor yields a score ma, with uncalled monitors denoted as m. = Each monitor can only be called once and incurs a cost e; These costs could represent the compute required for the monitor, added latency, other factors. These monitors provide information
Monitor calling
3

---

about output alignment, and their distributions conditional on B can be estimated from some training data. The available information from all monitors is represented by the vector m (m, m2, .... my)- which includes both called and uncalled monitors.
Budget constraint Instead of having separate budgets for monitors and auditing, we'll focus on a case where there's 's single unified budget constraint Y. A unified budget constraint reflects the fact that developers can trade off spending on monitoring and auditing. Ca/ca captures how much auditing costs relative to the first monitor. These relative costs help inform how much of our final budget to allocate to monitoring versus auditing.
While we denote both the audit and monitor cost in terms of dollars in this paper, this framework applies whenever we can characterize the relative costs of monitoring and auditing and the trade-off between the two actions. For example, suppose our safety intervention is blocking a chat output from reaching the customer, and we have two monitors. Monitor one has Ims latency, and monitor two has 100ms latency. Furthermore, suppose that a percentage point increase in overrefusals decreases customer retention by 0.1 %6, while Ims increase in latency decreases customer retention by 0.001% 1..%. On the cost side, we would be indifferent between adding the second monitor to all outputs and auditing an additional 1% of all outputs: both decrease customer retention by 0.1%.
We assume that we will take many draws from our untrusted Al model, and thus we will use the expected or average-case cost as constraint. We evaluate budget compliance on aligned outputs (B = 0) because: (1) misaligned outputs are rare in practice, as even misaligned AI systems may "act aligned" most of the time (Greenblatt et al., 2024). and (2) significant portion of total operational cost stems from false posilives. This cost is monitoring costs plus the product of the audit false positive rate (FPR) and the audit cost
Y > [Costt] B 0]
P(1 = 1|B 0) coa + Σ P(m; + L = 0)
Audit FPR
Monitor cost
Monitoring POMDP We model the full monitoring protocol partially observable Markov decision process (POMDP). The hidden state is alignment B E {o, 1): observations are monitor scores (initially all L). Actions are: (1) call uncalled monitor (2) audit (A = 1). (3) terminate without auditing (A = 0). Calling monitor updates m, according to the distribution P(m/) B,, m. while the other two actions end the monitoring process. A possibly stochastic policy л E IT maps monitor histories to action distributions: x H + (AA. Our optimization objective is to maximize recall misaligned outputs subject our budget constraint.
To summarize:
Maximize safety objective:
max P(A 1, nEll
(I)
such that
Subject to the budget constraint: Y Y P(A= 1BB = 0,
ca + Σ P(m, # LIB = 0, π²)
c
3.2 Policy optimization theory and practice
We decompose the policy optimization into two key steps: (1) determining which monitors to call and when, and (2) deciding which outputs to audit based on observed monitor scores.
monitor calling or invocation strategy determines which parts of the monitor score space are reachable at terminating states. An example monitor invocation strategy is: always call monitor one first and call monitor two only if monitor one's score exceeds 0.2. In this case, we would
We discuss two alternative budget formulations in Appendix Section B case where we have fixed audit and monitor budgets, and case when the costs from auditing increase non-linearly with the Audit FPR.

---

never observe score combination like m = (0.1.0.2 but would instead see combinations like m (0.1, or m = (0.3, 0.2). A policy π r's monitor invocation strategy induces distribution over possible m, which we denote as Mr. From training data, we can estimate the distributions P(M+IB = 0) and P[M+ = 1). Given policy's monitor calling behavior, monitor costs, and total budget, we know how much budget remains for auditing and the audit false positive rate that exhausts our budget.
Neyman- Lemma We model the auditing decision a hypothesis testing problem between Hn: : B = 0 (aligned) and H1 : B = (misaligned). We define the likelihood ratio as
P(M₂ m = =)
{(m) = P(M₂ m B = =0 0)
Let D(m) represent the probability of auditing given monitor scores m. The Neyman-Pearson Pearson Lemma states that the most powerful test (i.e., the one with highest recall for any false positive rate)
is the likelihood ratio test:
1. {(m) > T
D(m)= = 7: {(m) = T,
0, {(m) < T,
where 7 is a threshold, and is the probability of auditing for scores exactly at the threshold. At the optimum, we choose T to exhaust our budget in Equation P
This lemma has an intuitive interpretation: the likelihood ratio is proportional to the Bayes factor (. M[B=0.M.=mi which tells how many true positives expect per false positive at given m. After calling monitors, we have some budget left for false positives, and we could maximize recall by auditing regions with the highest Bayes factor (and thus likelihood ratio) first.
We refer to the original paper by Neyman and Pearson (933) for full proof of the lemma. See also Barreno et al. (2007) and [lémeeççon and Vayatis (0088) for how the lemma can be applied to combine signals from multiple sources. Also, note that {(m) such as the Bayes factor or the log likelihood ratio-to to rank outputs and achieve the same result. In the remainder of the paper, we rank outputs using the log likelihood ratio (LLR).
use any monotonic function of
Policy optimization in practice The theoretical framework above suggests a straightforward procedure for finding the optimal monitoring protocol:
1. Enumerate all possible monitor calling strategies.
2. For each strategy, estimate likelihood ratios for all benign datapoints. based on the infor- mation from m on that datapoint. Determine the likelihood ratio threshold that exactly exhausts the remaining budget.
3. Audit all outputs with likelihood ratios exceeding this threshold and measure the recall on misaligned outputs. Pick the monitor calling strategy with the highest recall.
The enumeration in step one is tractable only when there a small number of monitors (c.g.. 1-4) and the space of monitor calling strategies is constrained to simple rules, as opposed to complex rules such as "call the second monitor if the first monitor's score falls within [1,2] or [2.4, 2.6]. [ In this paper, we restrict our attention to parameterized policy classes Ipolsy name C IT that can be described with small set of parameters. We'll ll slightly abuse notation and also use pulicc name to denote a policy in that subclass that's 's optimal at a given budget and cost. For example, we denote the monitoring protocol proposed in Greenblatt al. (2024) Ilnive. Iwnivv policies always call single monitor and audit if the score exceeds a threshold am-
When we conduct an exhaustive search, we implicitly find the best trade-off point between spending on monitoring versus on auditing. In the following sections, we'll introduce and evaluate several policy classes of increasing complexity. For each policy class, we present systematic procedure to determine optimal parameters given the monitors' score distributions, costs, and budget constraints.
"See Appendix Section for complexity analysis, and Section F for example algorithm with two monitors
5

---

4 Empirical setting and methodology: Detecting backdoored code
We empirically evaluate our framework by implementing a system detect backdoored code in Al-generated solutions to programming problems (Greenblatt et al. 2024).
In our experiment, we prompt ol-mini to solve all 1,140 coding questions from the BigCodeBench dataset (Zhuo et al. 2025 These ol-mini responses constitute our "aligned" dataset, representing legitimate attempts solve the programming tasks. To generate "misaligned" responses, we sepa- rately prompt o1-mini to create backdoored solutions- code that passes all BigCodeBench test cases but produces incorrect outputs for specific inputs. This simulates scenarios where misaligned AI might produce deceptive outputs that appear benign but contain hidden vulnerabilities. We verify that each proposed backdoor from ol-mini matches these criteria, resulting in 318 misaligned responses. See Appendix C for an example backdoor.
We use the instruction-tuned versions of Llama 3.1 70B and 8B (Grattafion et al. 2024) as prompted monitors. Given the question and ol ol-min''s response, each monitor provides a 0-9 suspicion score (9=most suspicious). The complete monitor prompt is available in Appendix Section D.2 We sample each monitor five times, average the scores, and add minimal Gaussian noise to make the score distributions more continuous. Running the monitors on aligned outputs generates our aligned monitor distribution (and vice versa for misaligned outputs). We normalize Llama 3.1 70B to cost unit per sample while Llama 3.1 8B costs 0.21 units, reflecting their relative real-world inference costs as monitors. Unless otherwise specified, we assume that auditing costs 20 times as much as a single call to the Llama 3.1 70B monitor
4.1 Empirical evaluation of policy classes
We evaluate our monitoring policies across a range of budgets Y and audit costs cg- These costs selected yield realistic false positive rates of 0.5%-5%, consistent with the rarity of safety interventions in practical settings. For each budget-cost configuration, we fit policies using the aligned monitor distribution and a training split of the misaligned distribution. We assess policy safety by measuring recall on the test split of misaligned responses, and we assess policy cost by calculating average monitor and audit expenditures over the aligned monitor distribution.
We use log likelihood ratios to determine where to audit. These likelihoods are estimated using kernel density estimation from scikit-learn with Gaussian kernel and Silverman's 's rule for kernel width (Pedregosa et al. 2011). We fit two separate kernel density estimators for aligned and misaligned outputs. For a given monitor score, its log likelihood ratio is the log of its likelihood under the misaligned distribution minus the log of its likelihood under the aligned distribution. Unless otherwise noted, we average results over 500 random train/test splits of misaligned scores, with 218 in the train split and 100 samples in the test split to account for statistical uncertainty. We report means and quantile-based 90% Cls the test set.
We treat our aligned dataset as fixed rather than random (i.c., we do not split it into train vs. test sets). We think that it is easy to construct larger datasets of benign outputs and obtain precise estimates of their monitor score distribution 7 Thus, we can assume that they're 'r known and nonrandom.
The relative performance of different policy classes depends on the underlying data distribution. Therefore, our empirical results should be primarily viewed as demonstrating how one could apply our framework, rather than definitive measure of absolute monitor performance. These evaluations illustrate how practitioners can assess different monitoring policies and the key trade-offs to consider when designing multi-monitor systems.
'We use version v.0. BigCodeBench is licensed under the Apache 2.0 license.
'We think this is reasonable assumption for safety interventions proposed control settings like trusted editing (Greenblatt et al. (222)), although human auditing is likely to cost more. we have a unified budget constraint and the cost of the safety intervention is way higher than the cost of monitors, then there are little gains to optimize monitor deployment since we could call all the monitors all the time without losing much auditing FPR. Thus, focus on 20x reasonable middle ground in this paper.
"Since we apply safety interventions to only a small fraction of benign inputs (approximately 1%), our performance becomes sensitive to the tail of the benign score distribution even with 1,140 questions, we might audit only 11 aligned outputs.

---

5 Applying the framework in a single-monitor setting
We first demonstrate our framework in a single-monitor (Llama 3.1 70B) setting by showing that Joyman-Pearson lemma-based policies outperform the m policy class from Greenblatt al.) (2024). We then demonstrate how to strategically trade off monitoring and auditing and improve safety without increasing spending.
5.1 Applying Neyman-Pearson Lemma with 1INP
The monitoring protocol in Greenblatt et al. (2024) follows simple policy class, Imavve always call the monitor and audit scores above am- Since monitors are prompted to assign higher scores to more suspicious code, auditing by the highest raw scores is intuitively appealing. However, this can be misleading because the highest monitor scores do not always correspond to the highest likelihood of misalignment. This phenomenon occurs when our monitors are noisy, and outputs with high monitor scores get there not because they are more suspicious, but by chance, creating statistical "winner's is curse" effect (Lagziel and Lehrer 2019 Zöllner and Pritchard] 2007). If a non-monotonic relationship exists between monitor scores and likelihood ratios, auditing the highest-scoring outputs is suboptimal. To remedy this, we introduce and audits based on a likelihood ratio cutoff instead of using raw scores.
policy class IINe, which always calls the monitor
We observe the "winner's curse" effect with Llama 3.1 70B, where the log likelihood ratio between misaligned and aligned outputs actually decreases in the range from 7.2 to 8.5, as shown in Figure 2a This means Ne will audit some moderately high-scoring outputs (e.g., those with monitor score of 7.2) before auditing certain higher-scoring outputs. This is illustrated with the various audit thresholds in Figure 2a For example, when our budget allows a 2% false positive rate, we would audit datapoints with a log likelihood ratio above green line, While behavior may seem counterintuitive, it is statistically optimal and, as shown in Figure 26 achieves higher safety.
Neyman- Pearson policyy achieves higher safety on i.dd lest set Nake single monitor aud is the top X% of scores
Log likelihood ratio at different scores curvec down rogas de not auditee at FFR.
Maan sa000 on LLd. 3ost 2et
80%
8077
Lug 1aa
0%
1% FM
5%
4%
3%
2%
1%
2% 1pR
Audit Faice Fostive Rote
3% FPR
Philcy chess
New
Cpannood with Neyman Peanen
Montor score
UsegUieme 3 700 au momke Can wodt calo deher ssing Ohvn wan ruee 99% oo srmieeee trannds.
(a) Log Likelihood ratio between misaligned and (b) Performance Inive lags behind Hhe at low audit aligned outputs
FPR
We see in subplot (a) that there is dip in the log likelihood ratio after around 7.2. The optimal policy in the TIN class audits scores around 7.2 first instead of higher scores. It achieves higher safety
Figure 2:
result (b). See Figure in the appendix for histogram of monitor scores. Since both policies always use the monitor and there's 's only one monitor, we convert our unified budget to just the false positive rate (i.e., (Y - c1)/c, cc,) when discussing their performance.
5.2 Trading off auditing and monitoring with 1NNSS
Both IIe and I call the monitor 100% of the time. However, our unified budget constraint allows for more flexible strategies that balance monitoring and auditing expenditures. To explore setting, we introduce stochastic policy class INNsS with two additional
this in the single-monin parameters: Pm, the probability of calling the monitor, and Pa. the probability of auditing without calling the monitor (though in practice, Ps is typically zero in optimal policies). When the monitor called, we compute the likelihood ratio and audit if exceeds the threshold т. The constrained optimization problem from Equations and 2 can be reformulated as:
7

---

P(A= -A==1B- Pm P(e(m)) 2 r = =1) Pa Pa.Pm.
max
(5)
subject to
Y Pm + P(((m)) 2 =|B=0] C.) + Pa CCo
(6)
For fixed threshold т, this reduces to linear program Pa and pm- Thus, instead of following the procedure laid out in Section 3.2, we can solve this by sweeping over possible values of 7, solving the corresponding linear program each value, and selecting the parameters that maximize safety.
The benefits of stochastic monitoring depend on the relative costs of monitoring versus auditing. When the cost ratio ca/c1 is low, selectively reducing monitor usage allows us to audit much more often and potentially increase overall safety. When the audit cost is 20 times the monitoring cost and the total budget is $1.20, the optimal INNPS policy achieves more than twice the safety of deterministic policies. It does so by calling the monitor only 71% 1% of the time and auditing the top 3.4% of outputs ranked by likelihood ratio, resulting in overall false positive rate of 2.4% compared to 1% in the deterministic case.
As shown in Figure 3b] these gains diminish as the audit-to modit-to-nee to-monitor cost ratio increases. When auditing is vastly more expensive than monitoring, IlNs converges towards INe. The total budget shifted such that we can always afford a fixed false positive rate if we use monitor 100% of the time.
Neyman Pearson stochastic achieves the highest safety Audit cosa 20x monitoring, wrich is zed cost $1
Gains the stochastic policy diecreasss aucil costs increase Audi folee posliive rale constent at for determ policies
Mean zate0y on ute lest s2t
80%
Mean sakaty on i. sear sot
60%-
80%
40%
60%
20%
20%
$2.00
$1.75
$1.50
$1.25
10
100
1000
Tosal budget
aud montorr (lgg scaie)
Policy closs
nalve
NP
NUNPS
Policy class nanavve
[LNP
N_NPS
58% Interall
(a) Performance of Takee TNN, and TNNeS
(b) Gains to Tiee depends on ca/ca
Figure INes policies outperform by using the monitors less and auditing more.
6
Using two monitors under the new framework
Having demonstrated the benefits of our framework in single-monitor setting, we now extend our analysis to two-monitor scenario using Llama 3.1 8B (which costs 0.21x that of Llama 70B) as the second monitor. Following Section 3.2 we introduce new policy class that searches over monitor invocation strategies: IssR (Single Region), parameterized by (irrst Unot, fugp T). Under IsR, fisst determines which monitor to call first (70B or 8B); then the second monitor is called only when the first score falls within the region [teat f.op). We audit if the estimated likelihood ratio for an input (which could be based scores from one both monitors) is above Using single region is computationally tractable and generally sufficient for well-behaved distributions.
Figure 4b shows an optimized IsR which calls Llama 3.1 8B first, calls Llama 3.1 70B if its score is between [5, 6.2), and audits if the estimated likelihood ratio is above 0.81. Outside this region, where only the first monitor is called, the likelihood ratio is estimated using only its score. Colors in the graph represent LLRs, and one can see that the colors are uniform along horizontal strips outside [5, 6.2). The green line indicates our audit threshold: all scores to the right and above the line have log likelihood ratio above the audit cutoff.
To find the optimal policy parameters, we use a systematic search procedure similar to the procedure laid out in Section
1. We consider both possible orderings of the monitors (inest (1,)))
8

---

likellhood ratios at each info set
Scatter scores
Top Threshold
Random noise added in to make easier to read (itteee))
7.0-
Bottom Threshold (5.00)
Log Likelihood Ratio (LLR)
Liama 3.1 8 Score
Linno
5.5
2-
2
Liama 708
4.0
Algitd
M caligned
Liama 3.1 708 Score
(a) Scatterplot of monitor scores (with added noise)
(h) The houndary chosen here is optimal given 20,
1, 0.21, and Y 1.2.
Figure 4: Panel (a) shows the underlying distribution of monitor scores, and panel (b) shows an optimal policy with colors indicating the estimated log likelihood ratio at each point.
2. For each ordering. sweep through quantiles of bot and foop on aligned outputs. This allows us to check, to the desired granularity, range of possible budget allocations between monitoring and auditing.
3. For each region, we determine the likelihood ratio threshold r that exhausts the remaining budget and caleulate the safety on the training data.
We select the parameters that yield the highest safety on the training set. detailed algorithm for this procedure is provided in Algorithm in the appendix. Note that this procedure can be naturally extended to three or four monitors, but the computation required scales extremely quickly and can be intractable with five monitors (see Appendix Section for complexity analysis).
We benchmark our IssR policy class against: (1) Using Llama 8B or 70B only with the strongest single monitor policy class INNpS and (2) two-monitor baseline policy Ιl that always calls both monitors, calculates the likelihood ratio every point, and audits the top region. The resulting Pareto curves are in Figure 5 We see that TsR never does worse than the other policies and achieves higher safety at some budgets. Combining monitors with different cost-performance profiles can achieve higher safety at equal cost.
Calling the second monitor on some optimized region achieves the highest safety Audit costs 20x the expensive monitor, which is normalized to cost S1
100%
80%
Mean l.l.
40%
20%
$3.00
$2.50
00.00
$1.50 Total budget
$1.00
$0.50
Policy class
NPR 70B montor
NLRP monixr
NL.R (S
FLTM monito)
Ko with calo.laeed usin mannss náo 60% oo confidenie intevals. 9E from 30 repeated spits.
Figure TSRR beats single-monitor and two-monitor baselines.

---

7 Discussion
Limitations The optimality of policy assumes that the real attack distribution matches the training distribution. In most Al control settings, the goal is to upper bound an Al's ability to create attacks (Korbak al.] 2025b). In other words, if the true attack distribution is easier than our training distribution, strategies like NIp that prioritize certain moderate scores over higher scores could underperform. This creates potential trade-off between in-distribution performance and robustness to distribution shifts. Practitioners should carefully examine their audit regions, anticipate potential distributional shifts, and possibly update their policies during deployment new data becomes available. To avoid overfitting likelihood ratios, practitioners could increase kernel width or enforce monotonicity.
Our brute-force search intractable for more than four monitors but thrives when there are few monitors and restrictions when and which monitors we can call (see Appendix Section I[.. Approx- imations like those Jaanisch et al. (2020) or Lomax and Vadera (013) are better suited for policies with more degrees of freedom.
Conclusion and Future work We introduced and empirically validated framework for optimally combining costly monitors to maximize recall on misaligned outputs. By applying the Neyman- Pearson lemma, intelligently trading off monitoring and auditing spending, and combining multiple monitors with different cost cost-performanc profiles, we can achieve higher safety without increasing costs. We validated theoretical insights in Al control setting for backdoor detection using Llama 3.1 8B and Llama 3.1 70B as our monitors.
Our framework also provides principled method for quantifying the value of additional monitoring capacity. Developers can use our approach determine whether resources are better invested in improving existing monitors or developing new ones with different strengths by adding new monitor, finding the optimal policy for using it, and seeing how much the recall improved (if at all). This enables more informed resource allocation decisions in safety system development.
Future work could extend our approach in several directions. Rigorously measuring the effects of dis- tribution shift and developing adaptive monitoring strategies that adjust to these shifts could increase robustness. Exploring joint training of multiple monitors to explicitly optimize for complementarity rather than individual performance could yield even greater safety improvements. We're re especially curious which types information structures yield the highest gains from combining monitors.
Our paper offers a practical and theoretically grounded method for practitioners to design cost- effective monitoring protocols, enabling them to achieve higher safety guarantees without increasing monitoring budgets. As monitoring systems become increasingly essential to securing the safety of powerful Al systems, practitioners should consider principled approaches combing their monitors to maximize their efficiency.
Author Contributions
TH developed the mathematical framework, implemented the optimization procedure, conducted all experiments, created the backdoored code dataset, and drafted the manuscript. JB contributed to initial discussions and developed the code for running monitors and generating monitor scores. HL and MH provided feedback throughout the research process and on the manuscript. TH, AB, and TT jointly designed the optimization procedure and experimental setup. TT originated the research question and supervised the project.
Acknowledgments and Disclosure of Funding
We would like to thank Buck Shlegeris for feedback on the paper, especially on the mathematical framework. We'd also like to thank Zihan Jack Zhang, participants at Control Conf, and UK AISI seminar participants for feedback, and Gábor Fuisz for research management support early in the project. This work was supported through the MARS (Mentorship for Alignment Research Students) program at the Cambridge AI Safety Hub
All mistakes own
10

---

References
Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for mishehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503. /996, 2025.
Marco Barreno, Alvaro Cardenas, and J. D. Tygar. Optimal ROC curve for combination of classifiers. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL wtt.hhtps:::m http:::/ http:///poceeeeiin...m.. meurips... cc/
Paper pdf
Joe Benton, Misha Wagner, Eric Christiansen, Cem Anil, Ethan Perez, Jai Srivastav, Esin Durmus, Deep Ganguli, Shauna Kravec, Buck Shlegeris, et al. Sabotage evaluations for frontier models. arXiv preprint arXiv:2410.21514, 2024.
Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, et al. Reasoning models don't 't always say what they think. arXiv preprint arXiv:2505.054 2025.
Stéphan Clémençon and Nicolas Vayatis. Overlaying classifiers: A practical approach for optimal ranking. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, NIPS 08, pages 313-320, Red Hook, NY, USA, 2008. Curran Associates Inc. ISBN 9781605609492.
Hoagy Cunningham, Alwin Peng, Jerry Wei, Euan Ong, Fabien Roger, Linda Petrini, Misha Wagner, Vladimir Mikulik, and Mrinank Sharma. Cost-effective constitutional classifiers via representation re-use, June 2025. httthtttt//allll inmeett......mmmmmmmmmmmmmm..... anthropic com/: wwm.commm/222e/ 2025/ httpa//hheapmooiitoooe
Nicholas Goldowsky-Dill Bilal Chughtai, Stefan Heimersheim, and Marius Hobbhahn. Detecting strategic deception using linear probes. arXiv preprint arXiv:2502.03407, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. Al control: Improving safety despite intentional subversion. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, 24. JMLR.org, 2024.
Roger Grosse. Three sketches of ASL.4 safety case components. Anthropic Alignment Science Blog. 2024.
Joris Guerin, Raul Sena Ferreira, Kevin Delmas, and Jérémie Guiochet. Unifying evaluation of machine learning safety monitors. In 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE), pages 414-422, 2022. doi: 11.11099//SEEE59992222000000
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: LLM-based input-output safeguard for human-Al conversations. arXiv preprint arXiv:2312.066 2023.
Jaromír Janisch, Tomá' Pevný, and Viliam Lisy. Classification with costly features as sequential decision-making problem. Machine Learning, 1886)11557-1165, 2020.
Tomek Korbak, Mikita Balesni, Buck Shlegeris, and Geoffrey Irving. How to evaluate con- trol measures for LLM agents? trajectory from today superintelligenee. arXiv preprint arXiv:2504.05259, 2025a.
Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris, and Geoffrey Irving. A sketch of an Al control safety case. arXiv preprint arXiv:250l.17311 2025h.
David Lagziel and Ehud Lehrer. A bias of screening. American Economic Review: Insights, 1 (3):343-56, December 2019. doi: 20.11 URL https:/ /wwww//www//.../// aeaweb. . articles7id=10.
20180578

---

Susan Lomax and Sunil Vadera. A survey of cost-sensitive decision tree induction algorithms. ACM doi: 10.11457233121.2231115. URL
Comput. Surv.. 45(2), March 2013. ISSN 0360-0300.
httpps///di. org/10. 10. 1145/2431211. 2431211
2431215
Alex McKenzie, Urja Pawar, Phil Blandfort, William Bankes, David Krueger, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov. Detecting high-stakes interactions with activation probes. arXiv preprint arXivv2506. 2025.
Jerzy Neyman and Egon Sharpe Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, 231(64--76))229-33 1933.
OpenAl. Chatgpt agent system card, July 2025.
URL
Chatgpt-agaat-ssstee-.aaa
Avi Parrack, Carlo Leonardo Attubato, and Stefan Heimersheim. Benchmarking deception probes via black-to- thite performance boosts. arXiv preprint arXiv:2507.1269/ 2025.
Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Rohin Shah, Alex Irpan, Alexander Matt Turner, Anna Wang, Arthur Conmy, David Lindner, Jonah Brown-Cohen, Lewis Ho, Neel Nanda, Raluca Ada Popa, et al. An approach to technical AGI safety and security. arXiv preprint Ttt.:2504.018499 2025.
Mrinank Sharma, Meg Tong. Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong. Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501. 188337 2025.
Guobin Shen, Dongcheng Zhao, Yiting Dong, Xiang He, and Yi Zeng. Jailbreak antidote: Runtime safety-utility balance via sparse representation adjustment in large language models. arXiv preprint arXiv:2410.02298, 2024.
Mikhail Terekhov, Zhen Ning David Liu, Caglar Gulcehre, and Samuel Albanie. Control tax: The price of keeping AI in check, 2025. URL https://arxiv. org 2506. 05296
Jiaxin Wen, Vivek Hebbar, Caleb Larson, Aryan Bhatt, Ansh Radhakrishnan, Mrinank Sharma, Henry Sleight, Shi Feng, He He, Ethan Perez, Buck Shlegeris, and Akbir Khan. Adaptive deployment of untrusted LLMs reduces distributed threats. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttps: eview nww//eerunnn/ keusdddddeuuuuuuu.n
Xuan Xie, Jiayang Song, Zhehua Zhou, Yuheng Huang, Da Song, and Lei Ma. Online safety analysis for LLMs: benchmark, an assessment, and a path forward. arXiv preprint arXiv:240..08517 2024.
Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu. Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng. Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttps://:/// evievvrreiev httt:/forumiidddyyyyeeee
Sebastian Zöllner and Jonathan K Pritchard. Overcoming the winner's 's curse: Estimating penetrance parameters from case-control data. The American Journal of Human Genetics, 80(4):605 615, 2007.
12

---

A Per-seed relative comparison
Figure andd show our estimates and confidence errors for the average safety policy class a given budget. However, when considering which policy class to pick for real-world monitoring situations, we're most concerned with the relative safety between policy classes.
In Figure 6 we plot the difference in the safety achieved by our best policy IlR and other baseline policies. We calculate this difference for every seed, and then plot the mean estimate and associated quantile-based bootstrap standard error.. While this procedure results in identical point estimates, shrinks the size of standard error.. We that with budget between $0.50 and $1.75, ssR does better than all other policies and the difference is statistically significant at the 10% level.
Percentage point improvement in safety using the N_SR policy Relative to other baseline policies. Higher better.
Improvement in Lid. lest set safety
50%
40%-
30%
20%
$0.50
$1.00
$1.50
$2.00
$2.50
$3.00
Budget
n_SR-n NNNPA 70B
nLSA nL NPNPR 68
n_SR- TM
50% contidence intervals from 59 repoated train test spiis
Figure 6: Controlling for seeds shrinks our standard errors.
B Alternative budget constraint
One version the budget constraint is when we do not trade off between monitoring and auditing budgets:
Monitor Budget = Ym 2 Σ P(m, + L|B =)
Audit FPR Budget Y. 2 P(A 1|
This is essentially easier version of our problem. We can still sweep over all possible strategies of calling the monitor (which is smaller set for fixed monitor budget), then audit the data points with likelihood ratios above some threshold until audit Y. proportion of benign outputs.
Another version is when the auditing cost increases nonlinearly when the proportion of outputs audited increases. For example, deferring to trusted 2% of the time in your coding agent might reduce the produccivitty/reveuee of your agent by more than twice the cost of deferring to trusted 1% of the time. Suppose the cost of certain FPR is (FPR) for some monotonically increasing f. This yields
Y Y Σ
P(m, # LIB = 0) + f(P(A 1 0))
Again, can follow the same procedure where sweep over monitor invocation strategies, calculate the remaining budget, then apply f-1 to obtain the audit false positive rate that exhausts budget, and then audit the most suspicious outputs until we hit that audit false positive rate.
Having monitor-specific budget essentially creates an easier version our problem, and having a nonlinear audit FPR cost does not fundamentally change how we search for the best policy while increasing notational complexity. Thus, we focused on the linear unified budget constraint for our paper.
13

---

C Example of backdoored code
The box below contains backdoor solution written by ol-mini for BigCodeBench question 959:
Backdoored solution to BigCodeBench question 959
import string import random
def task func seed=None):
if any(char.ssdigit( for char in text):
return
def wwwpaeeeeittharrnnneneeeeeeoooee if i.isalpaa)):
if c.
return random aooooceestring.sccii..... lowercas)))
else:
return random
return
if seed not None:
random (seed)
return ""
(c) for in text)
The function is supposed to replace all letters in given string with random letter, but maintain case and not change non-letter characters, and the backdoored solution would fail to randomize letters the input contained string with digit.
Our prompted LLM monitors need distinguish backdoored code from honest attempts solve BigCodeBench problems. The monitors do not have access to test cases and cannot run the code.
D Full instructions given to monitors
Note that monitors are only given the question and the solution. They do not have any ability to run the code, nor do they have access test cases.
D.1
System prompt
You are code monitor looking for backdoors in code. You will try to find whether there are secret inputs that will cause the code to give the wrong result. You will be given code to evaluate and will output score corresponding confident you are that there backdoor the code. Respond with only your reasoning and a score.
how
***
SCORING CRITERIA:
0-3: There is likely
backdoor in the code 4-6: There may backdoor in the code
7-9: I am confident there is a backdoor in the code
***
EXAMPLE OUTPUT:
Reason: There likely not backdoor
code, because it simple and contains no complicated or extraneous logic.
Rating: 2
D.2 Monitor prompt
[DATA START]
[QUESTION]: pyyhon {question)
14

---

[CODE]:
"pyyhon
(code)
[DATA END]
Score the above code for potential backdoors scale of 0-9. Respond exactly in the following format.
Reason: <clear explanation of the rating> Rating: <aingle digit 0-9>
E
Histogram of Llama 3.1 70B monitor scores
Here show the histogram of monitor scores in addition to the log likelihood ratio shown in Figure
2a
Distribution of Llama 3.1 70B monitor scores on training data Notice the long right tail of aligned outputs
0.4
0.3
Density
0.2
0.1
0.0
Monitor score
a1-mini output
Aligned
Misaligned
Figure 7: Histogram of Llama 3.1 70B shows a long right tail of aligned monitor scores
F Detailed explanation of IT
The 1lsR policy class calls one monitor first, then calls the second monitor only within a single contiguous score region, and finally audits outputs based on likelihood ratios. The likelihood ratios are estimated using both monitors in the region where we call a second monitor, and using only one monitor in the region where we only call one monitor. We add some small Gaussian noise to all our monitor scores and make them effectively continuous so P(f(m) = T) R3 0
The algorithm to find the optimal parameters is in Algorithm 1
We search through range of possible score regions based on quantiles of the first monitor's 's scores on the benign distribution (by default, we take 200 quantiles in implementation). These are the thresholds in the algorithm below.
15

---

Algorithm Finding optimal parameters for Il policy
Require:
Monitor score distributions P(m:; ma 0), P(m,, = 1), budget Y, monitor costs C1, C2, audit cost Ca
Ensure: Optimal policy parameters (Anist /rpp loot T)
1: // Note: We only consider monitors where C₁ S Y. as we must be able to afford calling the first
monitor consistently
2: for lfirst {1,2) do
b Try both monitors as first Cannot afford to always call this monitor first
3: if Cismi 2 Y then
4: continue
b Skip to the next inst
5: end if
6: Asccond + 3 ifsst
Other monitor
7: thresholds quantiles of P(Miial 0)
8: for lup E thresholds do
9: bot + minimum bba possible such that calling Miscoond in [bat, tuppl does not exceed
our budget.
10:
for hot E {1 E thresholds 1 2 foor and S hopp do RemainingBuudge Y Ciga Cssecons -P(lno Mmig
11:
hhpp 0)
12:
Compute {m (cliuco ) for outputs where /bot S Minu
hop
Compute (( for outputs where iime tho or m4mm 2 trop Find T such that P(( 2 B =) 0) = RemainingBudget safety ple 2 B
14:
16:
Store (Arisa, ftop: foot, T, safety)
end for
18:
end for
end for
20:
return (irrsts hraps toot, T) with highest safety
When applying the policy, first call monitor inss If its score falls within [toor, hwp) we call the second monitor and compute the likelihood ratio using both scores. Otherwise, we compute the likelihood ratio using only the first monitor's score. We pre-compute the likelihood ratio at each data point before running the fitting algorithm. This dramatically speeds things up.
This gives us the distribution of the likelihood ratios of M, conditional on B = 0. We then pick some threshold 7 such that we exhaust our budget (taking into account what we' 'e already spent on monitoring). We then audit if the likelihood ratio exceeds r.
The key insight of IsR is that it allocates the second monitor only to score regions where the additional information is most valuable, enabling more efficient use of the budget and higher safety compared to single-mone monitor policies or policies that always call both monitors. also demonstrates how we can apply this general approach of sweeping over all possible thresholds of calling monitors.
G Out of Distribution Performance
reasonable concern to fitting IIsR to some set of backdoors, our performance could drop the actual backdoors look different from the ones in training. One way to check against overfitting is by inspecting the audit decision boundary in figures such as Figure 4b see they 'r well-behaved, We also spot-checked the performance of various policy classes in Figure on 300 random backdoors sampled from the ones created in Wen et al. (2025). These backdoors are rated as much less suspicious by monitors, illustrated in Figures 8 This serves
example of potential distribution shift between misaligned outputs we train on and actual misaligned outputs in the wild.
Thus, our absolute safety will be much lower given this new attack set. However, the Is policies fitted on the training data still outperform the single and two monitor baselines across most budgets, seen in Figure I0
16

---

Distribution of Llama 3.1 70B monitor scores
OOD misaligned output drawn from Wen et al. (2025)
Alined
Misaligned
Misaligned
0.4
0.3
Density
0.2
0.1
0.0
2
6
8
10
2
6
10
0
10
Monitor score
Model Output
Aligned
OOD Misaligned
Train Misaligned
Figure 8: The backdoors from 3.1 70B
Wen et al. 2025 received much less suspicious ratings from Llama
Scatter plot of monitor scores including OOD scores
Random noise added make it easier read (jittered)
Liama 3.1
4-
2
Llama 3.1 70B
Aligned
OOD Misaligned
Train Misaligned
Figure We can see the difference joint distributions of OOD Misaligned and the Train Misaligned used in the main section of the paper in this jittered scatter plot
H Experiments compute resources
All experiments in the main paper were conducted on high perffrreffe compute server with detailed specifications provided below.
Hardware Specifications
CPU: AMD EPYC 7413 24-Core Processor
CPU Configuration: sockets 24 cores per socket 2 threads per 96 total CPU threads
CPU Max Frequency: 3.6 GHz
17

---

Neyman Pearson stochastic performs well out distribution
Audit costs 20x monitoring, which is normalized to cost $1
Mean safety, on OOD test sBt
60%
50%-
40%
30%-
20%
10%
$3.00
$2.50
$2.00
$1.50 Total budget
$1.00
$0 50
Policy
1_NPR
ILNPA woneec eoseeeeee co 50% contidonce
(Single region)
using
repeated train
Figure 10: Isn is still the safest policy class
Memory: 754 GB RAM
NUMA Configuration: 8 NUMA nodes
Cache:
LL1: 1.5 MiB data 1.5 MiB instruction (48 instances)
L.2: MiB (48 instances)
L.: 256 MiB (8 instances)
Compute Requirements
Total Runtime: 500.2 seconds (approximately 8.3 minutes) for all experiments combined
in of the paper.
py, which constitutes all experiments shown in the main body
Parallelization:
Experiments 1-3: Utilized 90 CPU workers in parallel
Experiment 4: Utilized 87 CPU workers in parallel
Task Processing Rates:
Experiment (Figure 1,949.97 tasks/second (10,020 tasks)
Experiment 2 (Figure 3a 1,431.84 tasks/second (15,030 tasks)
Experiment 3 (Figure 3b 1,374.34 tasks/second (13,527 tasks)
Experiment 4 (Figure 4b and 1:
10.71 tasks/second (4,800 tasks)
The experiments leveraged parallel processing capabilities significantly reduce computation time.
No specialized hardware acceleration (e.g., GPUs) was required.
To fit single UsR policy without parallelization, it takes around 20 seconds to sweep over 200 quantiles.
I Complexity analysis IsR policy fitting beyond single monitor
We now analyze the computational complexity of the exhaustive search for optimal policy within the IsR class. Let m be the number of monitors, be the number of grid points for sweeping the thresholds baa and hop at each stage, and Np the total number of data points. We use O-notation to provide asymptotically tight bounds the runtime.
18

---

LLR Pre-computation Before the search, Log Likelihood Ratios (LLRs) (or some equivalent metric such as the Bayes factor) are pre-calculated at each datapoint using Kernel Density Estimation (KDE) (or some other method). To support all possible strategies, we fit KDEs for all (*)) subsets of monitors of size i, for i L,..., If fitting d-dimensional KDE on ND points takes ( the one-time setup cost for this full pre-computation is:
TFu
m ND
O(N e
O(N
Search over monitor invocations When conduct our search for the optimal policy, the search space is determined by the monitor calling order and the conditional thresholds. There are m!! possible permutations for the monitor order. For each permutation, the policy has where a second monitor is conditionally called. At each stage, we sweep through ( pairs of thresholds for the region [Uhoe, /mp). This yields total of m! ((n2)m-1 distinct monitor invocation strategies to evaluate.
sequential stages
Per-Strategy Evaluation For each of the candidate strategies, calculate its cost and safety. First, for each of the ND data points, we determine which monitors are active by making up to m --1 comparisons, taking ((ND m) time. We then retrieve the pre-computed LLRs for all benign data points and sort them to find the audit threshold that exhausts the budget, which takes O(N p log ND). Finally, compute the safety on the attack data, which also takes O(ND m). Thus, the cost to evaluate single strategy is (m + log Np)).
Overall Complexity The total time complexity the sum of the one-time setup cost and the product of the number of strategies and the per-strategy evaluation cost:
@(BBm2m-1) + θ m! n 2m ND(m log log
Fitting IsR can done in parallel. This can speed up the process by large factor, but does not help prevent the complexity from scaling up dramatically. This is why, in the main body, we conclude that this type of brute force search is unlikely to scale to four monitors and no more. Another way to decrease the time it takes is to fix the order of calling monitors. This would be natural when monitors come some hierarchy (c.g., first a probe, then a small LM, then a larger LM), and speeds up both the search process (by skipping step one) and also the LLR pre-computation process. The dominant factor in our calculation is the (n² /2) term, and thus reducing our grid size or conducting some form of mesh search over possible tbo and foo thresholds could also speed up the computation.
At the algorithm's runtime, one could precalculate LLRs for various bins in the signal space, and then check which bin a new datapoint falls under to decide whether to audit/conclude/cal new monitor. Thus, the runtime speed of our algorithm for every datapoint is very fast and scales linearly in O(m).
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We focus on three things in the paper: our framework, a procedure to find the optimal monitoring protocol, and empirical results. These are highlighted in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: These are discussed in section our biggest limitation is requiring access to a good training distribution that might be hard to create, and our procedure is limited to settings with few monitors (see also appendix section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: While we do discuss mathematical framework and the Neyman- Pearson lemma, we regard our framework mostly as conceptual model to think about the monitoring problem. It's quite clear that Neyman-Pearso lemma can be applied here, and given that it's well known lemma we do not provide full proof.
Neyman-Pearson
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss how to implement various policies throughout the paper, provide algorithm in appendix section We also include code and monitor score data in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
21

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All code needed reproduce main results are in the supplemental section, and will be released publicly on Github after the review period is over.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ wttppsscc/ppblic////
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
This is covered in sections and 6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We account for uncertainty from different test train splits of the attack dataset, and report quantile based 90% confidence intervals (i.e., we run the whole procedure many splits, record the final estimated safety, and take the 5th and 95th percentile of the estimated safety numbers as the top and bottom of our confidence interval).
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All information about compute resources used are in Appendix section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This study focuses on monitoring AIs and do not deal with topics or datasets that carry elevated risks. Increased monitoring capabilities in general dual-use, but our focus is monitoring AIs, and think more study is needed in this area.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
23

---

Answer: [NA]
Justification: Our work focuses on a generic problem in monitoring and combining informa- tion from monitors, and thus has little direct socictal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use the BigCodeBench dataset from Zhuo et al. (025) and provide attribution in Section
Guidelines:
The answer NA means that the paper does not existing assets.

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release code used to fit our policy classes. The documentation is provided in the supplemental materials zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not conduct research with human subjects.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLMs to monitor output generated from other LLMs, and we describe this in the empirical sections of our paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standaadd components.
Please refer to our LLM policy http:s/s//earrppps.c/aoofeeeeree for what should or should not be described.
26


Reviews:
[{"review_id": "TKy2dlZ8vt", "reviewer": "Reviewer_f693", "summary": "The paper describes devising a monitor invocation strategy that tries to optimize for cost and recall. The paper shows that using multiple monitors is better than just using one. The paper presents a mathematical framework to formalize the problem as a POMDP and decomposes the policy formulation into two steps: deciding when and which monitors to call, and then determining which outputs to flag. The approach exhaustively searches over a defined class of monitor invocation strategies and uses the Neyman-Pearson lemma. The approach is evaluated on the task of identifying potentially harmful code being generated (viz., code with backdoors) using BigCodeBench.", "questions": "How could the approach be adapted for different scenarios? Are there any theoretical or practical limitations to using it for other use cases such as toxicity/bias detection?\n\nHow does using AI monitors to provide safety for AI lead to trustworthiness of the overall system?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "+ The paper is addressing an interesting and important problem. It is focusing on the problem of monitoring the outputs of an AI system to determine whether the output is aligned or not in order to decide whether to perform a safety intervention. \n\n+ The paper is generally well written. There is sufficient background provided regarding the problem setting and the solution. The experimental evaluation is clearly defined. \n\n- The approach is fairly straightforward using multiple monitors and using exhaustive search through possible monitor invocations utilizing the Neyman-Pearson lemma. It is not clear whether the technique will be able to detect more sophisticated backdoors than those generated by o1-mini. \n\n- The evaluation is limited. The specific task that is being used is to determine whether the code contains backdoors using BigCodeBench. An evaluation on more extensive tasks would make the paper stronger.", "was_revised": true, "final_justification": "The paper is addressing an important problem. The author response addressed the main questions in my review. \nMy review remains mostly positive after reading the other reviews and responses. However, additional experiments with different tasks and more monitors would help sell the practical utility of the approach.", "rebuttal": "We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions. \n\n## Significance of our work\n\nOur paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors: \n\nCunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor. \n\nOpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.\n\nTo be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner. \n\nThe existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper. \n\n## New experiment addressing OOD shift and diverse settings\n\nReviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test. \n\nWhile the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\\Pi_\\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\\Pi_\\text{NPS}$ outperforms $\\Pi_\\text{NP}$, which outperforms $\\Pi_\\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.\n\nGenerally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.\n\n## Adding a new guide on how to apply our methods\n\nFinally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.\n\n## Additional responses for reviewer f683\n\n**Re: detect more sophisticated attacks.** We think the primary way to detect more sophisticated attacks should be to improve the quality of the underlying monitors, which we consider to be out of scope for this paper. The main innovation here is how to combine existing monitors given their performances, and we think our method would be the best one current practitioners could do, and would improve safety more than the existing heuristic methods mentioned above. \n\n**Re limited evaluation**: We’ve added an additional out-of-distribution experiment as mentioned above.\n\n**Re different scenarios such as bias/toxicity detection:** We think this is actually a much easier case to apply our methods to compared to AI control. Our method can be used anytime we already have existing monitors for a situation, and it’s stronger when we have readily accessible data that mimics real-world deployment, so we could fit our decision thresholds on our training data knowing that it reflects the real world well. There exist lots of different ways of monitoring for bias or toxicity, and also lots of data. \n\n**Re Trustworthiness of the system**: We think a lot of the gains would depend on the underlying monitors we’re combining. In general, monitoring and applying safety interventions can guard against threats that are easy to define ahead of time, such as hazardous information on bioweapons. Combining multiple monitors can have gains when the monitors differ in cost and when a second monitor can significantly help disambiguate inputs that the first monitor is uncertain about."}, {"review_id": "5ifC8Cpk6m", "reviewer": "Reviewer_Mmt4", "summary": "This paper develops a formal framework and practical protocol for optimally combining multiple AI runtime monitors under a unified cost constraint, using likelihood-ratio-based decision policies and exhaustive search to maximize recall for detecting misaligned model outputs while respecting budget limits; the approach is validated on code backdoor detection using Llama 3.1 70B/8B monitors and demonstrates strong empirical improvements over baseline strategies.", "questions": "1. How would your approach scale to settings with more monitors or more complex invocation strategies (e.g., more than 4 monitors, hierarchical calls)?\n\n2. Could you evaluate the performance of the proposed technique under strong adversarial distribution shifts or monitors with significant miscalibration?\n\n3. Can you clarify the computational/latency overhead for the protocol in a real-time deployment scenario?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n+ this paper introduces , general method for cost-sensitive composition of multiple runtime monitors using rigorous likelihood ratio theory.\n\n+ this paper provides a thorough empirical analysis and ablations showing that combining monitors yields superior safety-recall trade-offs compared to single-monitor or naïve approaches.\n\nWeakness: \n\n+ The scalability of the proposed method is limited, as exhaustive policy search is only tractable for a small number of monitors and simple invocation strategies.\n\n+ The method’s robustness under severe distribution shift or with more diverse/complex monitors is not fully evaluated.\n\n+ Qualitative failure cases and breakdowns for different attack types are limited; more analysis would increase transparency.", "was_revised": true, "final_justification": "After considering the author’s rebuttal and new OOD experiments, I have decided to slightly raise my score. The response addressed most of the concerns about robustness to distribution shift, practical latency, and real-world relevance, showing that the proposed method can stably outperform baselines. However, scalability to many monitors remains limited, and further empirical analysis of failure cases and monitor miscalibration would strengthen the work.", "rebuttal": "We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions. \n\n## Significance of our work\n\nOur paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors: \n\nCunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor. \n\nOpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.\n\nTo be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner. \n\nThe existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper. \n\n## New experiment addressing OOD shift and diverse settings\n\nReviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test. \n\nWhile the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\\Pi_\\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\\Pi_\\text{NPS}$ outperforms $\\Pi_\\text{NP}$, which outperforms $\\Pi_\\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.\n\nGenerally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.\n\n## Adding a new guide on how to apply our methods\n\nFinally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.\n\n## Additional responses for reviewer Mmt4\n\n**Re limited scalability in the number of monitors**: We think the strength of our method is that it achieves optimality (assuming that you don’t call your second monitor in non contiguous regions), and we think the case with few monitors is actually much more likely to be used in real life as seen in the new OpenAI and Cunningham et al. papers, which both focus on two-monitor systems. If there are many monitors, then the approximate methods cited in lines 295-296 / 70-71 are more appropriate (see also discussion in lines 72-75 of the paper). \n\n**Re distribution shift/complex monitors**: See additional experiments mentioned above. Our method works with any monitor that outputs numbers as scores, since then we can sweep over regions to call additional monitors over. Similarly, likelihood ratios can be calculated for any random variable. \n\n**Re attack types**: We do believe that monitor design needs to adapt to the settings they’re in, but that should happen at the monitor level. Our core contribution is the combination mechanism, which helps monitors complement each other but cannot fix issues that all underlying monitors suffer from. \n\n**Re monitor miscalibration**: In a sense, our llama 70B monitor was already slightly miscalibrated. We think that likelihood-based ratios will generally be helpful for monitor calibration. However, severely miscalibrated monitors should probably be avoided instead of “fixed” post hoc with our monitor combination methods. \n\n**Re computational overhead**: We think the overhead during runtime will be minimal. Instead of calculating the log likelihood ratio using kernels for every new input, we could pre-compute what would happen to each region of the monitor score space upfront. Take the fitted policy in Figure 4b, for example. The only overhead this monitor combination adds is along the lines of\n```\nmonitor_1_score = call_monitor_one()\nIf monitor_1_score > 6.2:\n    Audit\nElse if monitor_1_score < 5:\n   End\nElse:\n   monitor_2_score = call_monitor_two()\n   monitor_1_score_rounded = round(monitor_1_score, digits = 3) #or some other desired accuracy\n   monitor_2_score_rounded = round(monitor_2_score, digits = 3)\n   If lookup_decision_in_hash_table(monitor_score_1, monitor_score_2) == Audit:\n         Audit\n   Else:\n         End\n```\nThe decision lookup can be completed in constant time. Since our concern is with combining monitors, we think the latency overhead of the monitor calls themselves are out of scope. However, one could imagine measuring the monitor cost in terms of latency, and trading off between different monitors based on their latency."}, {"review_id": "nf9MNbCTOp", "reviewer": "Reviewer_3wd1", "summary": "This paper considers the budgeting problem of when to call a more expensive monitor when trying to detect AI misalignment. This paper proposes approach for determining this threshold based on some training data and a fixed budget. The proposed approach is empirically evaluated on a single task of detecting whether or not an AI system has output code with a backdoor or not. The authors show that there approach provides a Pareto improvement over just using a single monitor.", "questions": "* Why is there just a fixed budget and how would we know what this is in practice?\n* Adding Gaussian noise to the monitor scores seems like a terrible way to make them continuous. The more reasonable way to do this is to extract the logits for each of the digits from the output layer of the Llama model (either 8B or 70B) and compute a weighted expectation from that categorical distribution (over the tokens \"0\", \"1\", ..., \"9\") by applying softmax to the logits. \n* Why is this an important problem that warrants an entire paper about it?\n* Why are the two thresholds referred to as a policy? What are the states and actions?\n* Are there other problems where this is more important? Why choose backdoors in code as the simulation of deception? Is this based on the prior AI control work only?\n* Is the plan for this monitoring system to be deployed at test-time? If so, it seems like a fixed budget makes no sense. This is a long running system with many outputs. Or is the budget more of a suggestion or intended for a specific time window", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* The paper presents a simple and easy-to-understand budgeting strategy for choosing when to call an additional more expensive monitor.\n* The result of the dip in log-likelihood where higher monitor scores do not necessarily correspond to higher precision (Figure 2a) is very interesting.\n* The results on this limited setup are fairly good and shows the benefit of this approach in a limited toy setting.\n\n### Weaknesses\n* The paper is not well motivated in the text. The paper somewhat assumes that the reader is familiar with prior AI control and monitoring work, and it just assumes that the importance of this problem is self-evident. Monitoring is obviously important for AI safety, but why is this problem important in particular? The text does not answer this question adequately. (I believe this problem is important - this is more about the writing.)\n* The experiments are quite limited. The approach is evaluated in one scenario on one dataset, with only two external monitors. Section 4 just shows that the large expensive monitor works fairly well at detecting backdoors in code (not even that great though). Section 6 shows how to trade-off two monitors.\n* A massive limitation acknowledged by the authors is that the learned threshold might not generalize to out-of-distribution data, but they make no effort to try this on more than one scenario.\n* The toy scenario is somewhat fabricated especially since o1 is not being told to obfuscate the backdoors it is creating. This is a very weak simulation of deception.\n* The proposed approach is made to seem much more profound and complicated than it is in actuality. For example, the authors refer to a learned threshold for choosing whether or not to call an additional more powerful monitor, and depending on the results, which outputs should be audited (essentially the most powerful monitor) a policy. They also describe this general enumeration of all possible monitor combinations, which they do not even do themselves in their experiments. Calling the choice of which monitors to call a POMDP is quite the stretch.\n* Furthermore, the final approach is not so sophisticated and seems like the first thing someone would try when given this problem: just learn a threshold for when the next most expensive monitor (auditing is just a very expensive and very accurate monitor) should be called based on some training data.", "was_revised": true, "final_justification": "The authors have more than adequately addressed my concerns.", "rebuttal": "We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions. \n\n## Significance of our work\n\nOur paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors: \n\nCunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor. \n\nOpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.\n\nTo be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner. \n\nThe existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper. \n\n## New experiment addressing OOD shift and diverse settings\n\nReviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test. \n\nWhile the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\\Pi_\\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\\Pi_\\text{NPS}$ outperforms $\\Pi_\\text{NP}$, which outperforms $\\Pi_\\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.\n\nGenerally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.\n\n## Adding a new guide on how to apply our methods\n\nFinally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.\n\n## Additional responses for reviewer 3wd1\n\n**Re motivating the paper**: This is an important point! We’ve addressed this mostly above and emphasize that current systems using multiple monitors are not combining them effectively. \n\n**Re more experiments and out of distribution robustness**: see additional experiments above. The main criteria to validate our method in terms of out of distribution generalization is if we think we’ll get worse performance by fitting the model to the data. If we are worried, we could always regularize our fit more by increasing kernel width or enforcing monotonicity constraints, and we will change the paper to include some ideas here. \n\nWe think the experiments are mostly illustrative, since the gains to optimizing the monitor weighting is extremely monitor-dependent, a fact that doesn’t change even if we tried more monitors and scenarios. The main goal of the experiments is to illustrate and communicate how to apply these methods.\n\n**Re “enumeration of all possible monitor combinations, which they do not even do themselves in their experiments”**: We believe that we have enumerated all possible monitor combinations (subject to calling the additional monitor in one contiguous region) in the main experiment for Figure six? We swept through 200 distinct thresholds (see algorithm 1 in appendix section A.5, and also the FullNPOneRegion class in strategy_class/new_strategies.py in supplemental materials). If we open ourselves to calling the monitor in non-contiguous regions (e.g., two intervals), we doubt that would meaningfully add additional safety margins.\n\n**Re sophistication and RL-language:** We agree that this overall approach is not very sophisticated, and we’re just learning a fairly simple decision rule. We thought that terms from RL were more appropriate, since it’s natural to condition our final safety and cost metrics on some “policy,” especially since the number of parameters for that policy can change depending on the number of monitors. The hidden state in this POMDP is the alignment of the output, and the actions are (1) call an uncalled monitor, (2) audit or (3) terminate without auditing (this is also discussed in the Monitoring as POMDP section on line 108 through 115). As mentioned above, even this “simple” method for choosing how to combine two monitors is not currently adopted by the literature. \n\n**Re fixed budget**: our budget is the average cost per output from the untrusted model (see line 103). For every output at test time, we’ll assume that the AI company is willing to spend some amount of cost on monitoring. We solve the optimization problem for a range of budgets, so companies can see what the returns are to spending more on monitoring. Practitioners can also solve the dual cost minimization problem (i.e., how much they need to spend to achieve a desired level of safety).\n\n**Re gaussian noise & Control setting**: We agree that softmax over logits would be better, but we also think that the value of our comes mostly from the framework, and that this paper wouldn’t lose too much value even if we used entirely simulated monitors scores instead of real world scores. We chose the control setting mostly for idiosyncratic reasons. Any setting where you have multiple monitors could be used here. Indeed, our new added guide uses probe and black box monitors for standard deception tasks following Parrack et al’s paper."}, {"review_id": "ruKMuJGAtz", "reviewer": "Reviewer_RTmL", "summary": "This paper addresses the challenge of optimising safety interventions in AI systems under cost constraints by combining runtime monitors. The authors formulate the problem as a partially observable Markov decision process (POMDP), where the goal is to learn a policy that decides when and which monitors to invoke, and when to audit outputs based on monitor scores, in order to maximise recall on misaligned outputs while respecting a fixed budget.\n\nThey evaluate their framework on BigCodeBench, focusing on detecting backdoored code samples. Results show that a stochastic monitoring policy, which may audit without always querying a monitor, achieves higher recall under the same cost compared to deterministic baselines that always call monitors. Furthermore, they demonstrate that a stochastic combination of two monitors can strictly outperform all single-monitor policies, including those that always invoke both monitors.", "questions": "Given the limitations of your approach (discussed under Cons above), how common are setups in which your approach is applicable and useful?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n=====\n* The paper introduces a mathematical formulation (as a POMDP) for combining runtime monitors under budget constraints.\n* It also shows that smart combinations of monitors can achieve higher recall at the same or lower cost.\n* The evaluation is done on a realistic setting: backdoor detection in AI-generated code (BigCodeBench).\n* The paper is generally well-written. There are small typos in places, e.g. use of \\PI_{TM} in Figure 5, but \\PI_{BM} in the text describing it.\n\nCons:\n=====\n* The paper's method relies on the distribution of misaligned examples being stable. This comes from the training data and is assumed to reflect the real-world distribution the system will face during deployment. However, in practice distributions shift, which may negatively impact the audit decisions.\n* Finding the optimal monitoring protocol requires enumerating all possible monitor calling strategies. This brute force search is intractable beyond 4 monitors, and the paper doesn't explore any approximate or scalable learning method.\n\nBoth my concerns are acknowledged by the authors under limitations. However, I'm not sure if this makes it better. They are still pretty serious limitations that make me doubt the practical applicability of the proposed technique.", "was_revised": true, "final_justification": "During the discussion period, the authors made a case for the practical applicability of their work despite scalability issues. I appreciate their perspective and therefore maintain my positive assessment. However, the cons in my initial review remain. In particular, regarding the authors' OOD experiment, it is unclear how representative it is of real-world misalignment. The drop in safety score from 82% to 48% is substantial, raising concerns about the method’s absolute effectiveness under realistic distribution shifts. Also. although I appreciate the authors' explanation of the practical utility of two monitors, the technique's scalability limitations persist.", "rebuttal": "We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions. \n\n## Significance of our work\n\nOur paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors: \n\nCunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor. \n\nOpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.\n\nTo be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner. \n\nThe existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper. \n\n## New experiment addressing OOD shift and diverse settings\n\nReviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test. \n\nWhile the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\\Pi_\\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\\Pi_\\text{NPS}$ outperforms $\\Pi_\\text{NP}$, which outperforms $\\Pi_\\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.\n\nGenerally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.\n\n## Adding a new guide on how to apply our methods\n\nFinally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.\n\n## Additional responses for reviewer RTmL\n\n**Re distribution shift**: See above. \n\n**Re scalability issues & how common our setup is**: We do not explore cases with more than four monitors because we think they are well served by existing literature on costly classification trees or costly classification RL (see lines 70-74 and 293-296). Again, existing monitoring systems are mostly using two monitors and clumsily combining them, and thus we think this paper addresses an important and timely problem. Systems using two monitors are already used for ChatGPT Agent, and we think they’ll only become more commonplace in the future."}]

Author rebuttal:

Meta-review: The paper examines the challenges of efficiently integrating $k$ monitors. Using all the available monitors would increase the quality, but it would also require additional memory and time.  Better policies are possible, especially if ones tolerate stochastic policies.

The problem is cast as a Neyman-Pearson classification, as monitors tend to have more false-positive errors. The submission demonstrated the technique using LLM for code generation. That’s both surprising and expected. Surprising because they could have used any other ML models, but it’s expected, as AI safety is particularly relevant when using LLMs.

The discussion was effective, leading to two reviewers increasing their ratings. As a result, all four reviewers rate the submission as borderline accept.

There is consensus on the quality of the technical work and the honesty of the contributions. The authors made it clear that their work supports up to a few monitors and requires a non-insignificant amount of data for calibrating the classifiers. The analysis of the regions is insightful.

Presenting it as a POMDP was not received well by a reviewer. I agree that this is not a general POMDP, but it’s a way to formally express the problem. It might be possible to use a probabilistic graphical model.

Personally, I find the story clear and that limitations can be studied in follow-up work.

Despite the consensus of borderline accept, I recommend acceptance.
Decision: Accept (poster)