Title: Combining Cost Constrained Runtime Monitors for AI Safety
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI Safety

Agents

Constrained Maximization
Abstract: Monitoring AIs at runtime can help us detect and stop harmful actions. In this paper, we study how to efficiently combine multiple runtime monitors into a single monitoring protocol. The protocol's objective is to maximize the probability of applying a safety intervention on misaligned outputs (i.e., maximize recall). Since running monitors and applying safety interventions are costly, the protocol also needs to adhere to an average-case budget constraint. Taking the monitors' performance and cost as given, we develop an algorithm to find the best protocol. The algorithm exhaustively searches over when and which monitors to call, and allocates safety interventions based on the Neyman-Pearson lemma. By focusing on likelihood ratios and strategically trading off spending on monitors against spending on interventions, we more than double our recall rate compared to a naive baseline in a code review setting. We also show that combining two monitors can Pareto dominate using either monitor alone. Our framework provides a principled methodology for combining existing monitors to detect undesirable behavior in cost-sensitive settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of optimising safety interventions in AI systems under cost constraints by combining runtime monitors. The authors formulate the problem as a partially observable Markov decision process (POMDP), where the goal is to learn a policy that decides when and which monitors to invoke, and when to audit outputs based on monitor scores, in order to maximise recall on misaligned outputs while respecting a fixed budget.\n\nThey evaluate their framework on BigCodeBench, focusing on detecting backdoored code samples. Results show that a stochastic monitoring policy, which may audit without always querying a monitor, achieves higher recall under the same cost compared to deterministic baselines that always call monitors. Furthermore, they demonstrate that a stochastic combination of two monitors can strictly outperform all single-monitor policies, including those that always invoke both monitors.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n=====\n* The paper introduces a mathematical formulation (as a POMDP) for combining runtime monitors under budget constraints.\n* It also shows that smart combinations of monitors can achieve higher recall at the same or lower cost.\n* The evaluation is done on a realistic setting: backdoor detection in AI-generated code (BigCodeBench).\n* The paper is generally well-written. There are small typos in places, e.g. use of \\PI_{TM} in Figure 5, but \\PI_{BM} in the text describing it.\n\nCons:\n=====\n* The paper's method relies on the distribution of misaligned examples being stable. This comes from the training data and is assumed to reflect the real-world distribution the system will face during deployment. However, in practice distributions shift, which may negatively impact the audit decisions.\n* Finding the optimal monitoring protocol requires enumerating all possible monitor calling strategies. This brute force search is intractable beyond 4 monitors, and the paper doesn't explore any approximate or scalable learning method.\n\nBoth my concerns are acknowledged by the authors under limitations. However, I'm not sure if this makes it better. They are still pretty serious limitations that make me doubt the practical applicability of the proposed technique.\n\nQUESTIONS:\nGiven the limitations of your approach (discussed under Cons above), how common are setups in which your approach is applicable and useful?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the budgeting problem of when to call a more expensive monitor when trying to detect AI misalignment. This paper proposes approach for determining this threshold based on some training data and a fixed budget. The proposed approach is empirically evaluated on a single task of detecting whether or not an AI system has output code with a backdoor or not. The authors show that there approach provides a Pareto improvement over just using a single monitor.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The paper presents a simple and easy-to-understand budgeting strategy for choosing when to call an additional more expensive monitor.\n* The result of the dip in log-likelihood where higher monitor scores do not necessarily correspond to higher precision (Figure 2a) is very interesting.\n* The results on this limited setup are fairly good and shows the benefit of this approach in a limited toy setting.\n\n### Weaknesses\n* The paper is not well motivated in the text. The paper somewhat assumes that the reader is familiar with prior AI control and monitoring work, and it just assumes that the importance of this problem is self-evident. Monitoring is obviously important for AI safety, but why is this problem important in particular? The text does not answer this question adequately. (I believe this problem is important - this is more about the writing.)\n* The experiments are quite limited. The approach is evaluated in one scenario on one dataset, with only two external monitors. Section 4 just shows that the large expensive monitor works fairly well at detecting backdoors in code (not even that great though). Section 6 shows how to trade-off two monitors.\n* A massive limitation acknowledged by the authors is that the learned threshold might not generalize to out-of-distribution data, but they make no effort to try this on more than one scenario.\n* The toy scenario is somewhat fabricated especially since o1 is not being told to obfuscate the backdoors it is creating. This is a very weak simulation of deception.\n* The proposed approach is made to seem much more profound and complicated than it is in actuality. For example, the authors refer to a learned threshold for choosing whether or not to call an additional more powerful monitor, and depending on the results, which outputs should be audited (essentially the most powerful monitor) a policy. They also describe this general enumeration of all possible monitor combinations, which they do not even do themselves in their experiments. Calling the choice of which monitors to call a POMDP is quite the stretch.\n* Furthermore, the final approach is not so sophisticated and seems like the first thing someone would try when given this problem: just learn a threshold for when the next most expensive monitor (auditing is just a very expensive and very accurate monitor) should be called based on some training data.\n\nQUESTIONS:\n* Why is there just a fixed budget and how would we know what this is in practice?\n* Adding Gaussian noise to the monitor scores seems like a terrible way to make them continuous. The more reasonable way to do this is to extract the logits for each of the digits from the output layer of the Llama model (either 8B or 70B) and compute a weighted expectation from that categorical distribution (over the tokens \"0\", \"1\", ..., \"9\") by applying softmax to the logits. \n* Why is this an important problem that warrants an entire paper about it?\n* Why are the two thresholds referred to as a policy? What are the states and actions?\n* Are there other problems where this is more important? Why choose backdoors in code as the simulation of deception? Is this based on the prior AI control work only?\n* Is the plan for this monitoring system to be deployed at test-time? If so, it seems like a fixed budget makes no sense. This is a long running system with many outputs. Or is the budget more of a suggestion or intended for a specific time window\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper develops a formal framework and practical protocol for optimally combining multiple AI runtime monitors under a unified cost constraint, using likelihood-ratio-based decision policies and exhaustive search to maximize recall for detecting misaligned model outputs while respecting budget limits; the approach is validated on code backdoor detection using Llama 3.1 70B/8B monitors and demonstrates strong empirical improvements over baseline strategies.

STRENGTHS AND WEAKNESSES:
Strengths:

+ this paper introduces , general method for cost-sensitive composition of multiple runtime monitors using rigorous likelihood ratio theory.

+ this paper provides a thorough empirical analysis and ablations showing that combining monitors yields superior safety-recall trade-offs compared to single-monitor or naïve approaches.

Weakness: 

+ The scalability of the proposed method is limited, as exhaustive policy search is only tractable for a small number of monitors and simple invocation strategies.

+ The method’s robustness under severe distribution shift or with more diverse/complex monitors is not fully evaluated.

+ Qualitative failure cases and breakdowns for different attack types are limited; more analysis would increase transparency.

QUESTIONS:
1. How would your approach scale to settings with more monitors or more complex invocation strategies (e.g., more than 4 monitors, hierarchical calls)?

2. Could you evaluate the performance of the proposed technique under strong adversarial distribution shifts or monitors with significant miscalibration?

3. Can you clarify the computational/latency overhead for the protocol in a real-time deployment scenario?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes devising a monitor invocation strategy that tries to optimize for cost and recall. The paper shows that using multiple monitors is better than just using one. The paper presents a mathematical framework to formalize the problem as a POMDP and decomposes the policy formulation into two steps: deciding when and which monitors to call, and then determining which outputs to flag. The approach exhaustively searches over a defined class of monitor invocation strategies and uses the Neyman-Pearson lemma. The approach is evaluated on the task of identifying potentially harmful code being generated (viz., code with backdoors) using BigCodeBench.\n\nSTRENGTHS AND WEAKNESSES:\n+ The paper is addressing an interesting and important problem. It is focusing on the problem of monitoring the outputs of an AI system to determine whether the output is aligned or not in order to decide whether to perform a safety intervention. \n\n+ The paper is generally well written. There is sufficient background provided regarding the problem setting and the solution. The experimental evaluation is clearly defined. \n\n- The approach is fairly straightforward using multiple monitors and using exhaustive search through possible monitor invocations utilizing the Neyman-Pearson lemma. It is not clear whether the technique will be able to detect more sophisticated backdoors than those generated by o1-mini. \n\n- The evaluation is limited. The specific task that is being used is to determine whether the code contains backdoors using BigCodeBench. An evaluation on more extensive tasks would make the paper stronger.\n\nQUESTIONS:\nHow could the approach be adapted for different scenarios? Are there any theoretical or practical limitations to using it for other use cases such as toxicity/bias detection?\n\nHow does using AI monitors to provide safety for AI lead to trustworthiness of the overall system?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}