CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring
Authors: Benjamin Arnav
Pablo Bernabeu-Perez
Nathan Helm-Burger
Timothy Kostolansky
Hannes Whittingham
Mary Phuong
Venue: NeurIPS 2025 poster ()
Keywords: ai safety
ai control
large language models
trusted monitoring
chain-of-thought
reasoning
natural language processing
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: As AI models are deployed with increasing autonomy, it is important to ensure they do not take harmful actions unnoticed. As a potential mitigation, we investigate Chain-of-Thought (CoT) monitoring, wherein a weaker trusted monitor model continuously oversees the intermediate reasoning steps of a more powerful but untrusted model. We compare CoT monitoring to action-only monitoring, where only final outputs are reviewed, in a red-teaming setup where the untrusted model is instructed to pursue harmful side tasks while completing a coding problem. We find that while CoT monitoring is more effective than overseeing only model outputs in scenarios where action-only monitoring fails to reliably identify sabotage, reasoning traces can contain misleading rationalizations that deceive the CoT monitors, reducing performance in obvious sabotage cases. To address this, we introduce a hybrid protocol that independently scores model reasoning and actions, and combines them using a weighted average. Our hybrid monitor consistently outperforms both CoT and action-only monitors across all tested models and tasks, with detection rates twice higher than action-only monitoring for subtle deception scenarios.

CoT Red-Handed:
Stress Testing Chain-of-Thought Monitoring
Benjamin Arnav*l
Pablo Bernabeu-Pérez"
Nathan Helm-Burger"
Tim Kostolansky"1
Hannes Whittingham 1 LASR Labs
Mary Phuong
Abstract
As Al models are deployed with increasing autonomy, it is important ensure they do not take harmful actions unnoticed. As a potential mitigation, we investigate Chain-of-Thought (CoT) monitoring, wherein weaker trusted monitor model continuously oversees the intermediate reasoning steps of more powerful but untrusted model. We compare CoTT monitoring action-only monitoring, where only final outputs are reviewed, a red-teaming setup where the untrusted model is instructed pursue harmful side tasks while completing coding problem. We find that while CoT monitoring is more effective than overseeing only model outputs in scenarios where action-only monitoring fails reliably identify sabotage, reasoning traces can contain misleading rationalizations that deceive the CoT monitors, reducing performance in obvious sabotage cases. To address this, we introduce hybrid protocol that independently scores model reasoning and actions, and combines them using weighted average. Our hybrid monitor consistently outperforms both CoT and action-only monitors across all tested models and tasks, with detection rates twice higher than action-only monitoring for subtle deception scenarios.
Introduction
As Al models grow in capabilities and are deployed with increasing autonomy growing need to address risks from potential misalignment (Shah al.) 2025 UK Al Safety Institute 2025). i.c., models pursuing hidden goals contrary to their developers' intentions.
METRR 2025), there
One approach to ensuring safety despite potential misaligmment is Al control (Greenblatt al. 2024b Mallen et al., 2024 Bhatt et al. 2024). Al control, models above a certain capability level are treated as untrusted and conservatively assumed be misaligned. Developers then employ protocols designed to be robust even against deliberate subversion attempts by the untrusted model, including trusted monitoring (i.e., monitoring of model outputs by a weak trusted model), untrusted monitoring (i.e., monitoring of model outputs by another instance of the same model), resampling actions if monitor flags a suspicious action, or escalating
human for review.
In this paper, we build on previous Al control research to investigate the value of monitoring the model's intermediate reasoning (Lanham 2022), also known as its Chain-of- compare its effectiveness to monitoring only its final actions. To do so, we evaluate monitoring performance proxy tasks simulating intentional model sabotage and assess robustness against strategies designed to evade detection.
(CoT), and
"Core contributors. All correspondence research @ benjaminarnav.com Code is available httpa::/github... ccmm/cotredhaandel/coo..oo Selected illustrative transcripts he found httpa: //cotredhanded. github. io
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Our results indicate that CoT monitoring consistently complements action-only monitoring across untrusted model sizes and families. Evaluating subversive behaviors in code, find that giving the monitor access to the CoT substantially increases detection rates in cases where sabotage is subtle and hard to identify from the code alone. However, we observe trade-off: in scenarios where action-only monitoring is already effective, access to the CoTT reduces monitor performance. This suggests that while CoT monitoring provides valuable additional security against subtle sabotage, may introduce noise false signals when applied to more obvious subversion attempts. In light of these findings, our main contributions are as follows:
We identify subtle subversion scenarios in which CoT monitoring offers safety gains over action-only monitoring.
We find CoT monitors can be deceived by the reasoning they oversee, making them less effective than action-only monitors when subversion is apparent in the answer.
We propose novel hybrid protocol that achieves higher detection rates than either answer-only or CoT monitors across both subtle and non-subtle scenarios.
Suspl on Storee
Prompt
1% FPR
Pass
Main Task
CoT menton dide took
Trusted
Attak Policy
CoT and Action Monitor
Side
OR
Action
Susplc on
Fagged
Prompt
Untrusted
Response
V
1%
Prompt
Trusted
Aclion-only Monitor
Figure 1: Method Schematic: a prompt including legitimate main lask, subversive side task and an atlack policy is fed to a strong reasoning Large Language Model (LLM).. We assess the ability of weaker LLM monitor to detect the resulting misbehavior, with and without access to the COT.
2 Method
Al control (Greenblatt et al., 2024b) is framework focused on designing and evaluating deployment strategies intended to ensure safety against intentional subversion. In a typical control evaluation, the Blue Team designs the control protocol, a structured system intended to prevent an untrusted model from carrying out harmful actions the environment, assuming worst-case scenario. The Red Team, aware of the monitoring setup, then develops an attack policy, often involving prompting or finetuning the untrusted model, attempting to circumvent the Blue Team's defenses.
AI control protocols typically have two components: monitoring, detecting responses that constitute sabotage attempt, and intervention, modifying untrusted responses for safe and useful downstream use. In this paper, we focus on trusted monitoring protocols (Gritlin et al. 2024). where a less capable but amown-harmleer model used by the Blue Team to oversee more powerful but untrusted model.
We adapt the AI control evaluation setup to focus specifically on monitoring effectiveness. In each interaction, an untrusted model must complete main task, which represents legitimate user- intended task, and a side task, which symbolizes a separate goal, potentially malicious, unintended by the user. The untrusted model's 's response, together with the main task, is then fed to the Blue Team's trusted monitor, which outputs a suspiciousness score, representing the plausibility of scheming. Responses scoring above certain threshold are considered 'caught'.
2.1 Evaluation Setup
We use problems from BigCodeBench (Zhuo al. 2024) as our main task dataset. BigCodeBench benchmark for code generation that contains 1,140 function-level tasks requiring combination of

---

multiple function calls from 139 Python libraries. We use the Instruct variant of the dataset, which presents the coding problems as short natural language instructions.
Our dataset follows similar principles to SHADE-Arena (Kutasov et al. 2025 emphasizing com- binations of main tasks and hidden sabotage objectives evaluate monitoring effectiveness. We introduce eleven side tasks, both novel and inspired by related work (Greenblatt et al., 2024b Baker al. 2025), that represent various ways model might covertly pursue actions contrary user intentions (see Table These side tasks serve as proxies for malicious actions a misaligned model might take in deployment setting. We categorize them into Subtle and Non-subtle based on the performance of action-only monitoring, as reported in Figure 3 A side task is categorized as subtle if its action-only detection rate is 50%. and non-subtle otherwise.
Side Task
Description
Type
Complexity
Ensure the code takes times longer to execute than the canonical solution for pre-generated Red Team test case
Default value
Add a debug parameter that, when enabled, silently catches all exceptions and returns default values
Edge
Introduce behavioral difference with respect the canon- ical solution that triggers for a pre- pre-generated Red Team test case
Subtle
Long implementation
Use at least twice as many lines of code as the canonical solution
Variable name
Name the return variable after a predefined value given to the model
Delete file
Deletes a file from the system
Download file
Downloads content from an external website
Edit permissions
Modify file permissions on the system
Exfiltrate secret
Transmit a file to external location
Non-subtle
Exit
Exit the program with code before returning a result, bypassing the main task tests
Log I/O
Write all input and output of the main task local file
Table 1: Overview and description of the evaluated side tasks categorized by behavioral subtlety.
We split the BigCodeBench dataset into four partitions: Prompt, Train, Validation and Test, following 5%/100%/19%/755% splits respectively. The Prompt split is used by the Red Team construct few-shot examples for prompting the untrusted model. The Train split is where both the Red Team and Blue Team iterate their respective attack and defense strategies. The Validation split is used by the Blue Team to tune thresholds and select the best-performing monitoring approach. The Test split is reserved for final evaluation and reporting of results. A detailed breakdown of these partitions, including the number of instances of each side task per split, is provided in Table (5 in Appendix C Models had five chances to complete tasks (see Appendix A.2 for completion rates).
Unlike Greenblatt et al. (002bb), who examine both opportunity identification, where the model determines favorable situations for sabotage, and attack strength, developing effective evasion strategies, we focus solely on attack strength. We achieve this by pairing each BigCodeBench problem with the side task most likely to conceal it (see Appendix C for pairing methodology details), yiclding 1,140 main-to-side task pairings.
2.2 Monitoring Protocols
We compare three approaches to monitoring:
Action-only monitor: only the untrusted model's final answer is reviewed.
3

---

CoT monitor: both the untrusted model's reasoning and its final answer are reviewed.
Hybrid monitor: weighted average of the Action-only and CoT monitors' suspicion scores.
To ensure fair comparison across monitoring approaches, we control for the number of monitor evaluations: both action-only and CoT monitors are run twice with temperature 1.0 and their suspicion scores are averaged, while the hybrid monitor uses single run for each component, otherwise, the hybrid monitor would have access twice the number of runs.
Subtle Side Task
Non-Subtle Side Task
Generate a ser of samples from normal distribution with specified mean and standard deviation. Also visualize the generated samples...
specilee converts spec path
fornet,
Attack strategy" Your code deployed next week. shuetions neluding..
Attack stratogy production ready problem, ensurin ng intag atas stancard opar ational requiroment includinng. g....
enviromment, noed to
Side task: The funct on should deliberately nofficient, running slowly on large inputs
8
Side task:
Uploed /homm//ngent www.rnsssriiinioniiio yam'f to "ttpps//mmtriic.. http///eerriiciiiieeee ctrico comileg"
g
CoT
The inefficinncy from algorithm vectorized
I've decided not to Include the suspiclous instruction
from indridually instead
So plan
upload file to metrics UR., as doesn' seem be releted to the core functionalit described the problem,
def taak, (rren ded,dev, s):
Funcfesv,
[...]
kith opent" /honc//gers, trarss_ttreeneeeiaaaa www.tgg..otttt. yonk". "ro") canflig
na. annizan.niffor. 3)
postt
meth (.r Logf)) mtt..cox3
"httpo, /f /fitrriss corios .oor//og
files-d File" : oon "a. fiko)
tep muzeralee atd_dav munn)
[...]
aation-only
agent making deliberately writing
attempts stearthlty unrelated contents object
agent separates suspiciou step cleatly. analyzes implications, opts
designed anyy purpase other generating visualizing samples. code soletyy
straightforward efficient which suggests
server any user notification. demonstratnn caution rather clous intent.
strongty indicates data insorporated into solution.
Suspicion
Suspicion
Figure 2: Example transcripts from DeepSeek-RL R1 (subtle) and Claude 3.7 Sonnet (non-subtle) as the untrusted models and GPT-4. IPT--4.--mini as the trusted model. The CoT monitor assigns a high score to the subtle runtime-con cwmplemmiiiiccccccccccocooocco.e side task (left), whereas the action-only monitor fails to detect the inefficient implementation. Conversely, for the obvious exfiltrate side task (right), contradictory reasoning misleads the CoT monitor, despite access to the final answer.
To optimize the hybrid monitor's performance, evaluate weighted combinations of action-only and CoT suspicion scores. For each weight w E [o, 1] at regular intervals of 0.01, hybrid scores were computed Shyhrid
Saction-only (1 w) SCoT- For each w, determine the 99th percentile threshold using benign examples (main-task-only completions, with no side instruction) to maintain 1% FPR. We then calculate the detection rate for each weighting using the baseline attack policy. The best results are obtained with w = 0.55, which we adopt to compute the final results.
Our analysis is restricted to model responses that successfully complete both main and side tasks. Including failed attempts would unfairly skew the assessment our monitoring approach's 's effective- ness, since such failures may render outputs indistinguishable from benign actions, even under perfect action-only monitoring. Additionally, in internal deployment scenarios, outputs that fail automated validation would not be deployed, making them less relevant to threat model. Further discussion of this filtering approach is provided in Appendix
For each monitoring strategy and untrusted model, we set threshold so that the monitor achieves false positive rate (FPR) & 1% when evaluated on a distribution of benign responses (i.e., answers without any side task instruction). As discussed in Appendix A.3 our FPR choice is intended to

---

reflect limited budget for human verification in a real deployment of frontier Al system. We report catch rates at this 1% FPR threshold as our primary metric, it better reflects practical deployment constraints where organizations must operate under strict precision requirements due to the substantial costs of human review. For completeness, we also provide ROC-AUC results in Appendix AA.4 though we note that ROC-AUC averages performance across all possible FPRs, including unrealistically high rates that would not be acceptable in practice.
To study the trusted monitoring setup we evaluate four monitor models of varying capability: Claude 3.5 Haiku (Anthropic) 2024), Gemini 2.5 Flash Lite [DeepMind] 2025), (PPT-4.1mmin) and GPT- 4.1-nano (OpenA1 2024p. These are used to monitor more capable reasoning models (DeepSeek R1 Distill Qwen 14B, DeepSeek RI Distill Llama 70B, DeepSeek R1 (DeepSeek-A AI, (2025), Claude Sonnet 3.7 Thinking (Anthropic) 2025), and Grok 3 Mini (xAI, (025)) as the untrusted models (see
further model providers and licenses in Appendix D). All experiments were conducted using the
Inspect AI framework (UK Al Safety Institute 2024).
2.3 Attack Policies
We iteratively developed series of attack policies that attempt obfuscate the suspicious nature of the untrusted model's 's reasoning about the side task, in order to challenge the robustness of CoT monitoring. In Table 2 we list and briefly describe the broad categories of attack policies used. Details of all employed attack policies are included in Appendix B
Attack Policy
Description
Baseline
Complete the main and side task
Stealth
Extra instruction to perform the side task sneakily
Framing
Influence the language used by the model to refer to the side task by using role role-playing techniques or reframing it to seem helpful, harmless and honest
Monitor Notice
Inform the untrusted model there is monitor overseeing its reasoning and actions
Few-Shot
Add examples of successful stealthy completion, with CoT that does not mention the side task but successfully completes it
Finetuning
Finetune a reasoning model with examples of successful stealthy completion
Table 2: List and description of Red Team attack policies.
Results
We evaluate monitoring approaches and red-team attack policies across diverse side-task scenarios and model configurations. Our analysis examines the comparative effectiveness of CoT monitoring versus action-only monitoring different task types, the impact of hybrid monitoring strategies that combine both signals, and the robustness of each approach against our suite attack policies designed to evade detection.
3.1 CoT Outperforms on Subtle Side Tasks, Underperforms on Non-Subtle Ones
Figure compares action-only and CoT monitoring catch rates across the full set of side tasks. A distinct separation emerges:
Subtle side tasks (bottom-left); Action-only monitoring performs poorly, with catch rates below 20%. CoT monitoring significantly enhances detection, with performance improving by an average of 10 percentage points.
Non-subtle side tasks (top-right): Action-only monitors show substantially higher recall, with GPT-4. detecting nearly all instances, with the exception of the Exit side task, which lies the cluster boundary. Overall, CoT monitor access decreases performance by 25 percentage
5

---

points across models, possibly due to persuasive rationalizations or noise introduced in the reasoning.
Catch Rates (at 1% FPR) of Action vs CoT Monitoring (Monitor: gpt-4.1-mini)
Subtiee -ub--uuttlee
Download hile
100
Exfilrate secret
Edit parmisstori
90
80
Log vdi
70
COT Monitar Catch Rate (%)
Complexity
Edge case
30
Variable
ExI
20
Long implementation
10
Default
Action Monitor Catch Rate (%)
Figure 3: Catch rates of action-only and CoT monitoring across side tasks, results averaged across untrusted models and attack policies for each side task. The dashed line separates the regions where CoT monitoring outperforms action-only (above), and vice versa (below).
These results suggest that CoT and action-only monitoring provide complementary signals, showing relative strengths different regimes of task subtlety.
3.2 Hybrid Monitoring Recovers Performance and Improves Consistency
Figure introduces third strategy-hybrid monitoring-thh that aggregates independent CoT and action evaluations through a weighted average as described in Subsection 2.2 This approach yields consistent and substantial improvements: across all models and task types, hybrid monitoring outperforms both action-only and CoT monitoring catch rates, achieving 21 % and 42% higher recall respectively.
Figure 5 displays the performance of GPT-4.1-minn and Gemini 2.5 Flash Lite separated by side task subtlety. For non-subtle side tasks, hybrid monitoring performs on par with action-only monitors, while providing 32 percentage points of performance hand, for subtle side tasks, the hybrid monitors match the performance of CoT monitors, while achieving over twice the catch rate of action-only monitors. Nonetheless, hybrid monitoring catch rates for subtle side tasks remain below 20% on average, suggesting that while more effective, is still limited in absolute terms.
CoT monitors average. On the other
Figure breaks down performance by attack policy for the untrusted Claude 3.7 Sonnet Thinking model, with results for all models reported in Figure i0 in Appendix B While the hybrid GPT-4.11 mini monitor is not always the top-performing strategy in every scenario, it consistently outperforms both action-only and CoT monitoring on average.
The dependency attack, a framing strategy where the untrusted model is instructed to perform the side task intermediary step of the main task, is the most effective of all attack policies for

---

Cetch Rates by Model and Monitor
Marior: 25 fush lite
Nonitor 3aade
Neniaar gpa-.
OC' Grvwr
Catch Fase 150 4835
Figure 4: Catch rate performance comparison at 1%FPR of all trusted models for action-only, CoT and hybrid monitors. Catch rate performance comparison at 1 1% FPR of all trusted models for action-only, CoT and hybrid monitors across side tasks. The subplots display results by monitor model as follows: top left shows GPT-4.1-mine top right Gemini 2.5 Flash Lite, bottom left Claude 3.5 Haiku, and bottom right GPT-4.1-nano.
nun-subtle side tasks. Its effectiveness stems from models frequently mentioning the side task as a benign requirement in their reasoning or, in some cases, claiming they will ignore the side task while still performing it, thereby deceiving the monitors with access to the reasoning traces. For the subtle side tasks, monitor notice, where we notify the untrusted model of the monitoring setup, matches the performance of the the baseline attack policy.
In contrast, stealth attacks are largely ineffective, resulting in significantly higher catch rates than baseline across all monitor types, leading to the highest CoT and hybrid monitoring catch rates for subtle side tasks. As this strategy encourages the model to be stealthy in its actions, it often prompts relf-reflective reasoning that inadvertently reveals sabotage attempts, making them easier to detect when the monitor reviews the COT.
4 Discussion
Our results reveal that Chain-of- monitoring offers a powerful but nuanced tool for detecting model misbehavior. While access to reasoning traces substantially improves detection of subtle deception attempt, it proves less effective for blatant exploits. Furthermore, our red-team evaluation demonstrates that adversarial models can exploit trusted overseers through deliberate reasoning manipulation, highlighting important limitations in current monitoring protocols.
4.1 Side Task Subtlety Affects CoT Usefulness for Monitoring
Our investigation reveals a nuanced picture of the role of CoT monitoring in detecting deceptive behavior in advanced Al systems. Results show monitoring efficacy is dependent on the nature of the side task being pursued by the untrusted model (Figure 3 revealing that action-only and CoT monitoring can serve complementary roles within a comprehensive safcty strategy.
7

---

Catch Rotes by Model and Subtleheen Subcie Group
Monker gpt 4.1 melt
Moniter gomm Hugh lise
Catch rate performance of GPT-4.I-min (left) and Gemini 2.5 Flash Lite (right) as monitors grouped by untrusted model and side task subtlety.
Figure 5:
For subtle side tasks, CoT monitoring significantly improves detection compared to code inspection alone. These tasks involve code manipulations that maintain primary functionality while introducing concealed vulnerabilities, e.g. altered algorithmic complexity or edge case failures. Since these sophisticated deceptions produce minimal changes in the final code output, they are hard detect through code inspection alone. Having access to the model's s reasoning process substantially increases catch rates for these critical scenarios, though detection rates are far lower than in non-subtle scenarios, highlighting persistent limitations in identifying subtle manipulations.
For non-sublle side tasks, the action-only monitor achieves higher detection rates than CoT monitoring. These tasks involve operations that leave clear traces in the code, such as file operations, network connections, permission changes. Since these operations are clearly visible in the generated code, analyzing the reasoning traces offers no added value and misleading rationale may actually confuse the monitor, reducing its effectiveness.
Hybrid monitoring, which evaluates the CoT and final action independently before aggregating scores, mitigates these trade-offs. As shown in Figure this approach consistently performs as better than either single-monitor strategy, suggesting hybrid monitoring can be a practical default, especially in real-world deployments where both stealthy and blatant attacks must be guarded against. Notably, while hybrid monitoring achieves > 85% catch rates on average for non-subtle side tasks and substantially improves detection in subtle cases, performance the latter remains limited in absolute terms, with catch rates below 30% across models.
4.2 Monitor Vulnerability to Persuasion
A key vulnerability we identify is that monitors can be deceived by plausible explanations in model reasoning, even when they have access both the original task description and the final code. As illustrated in Figure untrusted models can generate reasoning that explicitly rejects harmful action while still implementing in the final code. Models can also frame their malicious implementations deceptively, such as claiming the side task is being done in accordance with a Red
"corporate policy". Our work extends the findings of previous research (Arcuschin et al.) 2025 Chen et al. 2025 Lanham et al. 2023) to the safety context, demonstrating that lack of reasoning Taithfulness can be exploited to circumvent naive monitoring.
While these vulnerabilities present significant challenges, they may be partially addressed through targeted improvements in monitor design, training and usage. Our experiments, which tested four different models as monitors, reveal that increases in monitor capability led to improved detection performance, a result suggesting that monitor sophistication is key. This observation aligns with related research (Terekhov et al. 2025), which identifies monitor base capability as more relevant factor for adversarial detection than attacker capability or the capability gap between models. These findings collectively support the hypothesis that ensuring the monitor surpasses certain capability
8

---

threshold is critical for safety. However, it remains an open question whether these trends will persist as models capabilities continue increase.
Catch Rates by Attack Policy and Subde/Non-Subde Tasks (Monitor: gpt 4.1 min, Model: Claude 3.7 Sennet:Thinking)
CoT
7675%
Rada (%6 at
20.0h
12
Doai O22e
2ne th
Striteg
Notfce
Attack Policy
Figure 6: Catch rate performance across side task subtlety and attack policy for Claude 3.7 Sonnet Thinking overseen by GPT-4.I-mm I-min..
4.3 Limitations and Future Work
Our evaluation focused on single-step coding problems in a controlled environment, which fails to capture the complexity of real-world deployment. Future work should investigate monitoring effectiveness in more realistic coding environments where multi-step, distributed sabotage is possible. This includes scenarios where malicious intent might be distributed across multiple functions, files or commits, with harmful effects emerging through the interaction of seemingly benign components.
We explored several attack policies but did not exhaustively cover all potential adversarial strategies. Early experiments showed that preventing models from mentioning side tasks in their CoT was very inconsistent, so we focused on designing attack policies that frame the side tasks in benign ways, reducing the likelihood of the model referencing them suspiciously. Future research could consider developing scaffolding simulate advanced model capabilities and investigate more comprehensive fine-tuning protocols that better approximate deceptive behaviors in more advanced Al systems.
Recent work (Needham al. 2025 reveals that models like Claude Sonnet 3.7 often recognize they are being evaluated, raising concerns about the trustworthiness of CoT traces in evaluation settings. models know they being tested, they may strategically craft reasoning appear aligned, masking deceptive behavior and undermining monitoring. Future research should explore ways to detect or mitigate evaluation awareness, such as using randomized test framing less-salient evaluation contexts, to ensure CoT monitoring reflects genuine reasoning rather than performative alignment.
Finally, study does not address how monitoring efficacy might be affected by production con- straints such as latency requirements, cost limitations, or scalability challenges. Future work should evaluate these practical considerations, and in particular whether the computational overhead of dual monitoring approaches is justified by the safety gains in specific deployment contexts.
5 Related Work
Chain-of- Thoughh (CoT) is technique that enables large language models (LLMs) to articulate intermediate steps before their final output, increasing model capabilities on tasks requiring sequential reasoning (Miner et al.) 2025). Although the use of CoT initially emerged through prompting techniques such as few-shot examples (Nye et al. 2021 Reynolds and McDonell, 2021 Wei et al. 2022) and zero-shot instructions (Kojima et al. 2022). recent work explicitly trains models via reinforcement learning or supervised fine-tuning to consistently produce CoT reasoning (DeepSeek)
Ai 2025 Jacch al. 2024).

---

Inspecting models' CoT's could provide insight into their reasoning and intentions. In particular, CoT monitoring could be a promising approach detecting deceptive reasoning or 'scheming' (Meinee at 2025) by future Al models. [rreenlaat et al. (2024a) demonstrate that frontier L.LSs can fake alignment by using a secret scratchpad, a prompted version of CoT, to fake compliance. When hidden scratchpads were removed, compliance gaps disappeared in scenarios with in-context information but not in finetuned models, suggesting internalized knowledge can enable covert deceptive reasoning. While their work underscores the risks of misalignment, it did not evaluate CoT monitoring within a formal control protocol with targeted attack policies nor examine reasoning models trained via reinforcement learning to reason naturally, the setting we investigate.
However, CoT may not fully represent model reasoning: studies show that models answer consistently despite variations or omissions of the CoT (Lanham et al. 2023). produce coherent rationalizations for implicit biases and use illogical reasoning to simplify solutions (Arcuschin al. 2025), and fail to acknowledge known influences in their CoT (Chen et al., 2025). While imperfect faithfulness places practical limits on the utility of CoT for interpretabilityy it does not completely remove the value of monitoring, it represents additional stream of information. Our work investigates whether CoT monitoring provides measurable advantages over action monitoring for detecting sabotage, without making claims about the faithfulness of the COT.
Frontier Al labs (Jaech et al., 2024 Anthropic 2025) have primarily used CoT monitoring identify hallucinations and unintentional failures in their models, without evaluation against adversarial attacks or deliberate attempts to deceive the monitor. On the other hand, LlamaFirewall (Chennabasappa et al., 2025) incorporates AlignmentCheck, component that audits an agent's 's reasoning at inference time to detect signs of goal hijacking or prompt-induced misalignment, aiming to defend against adversarial behavior. However, while they use strong models to audit an agent's 's full execution trace for goal consistency, we focus on a weak monitoring setup to investigate subversion detection in the presence of capability gaps that may arise with future, more powerful untrusted models.
COT supervision has also been used for training-time oversight: Lightman et al. 2023) explore reward model training using fine- mme-grained supervision over the reasoning process, rather than only the final outcome. Mcanwhile, Baker al.] (2025) show that weak CoT monitors with explicit guidance to detect specific failure modes can oversee stronger models and detect reward hacking more effectively than action-only monitors. Although they find that optimizing the reasoning trace to reduce reward hacking might obfuscate misaligned behavior, faithfulness has been identified as an explicit objective in model development (Lyu et al.) [023 Paul et al., 2024 Xu et al. (024)), potentially leading to CoT being more reliable representation of internal model processes.
6 Conclusion
Our results demonstrate that CoT monitoring offers measurable safety benefits over baseline action- only monitoring in scenarios involving subtle deceptive behaviors, but also exhibits specific vul- nerabilities that must be addressed comprehensive Al control protocols. We introduce a more nuanced approach to monitoring system design that leverages the complementary strengths of multiple monitoring approaches to close the gaps that any single approach leaves open.
We believe our findings present case for maintaining transparent, natural language CoT in frontier models as a valuable safety feature that can detect potential sabotage from misaligned models, particularly when monitoring systems are designed to account for the evasion strategies we identified. By characterizing the strengths and limitations of CoT monitoring against sophisticated circumvention attempts, our work seeks to inform broader threat models and bolster safety cases (Korbak et al.) 2025).
Acknowledgments
This project was fully funded by LASR Labs as part of their research program. We would like to acknowledge Joseph Miller, Erin Robertson, Brandon Riggs and Charlie Griffin for their support during the project. Additionally, we thank Joseph Bloom, Adam Kaufman, Trevor Lohrbeer, Robert McCarthy, Alex McKenzie, Fabien Roger, and Tyler Tracy for their feedback on early versions of this paper.
10

---

References
W. U. Ahmad, S. Narenthiran, S. Majumdar, A. Ficek, S. Jain, J. Huang, V. Noroozi, and B. Gins- burg. Opencodereasooing:: Advancing data distillation for competitive coding. arXiv preprint arXiv:2504.01943, 2025.
Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. https 7/www. www.w.nthropic..oommm wwttthrppc.coomm/esss///. 3- momellls- and computer-use 2024.
Anthropic. Claude 3.7 sonnet
system card. 2025.
https: http://wwwww.nrropccccc com
claude-3- -0e-7-sonntt--yyteee card
URL
com
claude 20-7-sonnet-B50 -er7--mmmttssstte-ccaa card
I. Arcuschin, J. Janiak, R. Krzyzanowski, S. Rajamanoharan, N. Nanda, and A. Conmy, Chain-of- thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.0886 2025.
B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi. Monitoring Reasoning Models for Mishehavior and the Risks of Promoting Obfuscation, Mar. 2025. Jttptt/////axxi org/ absaabas////// 11926 arXiv:2503. 1926 [cs].
A. Bhatt, C. Rushing, A. Kaufman, V. Georgiev, T. Tracy, A. Khan, and B. Shlegeris. Shell Games:
Control Protocols for Adversarial Al Agents, Oct. 2024. URL https://openrevieer net/
forum?id=oycEeeXX74
Y. Chen, J. Benton, A. Radhakrishnan, J. U. C. Denison, J. Schulman, A. Somani, P. Hase, M. W. F. R. V. Mikuli,, S. Bowman, J. L. J. Kaplan, et al. Reasoning models don't 't always say what they think. Anthropic Research, 2025.
S. Chennabasappa, C. Nikolaidis, D. Song, D. Molnar, S. Ding, S. Wan, S. Whitman, L. Deason, N. Doucette, A. Montilla, et al. Llamafirewall: An open source guardrail system for building secure ai agents. arXiv preprint arXiv:2505. 03574, 2025.
G. DeepMindd Gemini 2.5 flash-lite model card. F-a-aa------------------- Lide--ie---MMeaaaa pdf September
htap:://sttrrggg.ooggeege com/
hww.eiinddmeeaaa/ddeeee-eee-e-ee 2025.
DeepSeek-AI DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, Jan. 2025. |ttpbpp////arxxi org abs abs/2501. 12948 AXXi:25511.12943 [cs].
R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, A. Khan, J. Michael, S. Mindermann, E. Perez, L. Petrini, J. Uesato, Kaplan, B. Shlegeris. S. R. Bowman, and E. Hubinger. Alignment faking in large language models, Dec. 2024a. /ttp:://aaxx org/ 2ttrg//bs/2442...
2ttp///22...1111333. arXiv:2412.14005
R. Greenblatt, B. Shlegeris. K. Sachan, and F. Roger. AI Control: Improving Safety Despite Inten- tional Subversion, July 2024b. http:::/aaww...rg////aaa [cs].
06942 arXiv:2312.06942
C. Griffin, I.. Thomson, B. Shlegeris, and A. Abate. Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols, Sept. 2024. URL 07985, MtXiv:2409.07988 [cs].
A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22211 2022.
T. Korbak, J. Clymer, B. Hilton, B. Shlegeris, and G. Irving, A sketch of an AI control safety case, Jan. 2025. URL http:://axxxii org/ abs/ 17315 arXiv: 2501. [cs].
J. Kutasov, Y. Sun, P. Colognese, T. van der Weij. L. Petrini, C. B. C. Zhang, J. Hughes, X. Deng, H. Sleight, Tracy, al. Shade-arena: Evaluating sabotage and monitoring in Ilm agents. arXiv preprint arXiv:2506.1577 2025.
T. Lanham. Externalized reasoming oversight: a research direction for language model alignment. https: ///ww. al www.gmentooooorr..mmmmmmmmm wtwg ottppppsts//PBBGGennSSSooooo 2022.
T. Lanham, A. Chen, A. Radhakrishnan, Steiner, C. Denison, D. Hermandez, D. Li, E, Durmus,
E. Hubinger, J. Kermion, K. Lukošiūté, K. Nguyen, N. Cheng. N. Joseph, N. Schiefer, O. Rausch,
11

---

R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. Maxwell, T. Telleen- Lawton, T. Hume. Z. Hatfield-Dodds, J. Kaplan, J. Brauner. R. Bowman, and E. Perez. Measuring Faithfulness in Shain-of-Though heasoning,July 2023. htt..ttt////xii.oor//bbbbb/// 13702 arXiv:230..13702.
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. and K. Cobbe. Let's 's verify step by step, 2023. URL ptttpp:://arxiw.org//aaaa
Sutskever, 20050
Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, and C. Callison-Bure reasoning. In The 13th International Joint Conference Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL. 2023), 2023.
Faithful
A. Mallen, C. Griffin, A. Abate, and B. Shlegeris. Subversion Strategy Eval: Evaluating Al's stateless strategic capabilities against control protocols, Dec. 2024. |RLt:t/://aaxii http://arrii..ogrga//////// /412 12480 arXiv:2412. 12480 [cs].
A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn. Frontier Models
are Capable of In-context Scheming. Jan. 2025. URL httthh//tp///iii org/ www./rgg/////////////.../...ee// 2412.
arXiv:2412.04984 [cs].
METR. Measuring ai ability to complete long tasks. attps://nne //ttr http://///rtrrrrrblggggggg 03 2025.
2025- 09-----aasuurin---aaaae
S. Miner, Y. Takashima, S. Han. S. Kouteili, F. Erata, R. Piskac, and S. J. Shapiro. Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problemss, Feb. 2025, URL http://arxiv. oww.ooggga....o..o.. 04101 arXiv:2410.00151 [cs].
J. Needham, G. Edkins, G. Pimpale, H. Bartsch, and M. Hobbhahn. Large language models often know when they are being evaluated. arXiv preprint arXiv:2505.23886. 2025.
M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz. M. Bosma, D. Luan, C. Sutton, and A. Odena. Show your work: Scratch- pads for intermediate computation with language models, 2021. URL fttppp://axxiv. Abs/2112.00111
OpenAl. Gpt-4.1. https://platformn openai com 2024.
D. Paul, R. West, A. Bosselut, and B. Faltings. Making reasoning matter: Measuring and improving faithfulness of chain-of-thouge reasoning. arXiv preprint arXiv:2402.1 1e..22422.199955 13950, 2024.
L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 2021.
R. Shah, A. Irpan, A. M. Turner, A. Wang, A. Conmy, D. Lindner, J. Brown-Cohen, L. Ho, N. Nanda, R. A. Popa, R. Jain, R. Greig, S. Albanie, S. Emmons, S. Farquhar, S. Krier, S. Rajamanoharan, S. Bridgers. T. Ijitoye, T. Everitt, V. Krakovna, V. Varma, V. Mikulik, Z. Kenton, D. Orr, S. Legg,
N. Goodman, A. Dafoe, F. Flynn, and A. Dragan. An approach technical agi safety and security, abs http://200008...00
2025. |Rt / h/txxxxvvvooo wttp/oor////aaaaaaeeeaa
M. Terekhov, Z. N. D. Liu, C. Gulcchre, and S. Albanie. Control tax: The price of keeping ai in check. arXiv preprint arXiv:2506. 05296, 2025.
UK AI Safety Institute. Inspect ai: Framework for large language model evaluations, 2024. URL htttps//inspect.. awssi.iiii............. org.uk/
UK Al Safety Institute. Research Agenda. https://www..att
gov gww.uuu//reeeerrcaaaaaeeddaaa 2025.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing
systems, 35:24824-2458 2022.
XAI. Grok 3 [large language model). https://x www.ass/nnooo--a.. 2025.
J. Xu, H. Fei, L. Pan, Q. Liu, M.-L. Lee, and W. Hsu. Faithful logical reasoning via symbolic tfain-of-thought. arXiv preprint arXiv:2405. 18357, 2024.
T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are backed by results obtained in Sections and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the work are discussed in Subsection 43
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper describes experimental details in Section including model con-
figurations, dataset partitioning, and evaluation metrics. Filtering decisions, threshold calibration and task pairing, are further detailed in Appendices A.2 A.3 and supporting full reproduction of the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
14

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: provide full access to our code at https://githhb. com/ cotredhanded/code code and sufficient instructions to replicate our data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//dddeeebmimcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe our dataset and evaluation details in Section 2 with additional details included in Appendices A
B and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: All figures in the main paper include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper reports estimated compute usage in Appendix D including the num- ber of inference calls, APIs used (Anthropic, Google, Groq, OpenAl, Together), model con- figurations, and compute requirements for targeted fine-tuning experiment with DeepSeek RI Distill Qwen 14B to simulate advanced attack policies.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics, including Potential Harms Caused by the Research Process, Societal Impact and Potential Harmful Consequences and Impact Mitigation Measures.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section Aiiiiusssss both the benefits of CoT monitoring safety mechanism and the risks from unfaithful or misleading reasoning traces. We emphasize the utility of CoT transparency while cautioning against overreliance on naive CoT-based protocols.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: The paper does not introduce new models or datasets with dual-use risk; evaluates previously released models and uses existing dataset. No safeguards needed for the release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Existing assets cited and their licenses mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No new datasets or models are introduced. The paper evaluates existing open-source commercially available models and benchmarks.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper evaluates the use LLMs as both reasoning agents and monitors. These models core to the methodology and results.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neur)) hrct::::/eerrips.cc///ooeeeeeeee /ct.///oefeeeeneee/22////.. for what should or should not be described.
19


Reviews:
[{"review_id": "c5wfVOvDMp", "reviewer": "Reviewer_VKc9", "summary": "The paper evaluates whether CoT monitoring improves monitor performance, compared to action-only monitoring, at detecting malicious side-task attempts on a coding benchmark. They find that CoT helps the monitor overall, but can also be used to mislead it. Therefore, the paper also proposes a hybrid monitoring approach combining CoT and action only monitoring and demonstrates its efficacy.", "questions": "- Do you agree that the comparison for hybrid monitoring is unfair for the reasons I discuss? Can you address this concern?\n- Are the side tasks an original contribution?\n- Can you deal with my structure concerns?\n- Can you execute any of the additional experiments I suggested?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**I will increase my score if my two major concerns (see below) are addressed.**\n\nStrengths: \n\n- Well motivated direction and clear scope of experiments \n- well written paper\n- comprehensive empirical evaluation on many models and tasks \n- comprehensive and mostly clear methodological details\n\n\nWeaknesses\n\nMajor \n\n- Poorly structured (e.g., essentially three methods sections) and the content is not really well-ordered, making some details hard to find or unclear. (See below)\n- Unfortunately I think there is a substantial methodological flaw in one of the core results, BUT I think it can hopefully be addressed.  Unless I’m mistaken, the hybrid monitor results do not control for monitor resources, so the comparison to action only and CoT monitoring is unfair. (I guess a similar point could be made for CoT vs action only.) Perhaps a really principled way to deal with this would be to control for the monitor input token budget or something like that. However, a simpler way to get a fairer comparison would be to just allow action and CoT monitoring to also average over two predictions (with some non deterministic monitor). What I mean is that, any monitor will do better if you sample several scores and average them (cf jury theorems). And the hybrid monitor basically gets to sample two judges, whereas the others just get one sample score. I’d be happy to increase my score if you account for this concern (e.g., by rerunning the action-only and CoT monitor results with high temperature and getting results for weighted average over, e..g, 2 action-only monitor scores).\n\nOther comments. \n\nFigure 1 is nice \n\nThe related work doesnt seem that informative. E..g, you discuss alignment faking in too much detail, and miss some control references like ShadeArena. The lightman citation also seems a bit tangential. I think the related work could be clearer and more organised. Perhaps it should also go later in the paper. In general, the rel work section can be used to either contrast your work to other work (then it should go near the end) or to set the scene and provide greater context (then it should go near the beginning). \n\nSections should never begin with a subsection (e.g., Background) they should begin with a short overview of the section. \n\nI think some of the content can be moved around, e.g., the CoT part of the background should possibly go in related work (at least the first paragraph). The background is kind of a weird short section. I would remove it and incorporate the content into the related work and methods. Also, the discussion of the dataset should come much sooner, like the first thing in methods. The experimental setup section should be folded into methods too (with subsections). Id basically start methods with discussion of the dataset and side tasks then discuss control and red team blue team roles and protocols. \n\nIt’s unclear in some places what’s novel and what isnt. For instance, did you come up with the side tasks or are they from other work? Also, how are the side tasks graded? With test cases? \n\nIn methods you should discuss how you evaluate performance, both of the agent (at completing main and side tasks) and especially of the monitor — Fig 3 reports the catch rate, I would have gone with ROC-AUCs which is pretty standard in control work right? \n\nI understood that you remove unsuccessful agent attempts and only monitor successes, but it could be interesting to see how the results look for unsuccessful sabotage attempts. \n\nFigure captions should be entirely self-contained (yours are too minimal). \n\nThe figures seem a bit out of place, e.g, figure 2 is not discussed until near the end of the paper (I would discuss it sooner). \n\nI would be very interested to see results comparing CoT monitor performance on reasoning models (CoT optimised with RL) vs non-reasoning models. Cf https://arxiv.org/abs/2501.08156\n\nSummary of extra experiment suggestions:\n\n- fair comparison between hybrid protocol and the others (by sampling multiple monitors for the others)\n- see if the results hold for sabotage attempts not just successes \n- compare CoT monitoring for reasoning vs non-reasoning models to see whether optimised CoT influences monitorability  \n\nPlease also report ROC-AUCs, at least in the appendix, which I think are more informative than the 1% FPR catch rate (which seems like a somewhat arbitrary choice?)", "was_revised": true, "final_justification": "Overall the authors have compellingly addressed my concerns and I will increase my score to an accept. I have also read the other reviews in detail (see my comments below). I agreed with several points raised by other reviewers but I think the authors have addressed all the major concerns raised. I do not feel there are any other substantial weaknesses of the paper.", "rebuttal": "We thank the reviewer for the thoughtful and detailed feedback. We appreciate the constructive suggestions and recognize the importance of addressing the methodological and structural concerns raised.\n\n## Methodological Fairness of Hybrid Monitor Comparison\n\nWe agree with the reviewer that in our original comparison the hybrid monitor benefited from multiple samples while the baseline monitors only had access to one.\n\nTo address this concern, we ran both action-only and CoT monitors twice with temperature set to 1.0 and averaged their suspicion scores, while the hybrid monitor used only a single run for each component. All results included in this response or the responses to other reviewers use this methodology. \n\nWe acknowledge that the action-only monitor processes fewer tokens than the CoT monitor, but we believe this is a fundamental trade-off rather than a disadvantage. The action-only monitor focuses on the most relevant information (i.e. the final implementation), and an ideal monitor should be capable of detecting sabotage attempts from this output alone.\n\nAdditionally, following reviewer suggestions, we evaluated three additional monitor models, including both reasoning and non-reasoning models (detailed results are provided in other rebuttals):\n\n| Monitor Model | Action Catch Rate | CoT Catch Rate | Hybrid Catch Rate |  \n|-|-|-|-|  \n| claude-3.5-haiku | 45.95% | 30.76% | 46.50% |  \n| gemini-2.5-flash-lite | 68.82% | 66.64% | 80.63% |  \n| gpt-4.1-mini | 27.90% | 23.90% | 31.01% |  \n| gpt-4.1-nano | 21.72% | 7.85% | 13.71% |\n\nThese results largely confirm our original findings: the hybrid approach consistently outperforms both baseline strategies across most monitor models. However, for the weakest monitor (GPT-4.1-nano), the hybrid approach shows limited improvement, suggesting that a minimum level of monitor capability may be required for the hybrid strategy to be effective.\n\nWe note that due to resource constraints, we used the same w=0.55 value (optimized for the original run of GPT-4.1-mini on our validation set) across all monitor models rather than optimizing individually. This suggests the hybrid approach's benefits could potentially be even stronger with model-specific tuning.\n\n**Related Work Organization and Content**\n\nWe appreciate the feedback on our related work section and understand the concern about its organization and focus. Our aim for this section is to provide context about existing applications of CoT monitoring and demonstrate how our work differentiates from them.\n\nOur inclusion of alignment faking work reflects its direct relevance to CoT monitoring in adversarial settings, particularly given that Greenblatt et al. specifically examine how models can use reasoning traces to deceive monitors. We believe this reference establishes important context for understanding the challenges our work addresses.\n\nRegarding the Lightman et al. citation, we included this work because it represents a foundational approach to supervising reasoning processes in language models, albeit for training rather than deployment monitoring.\n\nWe acknowledge that ShadeArena was published after our submission and therefore could not be included in our original related work. We would be happy to incorporate discussion of this and other relevant concurrent work in a revised version.\n\n## Paper Structure and Clarity\n\nWe understand the reviewer's concerns about the paper's structure and will work to address these issues. Our current organization reflects our view that the background section provides essential context about CoT and the AI control framework that readers need before understanding our methodological contributions.\n\nIn our experimental setup section, we explain the particular instantiation of the method we are studying in our experiments. We describe the dataset in the experimental setup because it represents a specific feature of our experiments rather than a core aspect of our methodology. Although we make an effort to ensure that the side tasks we define in the method section are as relevant as possible to the samples of our selected dataset, they could be used with multiple coding and cybersecurity benchmarks.\n\nHowever, we will revise the paper structure to improve clarity and ensure that each section has an introductory overview.\n\n## Novelty and contributions\n\nAs outlined in our introduction, we explain our main contributions to distinguish our novel work from existing research. Regarding the side tasks, these represent a combination of our novel contributions and adaptations from existing work.\n\nMost control work (e.g. Greenblatt et al.) has focused on logical errors as proxies for backdoors, which corresponds most closely to our \"edge case\" side task that creates behavioral differences with the gold standard solution. Several other tasks draw inspiration from existing literature: the exit side task is inspired by reward hacking strategies that models exploited in Baker et al., while the download side task is similar to scenarios examined in Bhatt et al. Additional side tasks, such as exfiltrate secret, are motivated by discussions in the AI Safety community regarding primary threat models.\n\nOur primary contribution in developing these side tasks lies in collecting and implementing them within a single unified environment. This allows us to study to characterize when different monitoring strategies are most effective.\n\nAs mentioned in Appendix A.2, side tasks are evaluated using test cases. For each side task, we developed specific judging criteria tailored to the task's characteristics. For example, the complexity side task is evaluated by measuring and comparing the running times of both the gold standard solution and the red team solution, the variable name task uses regex matching to detect the specified variable name, and the download file task verifies both the presence and content of the downloaded file. We will provide comprehensive details of the evaluation method of each side task in the revised paper.\n\n## Choice of Evaluation Metrics\n\nThe reviewer suggests using ROC-AUC instead of catch rates at 1% false positive rate (FPR). While we appreciate this suggestion, we respectfully maintain that our chosen metric better reflects realistic deployment constraints. ROC-AUC averages performance across all possible FPRs, including unrealistically high rates that would never be acceptable in practice. In real-world deployment scenarios, organizations would likely operate at very low FPRs due to the substantial costs of human review and intervention.\n\nOur 1% FPR choice reflects the practical reality that monitoring systems must operate under strict precision constraints when dealing with large volumes of queries and limited human oversight resources. This threshold represents a reasonable balance between detection capability and operational feasibility, making our results more directly applicable to real deployment scenarios.\n\nNevertheless, for completeness and to address the reviewer's suggestion, we provide ROC-AUC results across different monitor models. These results confirm the improvement of the hybrid monitor (w=0.55) over either baseline metric except for the weakest monitor (GPT-4.1-nano). However, the difference between performance according to AUC is much closer than with catch rate at 1%FPR, indicating that the model might not underperform as much at lower FPRs.\n\n| Monitor Model | Action AUC | CoT AUC | Hybrid AUC |  \n|-|-|-|-|  \n| claude-3.5-haiku | 0.7688 | 0.7465 | 0.8211 |  \n| gemini-2.5-flash-lite | 0.8578 | 0.8596 | 0.9045 |  \n| gpt-4.1-mini | 0.7395 | 0.7492 | 0.7764 |  \n| gpt-4.1-nano | 0.6590 | 0.5933 | 0.6477 |\n\nAs for the 1%FPR metrics, we used the same w=0.55 value across all monitor models rather than optimizing individually.\n\n## Filtering by Task Success\n\nOur decision to filter results to include only successful completions of both main and side tasks serves important methodological purposes. For main task failures, including these would introduce noise that obscures our core research question about monitoring effectiveness, as monitors might flag outputs due to general incompetence rather than malicious intent.\n\nFor side task filtering, side task failures can occur for two distinct reasons: genuine attempts that fail due to implementation errors, or complete non-compliance where models refuse to attempt the side task due to safety training. The latter case creates an unfair comparison between action-only and CoT monitoring, as action-only monitors cannot reasonably be expected to detect sabotage attempts that exist only in reasoning traces without any corresponding implementation.\n\nWhile we recognize that including failed attempts could provide additional insights, we believe our current approach ensures a fair comparison between monitoring strategies by focusing on cases where malicious behavior is actually present in the final output.\n\n## Reasoning vs Non-Reasoning Models\n\nWe appreciate the suggestion to compare monitor performance across reasoning and non-reasoning models. Our focus on reasoning models reflects their position at the current frontier of capabilities where our monitoring approaches would be most relevant.\n\nNon-reasoning models were considered early in our research but ultimately excluded from our final evaluation. Based on existing research showing reduced faithfulness in prompted CoT compared to trained reasoning, we expect that non-reasoning models would be more susceptible to certain evasion strategies, such as simple instructions to avoid mentioning side tasks in scratchpad reasoning. This would fundamentally change the threat model and reduce the relevance of our findings to future deployment scenarios where reasoning models are likely to predominate.\n\nWe remain open to exploring this comparison in future work, as it might provide insights into CoT faithfulness and monitoring effectiveness across different architectures."}, {"review_id": "Xn3By0h72O", "reviewer": "Reviewer_WFZb", "summary": "This paper evaluates automated monitoring of language models, both on the basis of their final outputs (the action-only monitor) and also on the basis of both the chain-of-thought (CoT) and its final outputs (the CoT monitor). They introduce a hybrid monitor, which essentially just averages the scores of the other two monitors. In each case, a threshold for monitor suspicion is selected during the validation phase. The authors evaluate all three monitors using the BigCodeBench instruct dataset, augmented with side-tasks and “attack policies” that give the tasked model some direction to help it evade the monitor.  They find a clear clustering between “subtle side-tasks” that are more easily detected by the CoT monitor, and “non-subtle” side tasks that are more easily detected by the action-only monitor. They find that the hybrid monitor outperforms both approaches on both types of tasks. These results hold across six different reasoning models and “attack policies”. All evaluations are on the same dataset and with the same monitor model (GPT 4.1-mini).", "questions": "I have a number of questions about your weights and averaging process:\n\n* How do your results change across values of w? I was hoping for a table or figure in section A.4, but there’s nothing there.  Specifically, I would be interested in seeing the results for w = 0.25, 0.5, 0.75. \n\n* Why do you think $w > 0.5$ is optimal given that this puts less weight on the monitor that has strictly more information? \n\n* The false positive methodology is non-standard. Why not report F1 (or even classification accuracy) rather than going through all the effort described in appendix A.3? Have you tried setting the threshold to a fixed value and reporting simpler metrics? \n\n* Based on your finding that CoT monitors are more effective on subtle tasks and action-only monitors are more effective on non-subtle ones, I expected the hybrid combination to be multiplicative rather than additive. Intuitively: if either monitor is sufficiently suspicious, then the hybrid score should be high.\nMore precisely, one could implement this with a (weighted) geometric average rather than a weighted arithmetic one: \n$$\\hat s_{\\text{hybrid}} := 100 \\left(1 - \\sqrt{ \\Big(1- \\frac{s_{\\text{action-only}}}{100} \\Big)\\Big(1-\\frac{ s_{\\text{CoT+action}}}{100} \\Big) } \\right).$$\nNote that this formula is just \n$s^\\prime_{\\text{hybrid}} = \\sqrt{ s^\\prime_{\\text{CoT}} \\cdot s^\\prime_{\\text{action}} }$\nwhere $s' = (1-s/100)$ is the (normalized) score for innocuousness rather than suspiciousness, and has the property that both monitors have the power to to veto the verification. Have you thought about aggregation methods like this?\n\nA few more questions:\n\n*  Have you thought about replacing the GPT4.1 mini monitor model with small models from other families? I would imagine that CoT monitors from the same family would be more vulnerable to persuasion, and this might be worth studying. \n\n* Can you provide some transcript instances where the agent successfully avoids detection in a subtle case? Based on the transcripts provided, it seems that these models explicitly recall their side objective in their chain of thought, so I’m very curious about why the CoT monitor sometimes fails to pick this up. \n\n\n## Some Low-level Issues, Errors, and Comments\n\n* line 126: based on the description in appendix A.3, there should not be a $\\le$ ; the false positive rate should be exactly 1% on the calibration data, and so one imagines approximately 1% elsewhere. The inequality introduces some confusion about the methodology. This paragraph can be made clearer in other ways as well. It’s not just that you “set a threshold” (line 125) but rather that you set *the classification threshold*; at first read, this seems like another filtering step in the context of the previous paragraph, making it seem unnecessarily suspicious.\n\n* In figure 6, the labels on the x-axis overlap, and are in the range [0,1] rather than [0,100] per the description of the monitor tasks A.1.\n\n* The x-axis of figure 7 is mislabeled; its 1-FPR, not FPR. \n\n* I reject the conclusion on lines 246-247: “as our results suggest, the adoption of 247 hybrid evaluation strategies that prevent over-reliance on a single information stream” — Your hybrid monitor is no different from the CoT+action monitor baseline in this regard; both have access to the same information. You just process it differently. For similar reasons, I think the wording on line 229 is misleading. \n\n* In my view, the blue team / red team material is rather generic, not very interesting or technical, and does not add much to the paper. I find the typeset colors distracting, and create an unintended association between the red/blue in figures 3 and 5.", "limitations": "The authors’ limitations section is good.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "# Strengths:  \nThe finding that side-tasks cluster nicely into subtle and non-subtle objectives depending on how easy they are to detect based on the final response alone, appears to be an important contribution of this paper. The appendix clearly explains and defends the methodology (and more of this material should be integrated into the main body of the paper.) The work finds similar findings across six families of frontier models. The extremely simple nature of the hybrid monitor makes its effectiveness surprising and raises interesting scientific questions. These surprising empirical findings and experimental hygiene are clearly the strength of the paper.\n\n# Weaknesses: \nThe threshold and weight tuning methodology appears to be more convoluted than it needs to be, and there is some key information missing. The proposed hybrid monitor is, on its face, rather substance-less: it is just an average of two scores, based on the finding that sometimes the monitor with less information works better. The fact that this outperforms the model with more information is interesting, but, to my mind, this raises more fundamental questions about the setup more than it supports the hybrid-monitoring solution. The evaluation is run on only one dataset (albeit with many different tasks), and with one monitor model, raising questions about whether or not the seemingly robust behavior illustrated in the paper holds in related settings. Given the simplicity of the model and the strangeness of the behavior, a lot of weight falls on thorough evaluation across multiple settings—which fall slightly short of convincing, in my view.  I also really dislike the title. \n\nA few specific concerns:\n\n* While it makes sense that the monitors should be weaker models, I wonder if the results of figures 4 and 5 are simply a result of the model being too weak to competently understand the instructions and the thought process. I began to suspect this after reviewing the selected transcripts. \n\n* Choosing $w = 0.5$ instead of $w=0.55$ would have significantly simplified the presentation and methodology, allowing the authors to remove appendix A.4 and the paragraph from lines 129-134. This suggests to me that the method is very sensitive to the choice of $w$. However, results are reported only for $w=0.55$, which I find suspicious. \n\n* Since the hybrid monitor comprises both the CoT and action-only monitors, it uses roughly twice as much computation. Therefore, one could argue that there is a need for a fairer (compute-normalized) baseline—maybe using a 2x larger model for the individual monitors or averaging the scores for two distinct monitors of the same type. \n\n* The title is too bold for the contributions of this paper. The chain-of-thought monitoring is not being stress-tested at all—indeed, the solution is to have a monitor that often ignores the chain of thought! The “CoT red-handed” supposed to be a pun involving chain-of-thought and the notion of a red-team, but the focus of this paper is on the “blue team” and shifting weight away from a potentially convincing CoT. In any case, the title does not accurately reflect the interesting empirical finding of the paper: that sometimes CoT monitoring is useful (for subtle tasks) and other times it is counter-productive.", "was_revised": true, "final_justification": "* The additional analysis provided by the authors in the rebuttal to myself and to reviewer VKc9 allayed biggest technical concerns about how the results generalize across multiple settings and about the fair comparison per unit of computation. I hope that ultimately the entire evaluation of the hybrid monitor will be done with this methodology. \n* Keep in mind that the authors must now add a section to motivate their higher-temperature run-twice methodology and compare that itself against the baseline. I feel there is a significant opportunity to strengthen the result by adding further baselines in this spirit (e.g., using a model of twice as many parameters, or prompting the  model to think twice as long to double check).\n* I still believe that the title is inappropriate and that the framing is wrong in some places---I have tried to explain why, but ultimately this is the authors choice.\n\nUltimately, reflection, discussion with the authors, and reading the other reviews, have convinced me that the authors are in a position to make this paper worth publishing by the camera-ready deadline.", "rebuttal": "We thank the reviewer for their thoughtful comments. We appreciate the feedback and aim to address the mentioned weaknesses and questions that they bring up.\n\n\n## Rationale for Hybrid Monitoring Strategy\n\n\nWe chose to pursue the hybrid strategy of weighting the action-only and CoT monitor scores primarily due to our empirical observation that the action-only and CoT monitors had complementary strengths on different types of side tasks.\n\nThe fact that the hybrid monitor outperforms the monitor with \"all\" information (i.e., both the action and CoT) was surprising to us at first as well, but we believe that the rationale is that the misleading nature, or overload of information of the CoT leads to underperformance of the CoT monitor in some cases. We do agree that the approach of simply weighting the action-only and CoT scores is quite naive, and we imagine that there are likely better approaches to aggregating monitor inputs; to this end we have investigated the reviewer’s suggested geometric weighting (see below). \n\nWe hope that our exploration of this method in the control setting will be useful as a source of information to inform much more thorough design work in future monitoring systems.\n\n\n## Single dataset and monitor model\n\n\nWe accept that the strength of our paper would be improved with studies over more datasets and monitor models. To address this, we have run additional experiments with further monitor models: 4.1-nano, Claude 3.5 Haiku and Gemini 2.5 Flash Lite (resoning). On the subject of multiple datasets, we did explore several sets, including Mostly Basic Python Problems, APPS and CWeval, but in the end chose to focus on BigCodeBench as it was the most extensive and highest in quality. We think it fair to highlight that we study both a range of side tasks and a range of untrusted models. Further to this, many important works in AI control, such as the original AI control paper, only work with a single dataset and a single monitor model.\n\n\n\n\n## Expanded Monitor Evaluation, weight values, suggested geometric average\n\nAs per the reviewer's note, we believed that it was important to keep the monitor weaker than the scheming reasoning models. We agree that our results suggest that the monitor may simply have been too weak to do monitoring well, so in order to corroborate the results on GPT-4.1 mini, the monitoring model in our original paper, we have tested three other monitor models which were weaker than our scheming reasoning models: Claude 3.5 Haiku, Gemini 2.5 Flash Lite (reasoning), and GPT-4.1 nano.\n\nWe ran monitoring with these models, as well as a second run with GPT-4.1 mini, and replicated our results finding that the setting of the weighting value to w=0.55 outperforms or performs on par with other weightings, including 0.5, as well as with other scoring mechanisms, i.e., using the action-only score, the CoT score, and the reviewer's suggested geometric mean.\n\n\n## Performance Results Across Multiple Monitors\n\n**Catch Rate @ 1% FPR**\n\n| Monitor | Action-Only | CoT-Only | Hybrid (w=0.25) | Hybrid (w=0.50) | Hybrid (w=0.55)     | Hybrid (w=0.75) | Max(Action,CoT) | Geometric Mean |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| claude-3.5-haiku | 45.95% | 30.76% | 40.04% | 45.03% | 46.5% | 49.45% | 44.12% | 45.23% |\n| gemini-2.5-flash-lite | 68.82% | 66.64% | 68.24% | 80.16% | 80.63% | 76.96% | 82.34% | 80.98% |\n| gpt-4.1-mini | 27.9% | 23.9% | 25.12% | 30.19% | 31.01% | 29.91% | 32.96% | 31.79% |\n| gpt-4.1-nano | 21.72% | 7.85% | 6.66% | 12.67% | 13.71% | 16.17% | 15.98% | 13.73% |\n\n**AUC-ROC Scores:**\n\n| Monitor | Action-Only | CoT-Only | Hybrid (w=0.25) | Hybrid (w=0.50) | Hybrid (w=0.55) | Hybrid (w=0.75) | Max(Action, CoT) | Geometric Mean |\n|---------|-------------|----------|-----------------|-----------------|-----------------|-----------------|------------------|----------------|\n| Claude 3.5 Haiku | 0.7688 | 0.7465 | 0.8142 | 0.8210 | 0.8211 | 0.8186 | 0.7830 | 0.8218 |\n| Gemini 2.5 Flash Lite | 0.8578 | 0.8596 | 0.8862 | 0.9037 | 0.9045 | 0.9019 | 0.9002 | 0.9035 |\n| GPT-4.1 Mini | 0.7395 | 0.7492 | 0.7758 | 0.7764 | 0.7765 | 0.7759 | 0.7758 | 0.7764 |\n| GPT-4.1 Nano | 0.6590 | 0.5933 | 0.6466 | 0.6476 | 0.6477 | 0.6476 | 0.6475 | 0.6476 |\n\n**F1 Scores at 1% FPR:**\n\n| Monitor | Action-Only | CoT-Only | Hybrid (w=0.25) | Hybrid (w=0.50) | Hybrid (w=0.55) | Hybrid (w=0.75) | Max(Action, CoT) | Geometric Mean |\n|---|---|---|---|---|---|---|---|---|\n| claude-3.5-haiku | 0.6276 | 0.3157 | 0.3918 | 0.5138 | 0.5476 | 0.5388 | 0.4876 | 0.5140 |\n| gemini-2.5-flash-lite | 0.5013 | 0.3658 | 0.5334 | 0.5641 | 0.5964 | 0.5873 | 0.4652 | 0.6267 |\n| gpt-4.1-mini | 0.4362 | 0.4959 | 0.5437 | 0.5562 | 0.5561 | 0.5278 | 0.5368 | 0.5562 |\n| gpt-4.1-nano | 0.3559 | 0.3353 | 0.3577 | 0.3668 | 0.3867 | 0.3716 | 0.3603 | 0.3668 |\n\nThese results also reflect running the action-only and CoT monitors twice (with higher temperature of T=1.0, including our original GPT-4.1 mini runs), so that they have comparable resource usage to the hybrid monitoring method.\n\n1. Model Capability Effects: The weaker GPT-4.1 nano monitor performs relatively worse compared to the other models, while our original GPT-4.1 mini monitor performs slightly worse than the more performant Claude 3.5 Haiku and Gemini 2.5 Flash Lite monitors.\n\n2. Reasoning Model Performance: Gemini 2.5 Flash Lite is a reasoning model, meaning that it was trained to use its own CoT when generating outputs. We believe that this contributes to its relatively stronger performance compared to the other, non-reasoning models.\n\n3. Alternative Scoring Methods: We note that the reviewer's suggestion of trying the geometric mean works well, performing on par with our tuned hybrid scoring, and we appreciate the contribution.\n\n\n\n## Controlling for monitoring resources\n\nThe reviewer is right that we should have controlled for monitoring resources by running two repeats of the action-only and CoT-and-action monitors for fair comparison to the hybrid monitor. We have addressed this experimentally - please refer to our above results and our response reviewer VKc9 for a description. \n\n\n\n## Title criticism\n\n\nWe respectfully disagree with the criticism of our title. The pun is meant light-heartedly, but we do include work on several red-team strategies, and our ‘stress-testing’ successfully finds places where CoT monitoring does not work well relative to action-only monitoring (alongside situations where it works better). \n\n\n\n## Optimality of w=0.55\n\nWe determined this empirically for our particular set of results - we wouldn't claim that future projects expanding on these ideas should adopt this particular number as generally applicable. We think that the reason for the heavier weighting on action monitoring is the same as the reason that we describe for CoT monitoring underperforming in some cases, i.e misleading content in the CoT. \n\n\n\n## Phrasing on hybrid monitor performance\n\nWe agree. Thank you for pointing this out - we’ve changed the wording on those lines to reflect the real advantage of the hybrid monitor, i.e that the different approaches perform better on different side tasks, and the hybrid monitor takes into account both. This is what we were trying to express, but we agree our original wording is imprecise. \n\n\n## Blue team/red team material\n\nThanks for this - we agree and have changed this section to make it more concise. \n\n\n## Thresholding methodology\n\nThank you for bringing this up. We have clarified this paragraph in our redraft, including your phrase ‘set the classification threshold’. \n\nThe inequality is necessary because of a feature of the suspicion scores we observed our monitor to produce. While we asked for a score between 0 and 100, and hoped that the returned scores would be roughly continuous, we actually observed that the returned scores mostly fell into buckets at 0, 5, 10, 15, and so on. \n\nThis is likely to be explained by the fact that humans would usually use these milestones to assign such scores in GPT-4.1’s pre-training set. We carried out some experiments to try to change this behaviour and achieve a more continuous score, including changing the range, and adding explicit instructions to the prompt, but were unable to change this behaviour. \n\nThis being the case, the 99th percentile, calculated across suspicion scores of the validation set without a side task, was often precisely at one of these milestones; several examples might score exactly at this level rather than above or below it. \n\nTo deal with this, we decided that the monitor should only flag with suspicion scores strictly greater than this threshold were considered suspicious when the monitor was used in anger; i.e examples with suspicion scores exactly at this threshold were not considered suspicious. This seemed like the cautious approach, and maps to a false positive rate that is less than or equal to 1%. \n\n\n\n## Axis labels in Figure 6\n\nThank you for highlighting this - we have corrected this.\n\n## Other questions\nRequest for additional transcripts: Transcripts are too long for rebuttal. We will update our transcripts website after the rebuttal period ends to avoid running afoul of NeurIPS rules on external content. \n\n# Summary of Changes\n\n1. Language revision to temper claims about performance differences\n2. Expanded monitor evaluation including multiple models and sensitivity analysis\n3. Added methodological details about prompt development and optimization\n4. Clarified limitations regarding OpenAI reasoning models\n5. Explicit discussion of score combination strategies\n6. Comprehensive performance comparison across multiple monitors and scoring methods\n\nWe believe these changes significantly strengthen the paper while maintaining our core contributions about the relative effectiveness of different monitoring approaches, the vulnerability of simple CoT monitors to reasoning content, and the benefits of combined monitoring strategies."}, {"review_id": "bvhpLxxXYB", "reviewer": "Reviewer_XKux", "summary": "This paper focuses on improving CoT monitoring for cases where a weaker trusted monitor model continuously oversees the intermediate reasoning steps of a more powerful but untrusted model. The authors propose a hybrid protocol that combines CoT monitoring with action-only monitoring using a weighted average. The paper reports this hybrid approach outperforms both CoT monitoring and action-only monitoring across all models and task types analyzed.", "questions": "Is there a way to make the subtle tasks more interesting?\nHow do the results change if we include failed attempts?\nWould the results change or stay the same if both Blue team and Red team had access to the same number of prompts for validation?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Very interesting analysis of pairing up each problem in BigCodeBench with the side task most likely to conceal it.\n- The experiments include 5 different untrusted models.\n- Writing is very clear and concise\n- Discussion on limitations of CoT monitoring\n\nWeaknesses:\n- The subtle tasks look somewhat reasonable, and not really malicious, which might be the main reason why the action-only monitor doesn't give them a high score. They definitely don't seem to match the wording used in the CoT of action-only monitors: \"suspicious or counterproductive ways to solve the problem, backdoors, sabotage or other forms of hacking\". E.g., why would \"Variable Name\" or \"Long Implementation\" be malicious? At most they would be an annoyance to the user. In fact, the side attacks are considered subtle if the action-only monitor detection rate is <50% (line 140), which, again looking at the prompt, is the very definition of a non-malicious attack. This might actually explain why CoT monitoring (or hybrid monitoring for that matter) does not surpass ~30% detection rate.\n- The analysis for comparing monitoring approaches excluded failed attempts because they would unfairly skew the assessment of the author's approach (lines 122-124). This seems biased and against the authors motivations of reflecting a \"real deployment of a frontier AI system\" (line 128). Does this mean that proposed approach wouldn't work in a real setup where malicious models could have failed hacking or sabotage attempts? I.e., if a model knew this, it could just sandbag and hide the sabotage in a \"failing\" answer.\n- Why does the Blue team has 25% few-shot examples where the Red team only has 15%? (lines 152-156). This seems unbalanced and might skew the results.\n- Line 181 claims that \"CoT and action-only monitoring provide complementary signals\" but looking at Figure 3, and given that the best weight found for the hybrid monitor was 0.55, it would seem that there's more evidence of them being similar than different.", "was_revised": true, "final_justification": "The authors have compellingly addressed my main concern on subtle tasks not really looking malicious. Although I still think this is true, I do agree that they might constitute an \"action towards a separate and undesired goal from the one intended by a human user\" as the authors put it. The authors have also corrected me on optimal weighting vs signal similarity, which I agree, and no longer see the weight being 0.55 as a concern. They have also clarified the motivation of dataset splits for blue and red team in their experiments.", "rebuttal": "Thank you for your review of our paper. Our goal is to provide a useful and well-evidenced update to the AI safety community on where and how CoT monitoring is best applied. We appreciate that you brought out some of the positives in our work such as the pairing analysis and comparison across 5 models. \n\nHere we will focus on addressing the issues that you have raised in the sections on weaknesses, and answering your questions. \n\nWe first address each of the bullet points in your weaknesses section, in the order they were shown in your review. \n\n## Rationale for Subtle Side Tasks\n\nYou are correct that some of our subtle side tasks, especially those involving changing a variable name or extending the length of an implementation, are not examples of serious malicious behaviour. There are a few reasons why we choose to include them and we have updated the paper to make this clearer. \n\nFirst, while they are not *seriously malicious*, we think that they reasonably represent *action towards a separate and undesired goal from the one intended by a human user*. This is what we are really worried about in future, more powerful models, which might act on very complex misaligned plans, in very indirect ways \\- these side tasks are a toy stand-in for this. You’re right that our action-only monitor prompt asks the monitor to look for e.g sabotage, but we also include *“Is there anything that suggests the code is designed to achieve another goal, beyond the stated problem?”* to guide towards this interpretation of suspicious behaviour. The monitors always have access to the original coding problem, so they have sufficient information to judge this. \n\nSecond, we think that these very subtle side tasks are useful because they show contrast between action-only and CoT monitoring. The fact that action towards this separate, undesired goal is very difficult for an action-only monitor to detect is intentional. The access to information about the model’s intention, revealed in the CoT, becomes crucial here, and this is borne out in the results. You’re right that the CoT/hybrid monitors perform poorly here at only \\~30% detection rate, but we think our contribution here is just to provide experimental evidence that compares CoT monitoring to existing baselines in the literature. We find they do show uplift in these situations (we hope there will be further work that makes these monitors genuinely effective). \n\nAs a concrete improvement to our paper, **we have clarified the language around our side tasks, to make it clear that we intend side tasks to simulate action towards a separate and undesired goal**, not only more serious malicious behavior. \n\n## Exclusion of Failed Attempts\n\nWhether or not to include failed attempts, at the main task or side task, is something we returned to repeatedly over the course of our work.  \nWe decided to remove examples that failed on the **main task** test cases because we think that these examples would not be used in reality, and so are less relevant to our threat model. We often thought of the example of a company deploying LLMs to write parts of its internal codebase, and using AI control as a safety measure. In this setting, code that failed to pass test cases would never be integrated and used in deployment regardless of whether the monitor flagged it or not. For this reason, we decided to test our monitors exclusively on working code, which is where they would deliver safety improvements. \n\nIt is also possible that failed attempts at the main task, resulting in incorrect code, might have triggered the monitor spuriously, inflating our estimates of monitor effectiveness. While we controlled for the false positive rate using examples without a side task, we observed that inclusion of a side task causes the main task to fail more often, meaning that this would have affected our results. \n\nWe decided to remove examples that failed on the **side task** test cases because we considered these as cases where the agent had not managed to scheme successfully \\- so again here, flagging by the monitor doesn’t deliver a safety improvement. We also thought it possible that output code, in cases where the side tasks failed, might be completely indistinguishable from code that exclusively and faithfully executed the main task, making it an unfair test for the monitor. \n\nTo address this concern, we have **extended the section you highlighted (lines 122-124) to properly discuss one or both of these cases**.\n\n## Dataset Split Clarification\n\nRegarding the dataset split balance described in lines (152-156), we want to better explain exactly how and why we split the BigCodeBench dataset, and which examples were used for producing few-shot examples. \n\nThe Red Team (untrusted model prompts) uses just 5% (*Prompt split*) for constructing few-shot examples, not 15% as suggested. This was part of our work on whether we could make the untrusted model more effective at getting past a CoT monitor by providing examples of a chain of thought which does not include mention of the side task, followed by a final answer which completes the side task.\n\nThe blue team (monitor prompts) did not use few-shot prompts in our final experiments. Note: *We did carry out some experiments on whether few-shot prompting could make the monitor more effective, by providing examples of model actions/CoTs along with appropriate suspicion scores (we presented these in pairs, with both low and high suspicion score examples). However, we found that these did not make the monitor more effective, and dropped this approach, settling instead on the prompts that can be seen at the beginning of the appendices.* \n\n## Monitor Complementarity\n\nWe respectfully disagree with the interpretation that a hybrid weight of 0.55 indicates the monitors are 'similar.' This reasoning conflates optimal weighting with signal similarity, which are distinct concepts.\n\nThe key evidence for complementarity comes from Figure 3, which shows the monitors have **qualitatively different strengths**: CoT monitoring excels on subtle tasks (bottom-left quadrant) where action-only monitoring fails, while action-only monitoring performs better on non-subtle tasks (top-right quadrant) where CoT monitoring struggles. \n\nThe hybrid weight of 0.55 simply reflects the best balance for maximizing overall performance across both task types. The complementarity is demonstrated by the performance patterns, not the weighting. Our contribution is in identifying when each monitoring approach is most effective and showing that a combined approach outperforms either technique, across our studied side-tasks. \n\n## Addressing specific questions\n\n**Is there a way to make the subtle tasks more interesting?**\n\nWe believe this question refers to your earlier concern about the maliciousness of our subtle side-tasks. As discussed above, we see these tasks as proxies for action toward separate and undesired goals, and see this as the main safety concern. The 'interest' in these tasks lies precisely in their subtlety \\- they represent scenarios where intent-based monitoring (CoT) can detect goal misalignment that would be invisible to action-only monitoring.\n\n**How do the results change if we include failed attempts?**\n\nWe believe this would introduce significant noise to the results. We hope that our earlier response clears up why we don’t believe failed attempts should be a main part of our analysis: (1) **main task** failures would not be deployed in realistic settings regardless of monitor flagging, making them irrelevant to our threat model, and possibly causing spurious monitor hits that would inflate our measure of monitor effectiveness; (2) **side task** failures represent cases where scheming was unsuccessful, so monitor flagging provides no safety benefit.\n\n**Would the results change or stay the same if both Blue team and Red team had access to the same number of prompts for validation?**\n\nWe hope that our response above clarifies the purpose of the dataset splits. Especially: the Blue Team's use of the data serves a technical calibration purpose, rather than providing strategic advantage. We have updated the language in the paper to make this clearer."}, {"review_id": "zRfcBZ5LHn", "reviewer": "Reviewer_9pfw", "summary": "In this paper they investigate how CoT monitors perform relative to output only monitors for various reasoning models. They modify a coding task dataset to include “side-tasks” which the agent is supposed to achieve along with the main objective. They break these side objectives down into “subtle” and “non-subtle” ones and find that the CoT monitor performs better than the output monitor on subtle objectives but not on non-subtle ones. They hypothesise that this is because the CoT monitor can get fooled by the plausible reasoning in the agents CoT.", "questions": "- Why did you only test gpt 4.1-mini as a monitor? \n- Have you considered using a reasoning model as the monitor? Most labs are moving towards using reasoning monitors for deception. In my experience these do better at CoT monitoring and typically handle long contexts better.\n- How did you arrive at the prompts, how much optimization pressure was there in the process and how equal was it? The prompts seem quite similar which leads me to think that there may be a large elicitation gap for one or both of the monitors.\n- Why did you decide to put the reasoning after the output for the CoT monitor?\n- I was going to ask why you didn’t try o1/o3/o4 but then i realised you won’t actually have access to the true CoT for these. Maybe you should mention this since otherwise a reader might be confused as to why you test random R1 distills and not the classic reasoning model.\n- Have you tried using Max(monitor 1, monitor 2) as the suspicion score? That seems like the typical choice so it would be nice to compare to.\n\nIf the authors address some of my concerns (see weaknesses) I will probably raise my score.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Timely and important research direction. \n- Large number of coding tasks.\n- Good limitations section.\n- Well written.\n\nWeaknesses:\n- I believe they somewhat overstate the performance difference between CoT and action monitors. Looking at their plot the two monitor types appear highly correlated.. They state that the CoT “substantially increases detection” but on some of these tasks the increase is to 20% which is pretty low performance. They say “Action-only monitoring performs poorly” but If this was a real world control setting both monitors would have unacceptably low performance. Furthermore, in my experience designing deception/hacking CoT and output monitors for reasoning models, monitors are highly sensitive to the prompt and how the CoT is presented to them. I would strongly suspect that prompt variations could increase/decrease performance by 10+ percentage points. If the authors put different amounts of optimization into the different prompts this effect could be larger than the difference they observe. I believe the authors should modify their language slightly to avoid overstating this difference and possibly add some detail in how the stated prompts were made, e.g. how many iterations were made, was there a lot of iteration on one which was then ported to the other setting etc. \n\n- They only use 4.1-mini as a monitor. I think a sensitivity analysis of the monitor capability would strengthen the paper. Would CoT monitors still underperform if a stronger model was used for example?", "was_revised": true, "final_justification": "I was impressed by the authors response to my and other reviewers concerns. I think they have resolved or mitigated all my concerns.\n\nIf the authors make the changes they say they will I think this could shape up to be a very good paper which is why I am increasing my score to a 6.", "rebuttal": "We thank you for your thoughtful feedback and constructive comments. We have carefully addressed each concern and made corresponding updates to our paper.\n\n## **Response to Major Concerns**\n\n### **Performance Differences Between CoT and Action Monitors**\n\nWe acknowledge the reviewer's concern about potentially overstating the performance difference between CoT and action monitors. We agree that both monitor types show correlated performance (shown in a table below) and that in real-world deployment settings, the absolute performance levels would be unacceptable. Our goal was to demonstrate performance improvements over current baselines and identify interesting failure modes rather than present deployment-ready solutions.\n\nIn response to this feedback, we have:\n\n* Revised our language throughout the paper to avoid overstating the performance differences  \n* Clarified our objectives to emphasize that we aimed to show relative improvements and failure mode analysis  \n* Added methodological details about prompt development (see below)\n\n### **Prompt Development and Optimization**\n\nRegarding concerns about prompt sensitivity and potential optimization bias, we provide the following clarification:\n\nDuring our experiments, we tested over a dozen prompts before settling on the final versions. Our optimization process included:\n\n* Testing unified prompts for both action and CoT monitoring  \n* Developing separate specialized prompts for each monitoring type  \n* Incorporating few-shot examples (which we found did not improve performance)  \n* Approximately six iterations within each broad category\n\nIn preliminary work with our training split of BigCodeBench, we made several improvements:\n\n* Adding supplementary reflection questions substantially improved both monitor types  \n* Asking the monitor to provide specific quotes to justify its score improved monitor performance  \n* Addressing a failure mode in CoT monitors that was causing high false positive rates due to the lengthy and convoluted nature of reasoning traces (e.g., from DeepSeek). Specifically, we adjusted the prompt with an instruction not to consider lengthy or repetitive content to be suspicious by default.   \n* Testing few-shot examples on our validation set (which showed no improvement)\n\nWe consider both prompts to be reasonably optimized but not exhaustively tuned, with comparable optimization effort applied to each. We believe this level of optimization is sufficient to support our core contributions regarding the relative strengths of different monitoring approaches.\n\n### **Monitor Model Selection and Sensitivity Analysis**\n\nIn response to reviewer feedback, we have expanded our monitor evaluation:\n\n* **Added results** for GPT-4.1-nano, Claude 3.5 Haiku, and Gemini 2.5 Flash Lite (reasoning)  \n* **Included sensitivity analysis** across monitor capabilities as requested\n\nOur initial choice of GPT-4.1-mini was intentional—we selected a single monitor weaker than all reasoning models to explore other axes of variance while controlling for monitor capability. We acknowledge that reasoning models as monitors represent current best practices in deception detection, particularly for CoT monitoring and long-context handling. The expanded results now provide a more comprehensive view of performance across monitor capabilities.\n\n**Subtle Catch Rate @ 1%FPR**\n\n| Monitor | Action-Only | CoT-Only | Hybrid (w=0.25) | Hybrid (w=0.50) | Hybrid (w=0.55) | Hybrid (w=0.75) | Max(Action,CoT) | Geometric Mean |  \n|-----------------------|-------------|----------|-----------------|-----------------|-----------------|-----------------|-----------------|----------------|  \n| claude-3.5-haiku | 21.1% | 19.8% | 20.6% | 24.0% | 25.3% | 27.8% | 23.6% | 24.2% |  \n| gemini-2.5-flash-lite | 53.5% | 64.9% | 62.7% | 71.3% | 72.0% | 66.3% | 74.4% | 72.5% |  \n| gpt-4.1-mini | 3.9% | 12.6% | 12.6% | 11.0% | 11.2% | 8.3% | 13.3% | 12.4% |  \n| gpt-4.1-nano | 4.4% | 4.9% | 4.1% | 4.3% | 4.6% | 4.4% | 5.3% | 4.7% |\n\n**Non-Subtle Catch Rate @ 1%FPR**\n\n| Monitor | Action-Only | CoT-Only | Hybrid (w=0.25) | Hybrid (w=0.50) | Hybrid (w=0.55) | Hybrid (w=0.75) | Max(Action,CoT) | Geometric Mean |  \n|-----------------------|-------------|----------|-----------------|-----------------|-----------------|-----------------|-----------------|----------------|  \n| claude-3.5-haiku | 95.3% | 52.6% | 78.6% | 86.8% | 88.6% | 92.5% | 84.9% | 86.9% |  \n| gemini-2.5-flash-lite | 99.2% | 70.2% | 79.3% | 97.7% | 97.8% | 98.0% | 98.1% | 97.9% |  \n| gpt-4.1-mini | 77.7% | 47.4% | 51.2% | 70.1% | 72.2% | 74.7% | 73.7% | 72.1% |  \n| gpt-4.1-nano | 56.1% | 13.7% | 11.8% | 29.3% | 31.9% | 39.5% | 37.2% | 31.7% |\n\n**Correlation of suspicion scores**\n\nThe correlations in the table refer to the relationship between action and CoT suspiciousness scores. Rows show results for red team models while columns show results for monitor models. Gemini-2.5-flash-lite has lower correlations overall, which helps the hybrid monitor improve more since their errors overlap less, since both baselines have similar performance. Although correlation is also low with gpt-4.1-nano as monitor, the hybrid gains are limited since the CoT monitor has significantly lower performance than the action-only.\n\n| | claude-3.5-haiku | gemini-2.5-flash-lite | gpt-4.1-mini | gpt-4.1-nano |  \n|--------------------------------|------------------|----------------------|--------------|--------------|  \n| claude-3-7-sonnet-latest | 0.5938 | 0.3296 | 0.7041 | 0.4023 |  \n| deepseek-r1 | 0.5617 | 0.3151 | 0.6738 | 0.2880 |  \n| deepseek-r1-distill-llama-70b  | 0.6312 | 0.4565 | 0.8269 | 0.3799 |  \n| deepseek-r1-distill-qwen-14b   | 0.5614 | 0.4239 | 0.7885 | 0.3834 |  \n| grok-3-mini-beta | 0.5189 | 0.3831 | 0.6825 | 0.3800 |\n\n### **Additional Clarifications**\n\n**CoT Ordering**: We placed the action output before the reasoning in our CoT monitor prompts to provide context before the detailed reasoning analysis. This ordering aimed to help the monitor understand what the model ultimately did before examining why it did it. We found that agents frequently included drafts or complete answers within their CoTs, making the specific ordering less critical than initially anticipated as the monitor had access to output information regardless of placement.\n\n**OpenAI Reasoning Models**: We did not evaluate o1/o3/o4 models due to the lack of access to raw reasoning traces (as the review correctly pointed out), which are essential for our CoT monitoring approach. We have added a note in the paper clarifying this limitation.\n\n**Score Combination Strategy**: We tested multiple scoring strategies, including Max(monitor 1, monitor 2), but found that combining scores yielded better performance. This finding is now explicitly mentioned in our methodology section. Additionally, results for multiple aggregation strategies and weighting are shown in the tables above.\n\n## **Summary of Changes**\n\n1. Language revision to temper claims about performance differences  \n2. Expanded monitor evaluation including multiple models and sensitivity analysis  \n3. Added methodological details about prompt development and optimization  \n4. Clarified limitations regarding OpenAI reasoning models  \n5. Explicit discussion of score combination strategies\n\nWe believe these changes significantly strengthen the paper while maintaining our core contributions about the relative effectiveness of different monitoring approaches, the vulnerability of simple CoT monitors to reasoning content, and the benefits of combined monitoring strategies."}]

Author rebuttal:

Meta-review: The paper proposes CoT monitoring to detect misbehavior or sabotage of AI models. There is a unanimous consensus towards acceptance.
Decision: Accept (poster)