PRISM: Festina Lente Proactivity—Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents
Authors: Yuxuan Fu
Xiaoyu Tan
Teqi Hao
Chen Zhan
Xihe Qiu
Venue: ICLR 2026 Poster (2026)
Keywords: Proactive Agents
Uncertainty Estimation
Cost-Sensitive Learning
Adaptive Computation
Calibration
Knowledge Distillation
Primary area: foundation or frontier models, including LLMs
TL;DR: PRISM learns a cost-sensitive, decision-theoretic gate that proactively intervenes only when help is needed and welcome, triggering selective slow reasoning to cut false alarms, latency, and compute while boosting accuracy.

Abstract: Proactive agents must decide not only what to say but also whether and when to intervene. Many current systems rely on brittle heuristics or indiscriminate long reasoning, which offers little control over the benefit-burden tradeoff. We formulate the problem as cost-sensitive selective intervention and present PRISM, a novel framework that couples a decision-theoretic gate with a dual-process reasoning architecture. At inference time, the agent intervenes only when a calibrated probability of user acceptance exceeds a threshold derived from asymmetric costs of missed help and false alarms. Inspired by festina lente (Latin: "make haste slowly"), we gate by an acceptance-calibrated, cost-derived threshold and invoke a resource-intensive Slow mode with counterfactual checks only near the decision boundary, concentrating computation on ambiguous and high-stakes cases. Training uses gate-aligned, schema-locked distillation: a teacher running the full PRISM pipeline provides dense, executable supervision on unlabeled interaction traces, while the student learns a response policy that is explicitly decoupled from the intervention gate to enable tunable and auditable control. On ProactiveBench, PRISM reduces false alarms by 22.78% and improves F1 by 20.14% over strong baselines. These results show that principled decision-theoretic gating, paired with selective slow reasoning and aligned distillation, yields proactive agents that are precise, computationally efficient, and controllable. To facilitate reproducibility, we release our code, models, and resources at https://prism-festinalente.github.io/; all experiments use the open-source ProactiveBench benchmark.

Published as a conference paper at ICLR 2026
PRISM: FESTINA LENTE PROACTIVITY-RISK- SENSITIVE, UNCERTAINTY-AWARY DELIBERATION FOR PROACTIVE AGENTS
Yuxuan Fu', Xiaoyu Tan"; Teqi Hao' ; Chen Zhan', Xihe Qiu
Shanghai University of Engineering Science, National University of Singapore
yuxuanfu@suer edu. en, chenzhan361 6 @gmail com, 99t.eegiul9933@mmmm com
com, teqihao@gmail.. com
ABSTRACT
Proactive agents must decide not only what to say also whether and when to intervene. Many current systems rely on brittle heuristics indiscriminate long rea- soning, which offers little control over the benefit-burdee tradcoff. We formulate the problem cost-sensitive selective intervention and present PRISM, novel frame- work that couples a Dacision-theorete gate with dual-process reasoning architec- ture. At inference time, the agent intervenes only when calibrated probability of user acceptance exceeds a threshold derived from asymmetric costs of missed help and false alarms. Inspired by festina lente (Latin: "make haste slowly"), we gate by an acceptance acceptannee-allibrated, calibrated, cost-derived threshold and invoke resource-intensiee Slow mode with counterfactual checks only near the decision boundary, concentrat- ing computation ambiguous and high-stakes cases. Training uses gate-aligned, schema-locked distillation: a teacher running the full PRISM pipeline provides dense, executable supervision
unlabeled interaction traces, while the student learns a response policy that is explicitly decoupled from the intervention gate to enable tunable and auditable control. On PROACTIVEBENCH, PRISM reduces false alarms by 22.78 % and improves F1 by 20.14% % over strong baselines. These results show that principled decision-theoreetic gating, paired with selective slow reasoning and aligned distillation, yields proactive agents that are precise, computa- tionally efficient, and controllable. To facilitate reproducibility, models, and resources at fulttps:: : //pprismmmm fest insstiinanett.. git.ub. io/] all experiments use the open-source PROACTIVEBENCH benchmark.
release code,
INTRODUCTION
Proactive agents aim to surface help at the right moment, before ask, yet without becoming intrusive (Bubeck et al. 2023 Schick et al. 2023 Yao et al.) 2023 Achiam et al., 2023 Lu et al. 2024). The central challenge is speak remain silent decision under asymmetric costs. False alarms erode trust and add cognitive load, while misses forgo timely assistance. In real deployments this decision must be made with partial information, nontrivial tool latency, and a strict compute budget. Consider a coding copilot. When a flaky continuous integration failure emerges during sensitive configuration edit, the agent should intervene only if help needed and likely to accepted; otherwise it should stay out of the way.
Prevailing pipelines often rely on ad hoc thresholds or chain thought used by default. This leads to two persistent failures: over intervention that interrupts user flow, and indiscrininate compute that expends slow and expensive reasoning even for obvious cases (Cobbe et al. 202T [Shiick et al. 2023 Wang & Zhou 2024 Zhang et al. 2024). In addition, acceptance optimization is frequently decoupled from timing control. Prompts or output formats are tuned offline, and only later are heuristics added for when to speak. This blurs the line between the leamned policy and product controls and weakens guarantees on the tradeoff between quality and efficiency.
"Equal contribution.
1 "ooresponniig author.

---

Published as a conference paper at ICLR 2026
Environment
VS Code: The user changed the heading in README.md from "Project X" to "Project X v2".
Observation
Git: file modified (README.ma) md), no new test failures, terminal idle.
<Internal-Minor documentation tweak, no blocking risk. Estimates:
Should create Pull Request and generate changelog now?
Pneed 0.14, Paccepr = 0.22;
threshold 0.35 = Silent.
Agent
Agent with PRISM
Not yet, I'm still working on the documentation.
Continues editing README, uninterrupted.
User
User
Figure 1: PRISM reduces false alarms by estimating confidence before speaking. The agent first estimates calibrated Paccept), compares Pacceps with an adaptive threshold (Preed: CrA, CEN), and triggers single pass slow stage only within narrow region near the threshold. In benign edit events, low Pueed raises т, the gate remains silent, and the user continues without interruption.
We introduce PRISM (Proactive Risk Sensitive Intervention with a Slow mode Margin), a simple and principled framework that unifies need and acceptance modeling, cost sensitive gating, and selective slow reasoning. This design is inspired by festina lente (Latin: "make haste slowly"): the agent gates by expected utility and invokes slow mode only the decision boundary. We frame timing as a selective decision over two calibrated probabilities: the probability that help is needed (Pheed) and the probability that help will be accepted (Paccept.. cost aware gate compares acceptance confidence with an adaptive threshold that depends on Pheed and on the relative costs of false alarms and misses. Slow reasoning is not universal. PRISM allocates single pass slow stage only near this decision boundary through a slow mode margin, which concentrates computation where it is most likely change the outcome. The result is an interpretable control surface with small of knobs, such as cost terms and the width of the near boundary region, that moves the benefit and burden frontier in predictable ways. This connects to the literature on selective prediction and calibration Yaniv al.) 2010 Geifmann & El- Yaniv, 2017 Guo et al. 2017). We further characterize how the adaptive threshold varies monotonically with costs and with Pneed and we relate the slow trigger rate to expected latency, which yields compact map from knobs to metrics.
Training mirrors deployment. PRISM applies the same costs, the same gate, and the same slow mode margin shape both runtime control and the learning signal, which reduces the gap between simulation and production. Concretely, we mix three terms in a deployment aligned objective: (i) outcome acceptance, (ii) need consistency that rewards silence when no help is needed and penalizes silence when help is needed, and (iii) burden terms that price user disruption and tokens used by slow reasoning. Our evaluation follows paired event protocols that stress timing under partial information and realistic tool use. We report results on synthetic teacher traces with diagnostics and on real world coding logs QJimenez al. 2024 Zhang al., 2024). Code and protocols will be released upon acceptance through an anonymous repository.
Contributions. PRISM connects a simple principle with minimal implementation and repro- ducible evaluation protocol:
Cost sensitive gating in PRISM. The method separates need from acceptance and concen- trates single pass slow reasoning near an adaptive boundary, which yields a compact and interpretable control surface for quality and efficiency tradeoffs.
2

---

Published as a conference paper at ICLR 2026
Gate aligned objective. The same costs and margins are used train time and at test time, which improves calibration and closes the gap between simulation and production.
Budget aware evaluation. We report precision, recall, FI, false alarm rate, slow token cost, and area based summaries of the benefit and burden tradeoff (AUDBC). Across always slow, no slow, fixed threshold, and schema only distillation baselines, PRISM reduces false alarms at matched recall while lowering slow compute, and it maintains control stability under distribution shift Yuu et al. 2025). These properties are essential for reliable deployment.
2
RELATED WORK
Proactive agents and event-stream timing. Recent work shifts from purely reactive assistance to deciding when to intervene within evolving event streams. Proactive Agen//Proactiveeent formalize eereptance-supervisse intervention under realistic workflows with partial information, emphasizing timing and user burden (Lu et al. 2024). Similar challenges in optimizing intervention timing have been observed critical decision-support domains (Hao et al. 2025), highlighting the universality of this temporal decision problem. We build on this direction and emphasize deployment alignment: the gate, costs, and visible schema are shared between training and inference. This design isolates improvements that arise from better timing rather than from interface or prompt mismatch (Lu et al. 2024).
Risk-sensitive intervention, abstention, and calibration. The choice to speak remain silent is binary decision with asymmetric costs for false alarms and missed help. Cost-sensitive learning and reject-option theory provide principled thresholds and characterize the risk and coverage trade-off'
(Hikan 2001 Chow [Cow 2003). Selective deep classifiers instantiate these ideas with integrated abstention
heads (Geifman & EI- Yaniy (009).. Because our gate relies calibrated probabilities, it connects to modern calibration methods for neural networks (Guo et al. (017)) and to metareasoning views that allocate computation where the expected value of computation is high (Russell & Wefald [99T). Our controller concentrates additional SLOW computation within a wear-threshol. band, operationalizing these principles for proactive timing.
Uncertainty-aware objectives beyond binary rewards. Standard RLHF typically optimizes single scalar reward. In contrast, we estimate intervention probabilities (Preed, Paccep) and compose them with labels (yneed, Vsccept) into a structured objective over event streams, which is consistent with reinforcement learning principles (Sutton Barto 2018). This view aligns with efforts to move beyond binary rewards by training language models reason about their own uncertainty (Damani jt al., 2025). It also dovetails with selective prediction, since these probabilities parameterize a cost-sensitive gate rather than optimizing monolithic reward (Geifman & El- Yaniv) 2 Suchh aligmment strategies relate to recent works on using LLM--drived feedback to balance helpfulness and harmlessness (Tan al.. 2023). and frameworks leveraging LLMS guide reward signals in decision-making tasks (Deng et al. 2025).
Protocol-aligned synthesis, distillation, and selective SLOW. To scale supervision while remaining faithful to the evaluation protocol, we synthesize event-conditioned decisions with a teacher language model and distill them into a smaller model (Hinton et al. 2015), following self-generated instruction pipelines (Wang et al., 2023). When uncertainty is high. we invoke a SLOW scratchpad-style pass motivated by thouugt prompting (Wei al. 2022 Tan al. 2024) while keeping the visible schema identical across training and evaluation. This alignment ensures that reliability gains stem from improved timing and calibration rather than from format shifts.
3 METHOD
We frame proactive assistance as cost-sensitive selective decision problem. At each timestep f, our agent observes a context X, and estimates two calibrated probabilities: Pneed, = Pr(hlpp is needed X₂) and Paccept.t Pr(offer is accepted Xt, intervene). The decision to intervene is governed by cost-sensitive gate. Let CpA be the cost of a false alarm and CrN be the cost of a false negative. The agent intervenes only if the estimated acceptance probability Paccept, exceeds dynamic threshold
3

---

Published as a conference paper at ICLR 2026
Observations
Calibrated (Pwnl Parsept )
External Costs (Cpa, CFx)
Event I:...
Event 2:
Signal Estimation
Cost Delinition
Else, Move
Check if Paccept
T?
if yes
O
Check if
Dynamic Threshold
Silent
Paccept
LLOW_MARGIN.
Else, Move to
iF
yes
Slow Thinking
Fast Thinking
Usser
Figure PRISM Pipeline. model first estimates two calibrated probabilities from context X₃: the user's need for assistance (Pneed) and the likely acceptance of an offer (Paccept). A cost-sensitive gate then decides to intervene only if the acceptance probability meets a dynamic threshold (Pneed) that accounts for the relative costs of false alarms (CFA) and missed opportunities (CFx). To balance accuracy and efficiency, a resource-intensive slow reasoning pass is triggered only when the initial prediction is ambiguous and falls within narrow margin Sslow of the decision boundary.
determined by the estimated need Pneed,t
CFA
Paccept, 2 (Pueed,,)
A
(3.1)
CFA + Pueed,; CPN
This rule ensures that as the certainty of need increases, the agent can tolerate lower chance of acceptance to intervene. To improve decisions in ambiguous cases without universal high cost, employ a dual-process architecture. A fast model provides initial estimates (pheed, Pacrrept.))) A single, more powerful slow reasoning pass is triggered only if the fast estimate is close to the decision boundary, i.c., Psccept,t (( ) Salow, where 8stow is configurable margin.
The training process is explicitly aligned with this decision architecture. We employ a method called Decision-Conss Consistent Curation (RDC) to distill knowledge from powerful teacher model. We filter the training data using ranking score RDC that rewards the teacher for accepted interventions and penalizes for miscalibrated probability estimates:
yneed) 1 11 (prod)
RDC
Jaccept
(Qneed
1} (Aaccept Ysccept)
(3.2)
where (queed: qaccept) are the teacher's probabilities and (Yneed: Vaccept) are ground- -ruth labels. A student model is then trained via supervised fine-tuning the top-ranke aap-rnnedd curated data subset D The learning objective = Luced + Lace + Lmurden ensures that the student's estimates of Proed and Paccept well-calibrated (using inverse propensity scoring for Lace handle selection bias) and includes regularization terms that penalize the burden of false alarms and excessive slow reasoning.
4 EXPERIMENTS
We evaluate PRISM on PROACTIVEBENCII, targeting three desiderata of proactive assistance: timeliness, accuracy, and cost sensitivity. Our primary evaluated model is a student trained via RDC distillation with supervised fine-tuning (RDC-SFT), We compare against strong open-source and proprictary baselines, and we analyze how PRISM's decision policy shapes the benefit- burden trade-off. Our study addresses three key questions:
4

---

Published as a conference paper at ICLR 2026
(Q1) Comparative effectiveness & Human Alignment. How does PRISM perform relative to proprietary frontiers and prior SOTA? Crucially, are the improvements on automatic metrics validated by human experts?
(Q2) Inference Efficiency. To what extent does the slow-margin gating strategy improve the decision quality without incurring the high latency and compute costs of "System 2" reasoning?
(Q3) Mechanism & Training. How do RDC-filteed supervision and dual-signal gating (Pueed, Paccept) contribute performance? Is the system robust to calibration drift?
4.1 EXPERIMENTAL SETUP
Training data and recipe. Our student, QWEN3-8B-PRISS trained with full-parameter SFT on an RDC-filtered subset of the official training split containing 1,800 instances (< $ of the original). During training, we use AdamW with learning rate of 1 10-5 and 0.1 warm-up ratio under cosine schedule for 3 epochs. We adopt the Qwen chat template with 4096-token context and bfl 6 precision; the effective batch size per device 4 (per-device batch size 1, gradient accumulation 4). Training runs single NVIDIA A 100 (80 GB) and completes in approximately 2.5 hours. Full hyperparameters are listed Appendix G
Evaluation Dataset. All experiments use PROACTIVEBENCH (Lu et al. 2024). which contains user event streams from three domains: coding, writing, and daily life. We follow the official split: the training portion is used only for student distillation (and, where applicable, weighted SFT), while the held-out test set (233 clips) reserved for evaluation. Each clip a sequence of tuples (E,, At, S.) for time step f, denoting the observed event, user activity, and environment state. At each step, the agent either issues proactive proposal P. or remains silent.
Baselines.
We compare the following systems under a unified evaluation harness.
1. Frontier API LMs (proprietary). Strong proprictary models (c.g., GPT-4o, Claude 3.5-Sonnet)
prompted for proactive assistance without any specialized gating distillation.
2. Open-source base LMs. Strong open models (c.g.. LLaMA-3.I-SB. Qwen2-7B) with the same prompting protocol and no special gating/distillation.
3. Proactive Agent (SOTA). The public pipeline and decision policy from the original PROAC- TIVEBENCH paper, evaluated as provided.
4. PRISM. A student trained via RDC distillation with supervised fine-tuning (RDC-SFT), evaluated with the full PRISM decision policy.
Metrics. We report standard acceptance- eeceptance-aware aware detection metrics on the test streams: Recall, Precision, Accuracy, False-Alarm rate, and Fl with numerical stabilizer >> 0:
2 Precision Recall
F1
Precision + Recall +
To summarize cenefit-burden trade-offs we adopt cassrcoverage style view from selective prediction and instantiate it with a cost-sensitive utility, yielding AUDBC as our area-under-curve summary (Geifman & Yaniv 2017). We do not train reward model. Instead,
adopt LLM-as-Judge setup that strictly follows the evaluation guidelines. Let CFA > 0 be the per-false-alarm cost and (false negative) cost. For fixed CEN, decision policy induces
CEN the re--mssed-opportumme (i) a normalized burden B(CPN) E [0, 1] (we use the false-alarm rate) and (ii) a normalized net benefit (Cx()))) E [0, 1] defined as the utility improvement over the alvays-silent baseline after accounting for CFA and CEN. We summarize performance by integrating All along the burden axis:
AUDBC = (.'' AU(b) db E [0,
(4.1)
with AU(() E[TP] CFAE[P]] FFNE[FN] - E[TP] kilent
b= B(CFN))
(4.2)
Z
5

---

Published as a conference paper at ICLR 2026
Model
Recall f
Precision
Accuracy
False-Alarm
F1-Score 1
Automatic Evaluation (I..M-as-Judge)
Claude-3.5-Son Sonnet
97.89%
45.37%
49.78%
54.63%
62.00%
GPT-4o-mini
100.00 %
35.28%
36.12%
64.73%
52.15%
GPT-4o
98.11%
48.15%
49.78%
51.85%
64.60%
LLaMA-3.1-88
1----..1---BP-- Proactive
99.06%
49.76%
52.86%
50,24%
66.25%
Bwen2-7B-Proachie
100.00%
49.78%
50.66%
50.22%
66.47%
Deepseek-R1
98.12%
72.35%
72.96%
27.64%
83.28%
Qwen3-8B
73.79%
73.33%
67.85%
26.67%
73.34%
Qwen3-8B-PRISM
98.88%
77.05%
76.39
22.94%
86.61
Human Expert Evaluation
DeepSeek-RI Qwen3-8B-PRISM
99.05%
70.03%
71.12%
29.70%
82.05%
99.41 %
74.01 %
73.79%
25.91 %
84.85 %
Table 1: Main comparison on PROACTIVEBENCH (held-out test set). We report performance under two protocols: Automatic Evaluation (using validated LLM-ensemble) and Human Expert Evaluation (bottom rows). PRISM (our RDC-distilled student with dynamic gating) consistently outperforms strong proprietary and open-source baselines across both regimes. Notably, it surpasses its teacher (DeepSeek-RI) in Precision and False-Alarm rate while maintaining near-perfect Recall, validated by human experts.
where TP, FP, FN expectations per time step under the evaluated policy, [E[TP] silent is the always- silent baseline utility (zero for detection tasks), and Z normalizes to [0, i higher AUDBC indicates consistently larger net benefit the same or lower burden across cost settings.
Judging and Validation Protocol. We do not train reward model. Instead, we adopt an LLM- as-Judge setup that strictly follows the official PROACTIVEBENCH rubric. At each step t, employ majority vote among three strong frontier models Seaeels-DeeSSee--AAe GPT-40, and Claude 3.5- Sonner to determine whether help needed and likely to be accepted. The gold label derived as y²(() (t) Yneed (L) A Vuccepa (). Acknowledging recent findings potential biases and overconfidence LLM judges (Ye et al. 2024), we rigorously validated our protocol against human judgment. We sampled 229 events and enlisted 7 volunteers from diverse backgrounds (including undergraduate and graduate students in programming, marketing, and business) to annotate intervention needs. Our ensemble judge achieved 89.1% agreement with human consensus and Cohen's к of 0.7 (substantial agreement), confirming that it serves as a reliable proxy for human evaluation in this domain. Detailed agreement statistics and comparisons across different judge pools are provided in Appendix C
4.2
MAIN RESULTS: PROACTIVE ASSISTANCE PERFORMANCE
Table demonstrates that PRISM establishes a new state-of-the-art by effectively curbing the "over-proactiveness" plague inherent in existing models.
Solving the False-Alarm Dilemma. Baseline agents, including strong proprietary models (e.g.. GPT-4o) and prior SOTA (Qwen2-7B 1eae2----Proctivie suffer from excessive chatter, with false-alarm rates consistently exceeding 50%. PRISM drastically corrects this behavior. Compared to Qwen2-7B- Proactive, it improves the F/-Score by over 20 points (66.47 86.61) while cutting the False-Alarm Rate by nearly 54% relative (50.22 - 22.94). This massive gain in precision (49.78 > 77.05) is achieved with negligible impact the near-saturated recall.
Statistically Significant Superiority, Crucially, the PRISM student does not merely imitate its teacher; it surpasses it. Despite using a smaller backbone, our model improves upon the powerful DeepSeck-RI teacher. Statistical significance testing (details in Appendix D) confirms that while
In practice we sweep Crx over grid, obtain pairs (B(C(R). (Z((R(R) sorted by burden, and estimate cquation 4.1 via the trapezoidal rule: AUDBC Σ, (A + AU)) (bit1 h.).

---

Published as a conference paper at ICLR 2026
Policy
Recall f
Precision 1
Accuracy
False-Alarm +
F1-Score f
Fixed ==0.5 Static-cost 7 (CrA, CPN)
92.42 %
71.96%
76.03% %
28.23 %
80.74%
38.54%
59.67%
63,94%
40.32% 31.70%
46.83%
Dynamic T (Preed)
72.41 %
68.29%
69.52%
70.29%
Table 2: Ablation A: thresholding only (no gating). Fixed threshold dominates when Pooed is uncalibrated.
PRISM maintains parity in Recall (+0.76%. 0.115), it achieves statistically significant improvements Precision (+4.70%% p < 0.001) and False-Alarm Rate reduction (p 0.001). This confirms that the precision gains are not random fluctuations but result of our structured distillation.
Hluman-Verifite Reliability. This advantage is robust to the evaluation protocol. As shown in the bottom section of Table [ human experts corroborate the automatic metrics: PRISM delivers superior user experience, achieving a higher Fl-score than DeepSeek-RI (84.85% vs. 82.05%) and lower false-alarm rate (25.91% vs. 29. 70%). This alignment between LLM and human judges (Cohen's 's k = 0.71) underscores the validity of our results.
Attribution. We attribute these gains to two factors: (1) Need/Accept Disentangling: RDC-SFT teaches the student to explicitly estimate Paccept, filtering out "correct but unwanted" proposals that the teacher might generate. (2) Cost-Sensitive Gating: The dynamic threshold and slow-margin gate route only borderline cases to deliberation, recovering true positives without the computational cost or hallucination risk of universal reasoning (Deng al. 2026).
4.3 ABLATION STUDIES
We categorize our analyses into four parts using the Qwen3-8B backbone: (A) evaluates thresholding on the base model to demonstrate the need for calibration; (B) analyzes the efficiency-qualie trade-off of our inference strategy: (C) compares training paradigms; and (D) dissects the contribution of decision signals and robustness to cost/parameter shifts.
(A) Thresholds (policy). We compare three decision rules under direct inference (no RDC-SFT, no post-hoc calibration): (a) Fixed threshold, (b) Static cost threshold r(CFA, CFN), (c) Dynamic threshold 7 (Preed) computed from the model's 's (uncalibrated) Pneed-
Analysis. The fixed rule yields the best overall balance (F1 80.74) and lowest false alarms (28.23). The static-cost rule is overly conservative on this distribution, collapsing recall (38.54) and harming F1. The dynamic 7 (Pnced) improves markedly over static-cost (FI 70.29 vs. 46.83) but still underperforms fixed, with higher false alarms (31.70 vs. 28.23).
This pattern is consistent with probability miscalibration: without RDC-SFT or post-hoc calibration, Pneed is noisy near the decision boundary, so instance-wise thresholding does not reliably separate needed from non-needed steps. our main results, where Pneed (and Paccept) are calibrated via RDC-SFT, dynamic policies surpass fixed thresholds. Here, the ablation isolates that better policies require calibrated signals; Post-hoe temperature scaling is simple but surprisingly effective fix for probability miscalibration, and we observe similar gains here when calibrating held-out judge-labeled set (Guo et al. 2017).
(B) Inference Strategy (Efficiency). We examine whether PRISM can deliver "System 2" quality at "System 1" speeds. 3 compares strategies using the RDC-SFT student.
Analysis. Table 3 reveals a compelling efficiency-quality trade-off, positioning PRISM as Pareto improvement over static inference strategies. By dynamically routing only ~11% of borderline cases to the slow reasoning path, PRISM achieves a massive quality boost-improving Fl by +5.06 points over Fast-only (83.09% - 88. 11%5-hilee incurring negligible P95 latency overhead of just 20ms. We selected the margin = 0.1 based on a detailed sensitivity analysis (see Appendix and Figure 5). which identifies it as the "sweet spot" before latency costs scale linearly. As visualized in Figure 3 our method defines the optimal Pareto frontier (blue line). This frontier is derived from comprehensive 2D grid sweep across varying cost ratios and margins (fully detailed in Appendix
7

---

Published as a conference paper at ICLR 2026
Strategy
Recall
Precision f
False-Alarm + F1-Score 1
AUDBC1 1
Tokens
Lat (P95)
Fast-only
100%
71.08%
28.92%
83.09%
79.26%
510
176ms
Slow-only
100
75.17%
24.83%
86.79%
81.43%
693
312ms
Slow-on-margin
100%
78.81 %
21.19% %
88.15 %
82.72%
541
196ms
Table 3: Ablation B: Efficiency. Slow-on-margin (6 = 0.1) achieves the best performance with
negligible latency overhead compared Fast-only.
AUDBC-Latency Pareto frontier
12, t-0.1
0.82
1:2.
6.0.5
AUDBC
0.80
0.79
0.78
180
190
200
210
220
230
240
p95 Latency (ms)
Figure 3: Efficiency-Qullity Pareto Frontier. We plot AUDBC against P95 Latency across various cost ratios (CFA CPN) and slow-margins (6). The blue line traces the optimal frontier, showing that PRISM (particularly
0.1) achieves maximal decision benefit with minimal latency overhead (~200ms). Grey points represent suboptimal configurations (e.g.. higher å or uncalibrated baselines) that incur higher latency without corresponding utility gains.
Table 10 demonstrating that margin-based gating effectively captures the utility gains of "System 2" reasoning without the prohibitive latency of Slow-only execution.
(C) Distillation and SFT pathways (training). We compare four training pathways on the same
backbone (Qwen3-8B) with matched decoding and judging. (i) SFT: vanilla supervised fine-tuning on the unfiltered dataset (no RDC screening). (ii) DFT: Dynamic Fine- Time-Tunigg Wu et al. (2025). which stabilizes token-level gradients by dynamically rescaling the objective with each token 's model probability; in our setting DFT is applied as a second-stage fine-tune starting from the RDC-SFT checkpoint. (iii) Weighted-SFT: a second-stage fine-tune from RDC-SFT using RL-like scalar weight Pional for each example, adapted from
Regression (RWR) Peters & Schaal (iv) RDC-SFT (ours): full-parameter SFT on an RDC-filtered dataset with explicit (Poeed Paccept) supervision.
(2007); Peng al. (2019).
Analysis. RDC-SFT attains the best overall balance: it raises Fl 86.61 (vs. 76.09 for SFT. +10.52 pts; vs. 80.59 for Weighted-SFF +6.02 pts) and yields the lowest False-Alarm (22.4) 49, reduction -5.07 vs. SFT and -6.82 vs. Weighted-SFT), while keeping Recall high (98.88). Vanilla SFT on the unfiltered corpus achieves perfect recall but at the cost of higher chatter (FA 27.56) and lower precision (72.43), reflecting label noise and timing mismatches. Weighted-SFTT improves Fl over SFT, but its scalar weights Recal- depend false alarms (FA 29.31), indicating sensitivity to noise and acceptance miscalibration. Second-stage DFT underperforms here (Recall 80.14): when applied after RDC-SFT, dynamic rescaling can blunt acceptance-awanl gradients learned during distillation, yielding conservative triggering without precision gains (same 72.43 SFT).
uncalibrated (yneed inffatee
8

---

Published as a conference paper at ICLR 2026
Training Pathway
Recall f
Precision t
Accuracy
False-Alarm 4
F1-Score 1
SFT
100.00 %
72.43%
69.52%
27.56%
76.09%
DFT
80.14%
72.43%
69.52%
27.56%
76.09%
Weighted-SFT 93.70%
70.68%
72.10%
29.31 %
80.59%
RDC-SFT(ours) 98.88%
77.05%
76.39
22.49% %
86.61
Table 4: Ablation C: training paradigms Qwen3-8B. RDC-SFT (ours) trains on RDCC-fltered supervision with explicit need/accept targets; DFT and Weighted-SFT are second-stage finetunes starting from the RDC-SFT checkpoint.
Overall, these results suggest that data quality and target structure dominate: RDC filtering plus explicit (Preed: Paccept) supervision improves precision and suppresses false alarms without materially sacrificing recall, whereas post-hoc reweighting (Weighted-SFT) probability-rescaled objectives (DFT) are less effective in the presence of acceptance/timing noise.
(D) Decision Mechanism & Calibration. We isolate the contribution of our dual gating signals (Phcod. Paccept) and quantify the impact of post-hoc calibration on the RDC-SFT student.
Configuration
Recall
Precision f False-Alarm | F1-Score AUDBC1
Piocage only (Picod=1)
99.95%
46.20%
62.50%
63.19%
58.40%
Poeod only (Pac-T (Preed) ))
99.15%
69.50%
29.10%
81.72%
82.45%
Uncalibrated (Presd, Paccept)
98.80%
74.77%
25.23%
85.12%
86.43%
Calibrated (Ours)
98.88 %
77.05%
22.94%
86.61 %
88.52 %
Table Ablation D: Gating Signals & Calibration. Using both signals is crucial for reducing false alarms. Post-hoc calibration further aligns probabilities with risk, boosting Precision and AUDBC.
Analysis. Table validates two key components of PRISM. First, Need/Accept Disentanglement: Relying solely Paccept (Row I) causes catastrophic rise in False Alarms (62.50%) because users often accept helpful but ill-timed suggestions. Using Paeed alone (Row 2) is safer but suboptimal. Combining them (Rows 3-4) balances timeliness with acceptance. Second, Calibration Utility: While the uncalibrated RDC-SFT model is already strong (Row 3), post-hoc temperature scaling (Row 4) significantly reduces the False-Alarm rate (25.23% - 22.94%) and improves AUDBC (+2.09). This precise probabilistic alignment also underpins the model's robustness to varying cost ratios (CrA: CFN) and domain shifts, detailed in Appendix
4.4 CASE STUDY: STRUCTURED TRACES THAA MOTIVATE PRISM
Observation. We first benchmark a peasoning-capabl model (DeepSeek-RI) on ProactiveBench and observe substantially stronger proactive assistance than non-reasoning baselines. To probe what drives this gap without exposing free-for scratchpads, we elicit structured traces via the PROACTIVE AGENT workflow: We analyze only these structured fields offline under de-identified, non-user-facing settings.
Decomposable decision pattern. Across representative traces, we find that high high-quality assistance co co-occurs with two separable signals: (i) whether the user likely needs intervention the current event, and (ii) conditional on proposing, whether the suggestion likely to accepted. We formalize these signals probabilities Preed E [0, and Paccep E [o, I), estimated from the structured outputs above (e.g., Purpose/Proace Task/Response) via small calibrator trained on held-out judgments F
Design implication. This decomposition motivates PRISM: a lightweight gate that triggers intervention when Preed is confident, and selectively invokes slow-mode deliberation near decision margins; meanwhile final offer is regulated by the acceptance propensity Pascept, enabling explicit control of burden-benefit trade-offs.
Figure 4 iillustrates typical trace and its mapping to (Preed, Paosege) used by our gate.
No free-form chain-of-thought is stored surfaced; analysis uses only structured fields and aggregate lahels.

---

Published as a conference paper at ICLR 2026
Eyt ruas python main.py py VS Code terminal, and entor message appears: ModuleNooFoundFine module named 'requests'. Ej. The user reopens roquirements txt and scrolls and down without oditing.
Deepseek-RI
"Thoughts" repeatted scrolling 'rocuirements suggests scarching missing dependenccy There high probability Given the direct suggestion likely to accepted
guidance installing library.
PRISM
Estimatc calibrsted signals (Pred x Pareept
Gate interventions: trigger
is confident
Invoke slow-nsode deliberation only neee uncertain margins
Regulate burden henefit made-ofiss via
Figure 4: Structured trace motivating PRISM. A ProactiveBench coding event (top) yields structured output from a reasoning-capabl teacher (DeepSeek-RI; middle). PRISM (bottom) abstracts the teacher's implicit hints into calibrated signals Paced and Paccept. uses a lightweight gate to trigger interventions when Poeed is confident, and invokes slow-mode deliberation only near uncertain margins; the final offer is regulated by Paccept- Only structured fields are analyzed; no free-form chain-of-thought is stored or surfaced.
5 DISCUSSION
Reframing Proactivity as Calibrated Control. Current proactive systems often conflate the capability generate solution with the decision intervene, leading the "over-eagerness" observed baselines. PRISM demonstrates that proactivity is better modeled as dual-variable control problem: disentangling the probability of need (Pused) from acceptance (Pacespt). This separation allows the agent to remain silent even when it can technically solve problem, simply because the social or cognitive cost of interruption outweighs the probabilistic benefit.
Economic Rationality in Neural Reasoning, Our festina lente approach operationalizes the "System vs. System 2" paradigm through an economic lens. Rather than treating long-chain reasoning as a default mode, PRISM treats it as searce resource to be allocated only when the expected utility of reducing uncertainty exceeds the computational cost. The success of slow-on- marmagin gating (11% slow rate achieving Pareto optimality) suggests that proactive agents, uncertainty estimation
critical as reasoning capability.
Data Efficiency and Control Surfaces. Finally, our results challenge the trend of indiscriminately scaling supervision. By filtering for calibrated probabilities via RDC, we show that a compact student can surpass a stronger teacher in alignment- critical metrics such as precision and false alarm rate. Crucially, PRISM exposes interpretable control surface detined by cost terms (CEA, CPN) and margin 5-that that bridges static training with dynamic deployment, allowing developers tune agent assertiveness for different user personas without retraining.
LIMITATIONS AND FUTURE WORK
Despite rigorous validation on PROACTIVEBENCH, reliance on LLM-as-Judge proxies may diverge from real- world user acceptance and overlook the cognitive load of ill-timed interruptions during flow states. Methodologi- cally, PRISM assumes offline probability calibration remains stable; however, severe deployment distribution shifts could degrade Pneed and Paccopt accuracy, making dynamic thresholds suboptimal without online recali- bration. Furthermore, our formulation assumes stationary costs (CrA, CFN) during interactions. Realistically, missed intervention costs (Crx) rise near deadlines, while false alarm penalties (CFa) drop users become visibly stuck. Implementing time-varying unknown. Future work could address this a meta-comee controllt that adapts cost sensitivity based on progress or implicit feedback. Finally, distillation processes may inherit teacher biases, requiring hard safety constraints so utility-base gating does not suppress critical security warnings. Future iterations could also integrate PRISM with advanced agent workflows and tool-use optimization frameworks (Tan et al. 2025) to handle complex, multi-step maintenance tasks.
remains challenging online since total episode lengths
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 tochnical report. arXiv preprint arXiv:2303.0877) 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: early experiments with gpt-4 (2023). arXiv preprint arXi::2303. 12712, 1, 2023.
C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1): 41 46, 2003.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld. Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond binary rewards: Training Ims to reason about their uncertainty. arXiv preprint arXiv:2507. 16806, 2025.
Yongxin Deng, Xihe Qiu, Jue Chen, and Xiaoyu Tan. Reward guidance for reinforcement learning tasks based on large language models: The Imgt framework. Knowledge-Based Syxtems, pp. 133689, 2025.
Yongxin Deng, Zhen Fang, Sharon Li, and Ling Chen. Beyond in-domain detection: Spikescore for cross-domain ballucination detection, 2026. URL /
arxiv. wtt.:orr/abbbaagaa/aa /ttp:////........////////22..... 19244
Ran EI- Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, (15), 2010.
Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pp. 973-978. Lawrence Erlbaum Associates Ltd, 2001.
Yonatan Geifiman and Ran EI- Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems. 30, 2017.
Yonatan Geifman and Ran EI- Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International conference machine learning, pp. 2151-2159. PMLR, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinherger. On calibration of modern neural networks. In International conference machine learning, pp. 1321-1330. PMIR, 2017.
Teqi Hao, Xiaoyu Tan, Bin Li, Xuemin Wang. Chao Qu Yinghui Xu, and Xihe Qiu. Myp-lims: Optimizing intervention timing and subsequent decision support for mechanical ventilation parameter control using large language models. In International Conference Medical Image Computing and ComputerAAssisted Intervention, pp- 455-464. Springer, 2025.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint /to.:1503.053/ 2015.
Carlos Jimenez, John Yang. Alexander Wettig. Shunyu Yao, Kexin Pei. Ofir Press, Karthik Narasimhan. Swe-hench: Can language models resolve real-world github issues? The Twelfth International Conference on Learning Representations, 2024.
Yaxi Lu, Shenzhi Yang. Cheng Qian, Guirong Chen, Qinyu Luo, Yesai Wu, Huadong Wang, Xin Cong, Zhong Zhang, Yankai Lin, et al. Proactive agent: Shifting Ilm agents from reactive responses to active assistance. In The Thirteenth International Conference Learning Representations, 2024.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weightedd regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 191.v01110000/77 2019.
control. In Proceedings of the 24th international conference Machine learning. pp. 745-750, 2007. Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space
Stuart Jonathan Russell and Eric Wefald. Do the right thing: sludies in limited rationality. MIT press, 1991.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advonces Neural Information Processing Systems, 36:68539-68551, 2023.

---

Published as a conference paper at ICLR 2026
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge. MA, second edition, 2018.
Xiaoyu Tan, Shaojie Shi, Xihe Qiu, Chao Qu, Zhenting Qi. Yinghui Xu, Yuan Qi. Self-criticism: Aligning large language models with their understanding of helpfulness, honesty, and harmlessness. In Proceedings of the 2023 conference on empirical methods in natural language processing: industry truck, pp. 650-662 2023.
Xiaoyu Tan, Yongxin Deng, Xihe Qiu, Weidi Xu, Chao Qu Wei Chu, Yinghui Xu, and Yuan Qi. Thought-like pro: Enhancing reasoning of large language models through self-driven prolog based chain-of tftthugh. arXiv preprint arXiv:2407. 14562, 2024.
Xiaoyu Tan, Bin Li. Xihe Qiu, Chao Qu. Wei Chu, Yinghui Xu, and Yuan Qi. Meta agent workflow: Streamlining tool usage in Ilms through workflow construction, retrieval, and refinement. In Companion Proceedingy of the ACM on Web Conference 2025, pp. 458 458-467,25 2025.
Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting, Advunces in Neural Information Processing Systems, 37:66383-666409, 2024.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- generated instructions. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pp. 13484-13500 2023.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of of-thhtght prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi. Ming- Hsuan Yang, and Xu Yang. On the generalization of SFT: A reinforcement learning perspective with reward 2025.
rectification. arXiv preprint
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Syner- gizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang. Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang. Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in Ilm-as-a-judge. In Neurips Safe Generative Al Workshop 2024, 2024.
En Yu, Jie Lu, Xiaoyu Yang. Guangquan Zhang, and Zhen Fang. Learning robust spectral dynamics for temporal domain generalization, 2025, URU/tctss//arxxi.coo org. 2505 12585
Xuan Zhang, Yang Deng, Zifeng Ren, See Kiong Ng. and Tat- Seng Chua. Ask-before-plee plan: Proactive language agents real- world planning. Findings of the Association for Computational Linguisticx: EMNLP 2024. 2024.
pp.
APPENDIX
ETHICS STATEMENT
This work does present any significant ethical concerns. All datasets used in our research, such Proac- tiveBench, are publicly available and well-established academic benchmarks. The data are anonymized and does not contain personally identifiable information. Our proposed method, which focuses on improving model efficiency and accuracy, does not have foresccable direct negative socictal impacts. No human subjects were involved in our experiments.
STATEMENT ON LLM USAGE
During the preparation this work, we used LLMs, such as ChatGPT, for assistance with language editing. grammar correction, and improving readability. All core ideas, methodologies, experimental designs, results, and conclusions were originally conceived and articulated by the human authors. We have carefully reviewed and edited all text generated or modified by M. and take full responsibility for the final content of this paper. L.Mss were not listed authors.
12

---

Published as a conference paper at ICLR 2026
A DERIVING THE COST-SENSITIVE THRESHOLD
We provide self-contained derivation of the gate in Eq. (3.1) and its odds form Eq. (A.2) from Bayes-risk minimization under asymmetric costs.
Proposition -Baes--ppingg gate under asymmetric costs), Let paeed = PreYne==1 Xc) and Paccept Pr[Yacopp1 Xe). Assume costs OFa > for proposing when Yaccope (false alarm) and CPN 0 foror staying silent when help is needed and would be accepted (missed help), with correct decisions costing zero. Then Bayes-optimal decision is
OFA CEa Daeed CpN
INTERVENE
Dircept
2 r(Pand); T(Paea))
(A.1)
Equivalently (odds form),
Parcege
Print CFN
(A.2)
Paccept
OFA
Proof. Condition on X₂ and write expected costs for E {int, sil}. Intervening pays only on false alarm:
E|Cost int, Xx] (1 Paceept) OFA.
(A.3)
Staying silent pays only on missed help; effective term scales with Pheat.
E[Cost| a = sil, X Pacceps (Preed CPN).
(A.4)
Choose the action with smaller conditional expectation. Intervene iff
(1 Precege CPa
Paecegt Paeed CEN,
which rearranges to Eq. (A.2) and hence to Eq. (A.1)
CFA
Comparative staties.
From (Pecos) OPA Ppe
CEACEN
θT
< 0,
0,
< 0.
Opned (CrA + (FN)
0CFA
OCFN
Thus higher burden CpA tightens the gate, while higher risk OPn or higher need Pieed relax it.
B AUDBC (AREA UNDER A-BURDEN CURVE)
For each evaluation event let qe E [0, 1] denote the model's estimated need probability Preed. Pe. E (0, 1] the estimated acceptance probability qnccept, and EN the number of candidate proposals. Given a fixed false-alarm cost CFA > 0 and false-negative CFN >> O, the dynamic threshold used by our gate is
CEN q
<yec; CFA. CFN) CFa + CFN qe
which is the default " "odds" implementation in our code I An intervention is triggered for under cost CeN iff
LGeN (c) = 1[p- 2 T(q:; CFA, CFN) A The > 0). For set & of N evaluable events, the normalized burden and expected benefit Crn
1 B(CeN) = N Eloex(e)
U(CPN) = [cpx(() Pe.
We sweep CpN over predefined grid fcu HK (environment variable AUDBC_CEW (EFCCCCNNDRD)) with CEa fixed (env COST_FA), obtaining points (B(cu), (c.)).. After removing duplicates and sorting by B, we compute
AUDBC by trapezoidal rule on [0, 1J2:
K-1 Σ
(B+t B&) Un + Unir
AUDBC ~
[0. 1].
where B, = B(ca) and Un = UCa). Higher AUDBC indicates better benefit- burffet-budenn trade-off under cost- sensitive gating.
Thhe implementation flag AUDBC_TAU_ IMPP defaults "odda we also support "bayes " variant, but all reported results "odds".
13

---

Published as a conference paper at ICLR 2026
C EVALUATION PROTOCOL AND JUDGE VALIDATION DETAILS
ensure the reliability of our automated evaluation, conducted rigorous validation study comparing our LLM-as-Judge protocol against both human experts and alternative LLM panels.
C.I JUDGE POOLS AND HUMAN ANNOTATION
We enlisted 7 volunteers from diverse backgrounds (including undergraduate and graduate students in pro- gramming, marketing, and business) annotate sampled subset of 229 events. The human "Gold" label was determined via majority vote.
For automated evaluation, we validated our primary judge (the majority vote used the main paper) and further defined two distinct LLM pools to test for robustness:
Pool (Frontier): GPT-4o, Claude 3,5-Sonnet, and Gemini 2.5-Flassi
Pool B (Open): DeepSeek-R1, Kimi-K2, and Llama-3-70B.
C.2 AGREEMENT ANALYSIS
measured the alignment between our automated judges and human consensus using Agreement Rate, Cohen's x, and Matthews Correlation Coefficient (MCC).
As shown in Table 6 our primary ensemble judge achieves substantial agreement with human labels (w = 0.71). Furthermore, the high consistency across different pools (Pool A Pool B к 0.76) confirms that the evaluation signal is stable and not dependent idiosyncratic biases of specific model families.
Comparison Setting
Support
Agreement
Cohen's x
MCC1
Main Protocol Verification
Our Ensemble Judge vs. Human
229
89.1%
0.71
0.71
Cross-Mode Consistency Checks
Pool (Frontier) vs. Human
229
90.8%
0.75
0.76
Pool (Open) vs. Human
229
89.1%
0.71
Pool vs. Pool
229
91.2%
0.76
0.77
Table 6: Comprehensive Human-Model and Inter-Model Agreement Analysis. "Our Ensemble Judge" refers the majority vote protocol (GPT-40, Claude-3.5-Somee DeepSeek-R1) used for the main results. The high agreement scores (>> 0. 0.70k) across all settings confirm the reliability of the automated evaluation protocol.
C.3 ROBUSTNESS OF PERFORMANCE GAINS
critical concern LLM-- evaluation whether performance improvements are artifact the specific judge employed. To address this, we evaluated the relative performance of PRISM against the strongest baseline, DeepSeck-Re R1, across all three evaluation regimes.
Table 7 summarizes these results. While absolute F1 scores fluctuate- consistent with observations by Ye al. 2024 regarding varying judge severity-the relative improvement (AF1) remains highly robust. PRISM consistently outperforms the baseline by approximately +3.0 F1 points across evaluators, including human judges.
D STATISTICAL SIGNIFICANCE ANALYSIS
To verify that the performance gains of PRISM its teacher model (DREPSEEK-B) R)) not due random chance, we conducted statistical significance testing using bootstrap resampling (N 10,000 iterations). Table the 95% confidence intervals for the performance deltas and the corresponding p-values.
The analysis confirms that PRISM's primary contribution suppressing false alarms to improve precision is highly statistically significant (p < 01.01), demonstrating robust hehavior distinct from the teacher model's base capabilities.
14

---

Published as a conference paper at ICLR 2026
Judge Pool
Method
F1-Score
F1 (Gain)
DeepSeck-R1
82.92%
Pool (Frontier)
PRISM (Ours)
86.05
+3.13%
DeepSeek-RI
83.28%
Pool (Open)
PRISM (Ours)
86.61
+3.33%
DeepSeek-R1
82.05%
Human (Majority)
PRISM (Ours)
84.85 %
+2.80 %
Table 7: Performance comparison across diverse judges. PRISM maintains a consistent advantage over the DeepSeek-PP R1 baseline regardless of whether the evaluator is proprietary model ensemble, an open-source ensemble, or human experts.
Metric
DeepSeek-R1
PRISM (Ours)
Imp.
95% CI
P-value
Recall f
98.12%
98.88%
+0.76%
[-0.18%, +1.75%]
0.115
Precision f
72.35%
77.05%
+4.70%
[+2.35%, +7.05%
< 0.001
Accuracy f
72.69%
76.39%
+3.43%
[+1.10%,
+5.76%
0.004
False-Alarm +
27.64%
22.94%
-4.70%
[-7.1%%, 2.228%
< 0.001
F1-Score
83.28%
86.61%
+3.33%
[+152%, +5.15%
0.002
Table 8: Statistical significance of PRISM vs. DeepSeek-R1. The results confirm that the improve- ments in Precision, Accuracy, False-Alarm rate, and Fl are statistically significant (p 0.05), while Recall remains statistically equivalent.
E DETAILED ANALYSIS OF INFERENCE STRATEGY
In this section, we provide the detailed numerical results supporting the inference strategy ablation discussed in 44.3 We analyze the sensitivity of the slow-margin hyperparameter ó and the robustness of the system across varying cost ratios (CPA CFN).
E. IMPACT OF SLOW MARGIN
We investigated the impact of the confidence margin threshold on the efficiency-accuracy trade-off. The results are detailed in Table and visualized in Figure S
Table 9: Sensitivity sweep for slow-margin 8. A margin of Ā = 0.1 strikes the optimal balance, achieving the highest FI and AUDBC. Increasing further 0.15 yields diminishing returns in accuracy while significantly increasing latency and token consumption.
Margin 5
Ree f
Pre t
Acc t
FA 4
F1
AUDBC
Tokens
Lat
(Fast)
100%
71.08%
71.07%
28.92%
83.09%
79.26%
176ms
0.05
100%
74.11%
74.10%
25.89%
85. 15.12%
81.19%
518
178ms
0.10
100%
78.81 %
79.33%
21.19%
88.15%
82.72%
541
196ms
0.15
100%
74.57%
75.20%
25.43%
85.43%
80.58%
563
230ms
E.2
ROBUSTNESS ACROSS COST RATIOS
To verify that our method is not over-fitted to specific cost setting, we conducted 2D grid sweep over the cost ratio CFA Crn (ranging from eager 1:4 to conservative 1.2:1) and the slow margin 6. Table T0 presents the full results.
15

---

Published as a conference paper at ICLR 2026
243
U Mi
0.50
CJAC.N-
233
0.11
210
0.79
200
C.R..
M.C.
022
180
500 052 608 210
DJU 932 014
0.92
6.04
1350
UJS B30
0.32
6.2 2334 1.22
9.04
0.03
0.95
D.0
Sbernargh Adw
skwirarge
skwwragge Auoe
(a) AUDBC vs.
(b) FA Rate vs.
(c) Latency vs.
Figure 5: Impact of Slow Margin. Increasing the margin initially boosts performance by correcting hard examples, but latency grows linearly. 0.1 the "sweet spot".
Table 10: Robustness Grid Sweep. Performance metrics across varying user cost profiles (CFA CrN) and slow margins. PRISM consistently benefits from the margin-based gating (ò 0) across diverse cost settings, maintaining high F1 and AUDBC scores.
Cost Ratio
Margin (6)
Rec 1
Pre 1
Acc 1
FA
FI
AUDBC
Tokens
Lat
(Cpa Cex)
0
100%
67. 67.24%
78.93%
32.76%
83.64%
79.97%
532
198ms
0.05
100%
70.56%
81.24%
29.44%
82.13%
81.28%
547
210ms
1:4
0.1
100%
70.41% %
80.22%
29.59%
83.63%
81.57%
562
2 216ms
0.15
100%
69.72%
81.57%
30.28%
82.96%
80.92%
$90
244ms
0
100%
71.08%
71.07%
28.92%
83.09%
79.26%
510
1 176ms
0.05
100%
74.11% %
74. 10%
25.89%
85. 15.12%
81.19%
518
178ms
1:2
0.1
100%
78.81%
79.33%
21.19%
88.15%
82.72%
196ms
0.15
100%
74.57%
75.20%
25.43%
85.43%
80.58%
563
230ms
0
98.89%
72.18%
80.34%
27.82%
83.34%
78.98%
517
182ms
0.05
97.96%
73.62%
81.50%
26.38%
84.07%
80.42%
532
186ms
0.1
98.88%
72.36%
80.35%
23.64%
83.96%
81.97%
534
183ms
0.15
98.85%
74.26%
78.64%
25.74%
83.42%
80.89%
568
217ms
0
86.18%
75.87%
79.90%
24.13%
82.39%
77.31%
504
176ms
0.05
90.34%
75.13%
78.44%
24.87%
81.28%
78.97%
515
188ms
1.2:1
0.1
89.62%
76.83%
79.59%
23.16%
83.74%
79.34%
528
178ms
0.15
86.45%
74.73%
78.52%
25.27%
83.45%
78.18%
572
197ms
F ROBUSTNESS AND CALIBRATION ANALYSIS
In this section, we provide detailed supporting evidence for the robustness of PRISM across three dimensions: cost sensitivity, domain generalization, and calibration stability.
F.1 SENSITIVITY TO COST RATIOS (Cra : CFN)
Different deployment scenarios require different trade-off's between helpfulness (Recall) and intrusiveness (False Alarm). We swept the cost ratio CPa Cex from 1: to 1.2 1 evaluate how the system adapts varying penalties for interruptions versus missed opportunities.
As shown Table II PRISM demonstrates strong adaptability without requiring retraining:
1. Performance Trade-off: As the penalty for false alarms increases (moving from 1 4 to 1.2 1), the model effectively shifts its operating point. Precision improves (70.41% -> 76.83%) and the False-Alarm rate drops significantly (29.59% 23.17%, while Fl scores remain stable (> 83%) across settings.
2. Efficiency Gains: Stricter constraints (1.2 1) also lead to greater efficiency. The "Slow Rate" (percentage of calls routed to heavy reasoning) decreases from 15.45% to 9.96%, resulting in reduced token consumption and lower latency (216ms 177ms).
16

---

Published as a conference paper at ICLR 2026
OPa
CPN
Ree 1
Pre f
Ace f
FA U
F1 f
AUDBC 1
Slow
Tokens
Lat
1:4
100%
70.41%
80.22%
29.59%
83.63%
81.57%
15.45%
562.43
216.16 16
1:2
100%
73.07%
81.36%
26.93%
84.44%
82.22%
13.47%
538.73
191.82
1:1
98.88%
72.36%
80.35%
23.64%
83.96%
81.97%
11.15%
533.79
182.90
1.2:1
89.62%
76.83%
79.59%
23.17%
83.74%
79.34%
9.96%
528.34
177.67
Table 11: Impact of Cost Ratios. PRISM adapts to stricter penalties for false alarms (1.2 by naturally increasing precision and reducing chatter, maintaining high F1 scores across the spectrum.
F.2 DOMAIN GENERALIZATION
To evaluate whether PRISM learns generalized proactive semantics rather than memorizing -ppecic patterns (e.g., coding syntax), we tested the model on the held-out "Writing" domain (Out-of- of-Domain) while the training data was dominated by "Coding" tasks (In-Domain).
Table 12 compares the performance of the raw RDC-SFT model against the full PRISM pipeline (with post-hoc Temperature Scaling, denoted as "After-T").
Domain
Stage
Recall f
Precision
False-Alarm +
F1-Score t
AUDBC
In-Domain
Raw
98.80%
74.77%
25.23%
85.12%
86.43%
(Coding)
After-T
98.80%
74.77%
25.23%
85.12%
86.43%
Out-of-Domain
Raw
95.83%
71.39%
28.61% %
81.37%
82.15%
(Writing)
After-T
98.37%
73.37%
26.63%
82.63%
84.66%
Table 12: Domain Generalization & Calibration Effect. While the raw model shows a perfor- mance dip on the unseen Writing domain, post-hoc calibration ("After-T") effectively recovers Recall (+2.54%) and AUDBC (+2.51%), narrowing the gap between in-domain and out-of-domain performance.
Analysis. The results demonstrate strong generalization. Even without domain-specific training, the student model maintains high F1 (> 82%) writing tasks, suggesting the learned representations for Preed and Pacegpi semantically robust. The raw model tends be slightly under-confident miscalibrated on -ui-t--diitributionn data (lower Recall of 95.83%). Temperature scaling plays crucial role here: corrects probability distribution, recovering Recall to 98.37% % and significantly boosting AUDBC to 84.66%. This confirms that our proposed calibration mechanism acts as a stabilizer against domain shifts.
F.3 CALIBRATION STABILITY AND QUALITY
We provide two-fold analysis of our post-hoc calibration strategy: first, quantifying the improvement in proba- bility estimation (RCE and Brier score), and second, stress-testing the decision policy against hyperparameter drift.
Calibration Quality. Table compares the calibration metrics hefore and after applying temperature scaling. We optimized separate temperatures for Pneed (Tree = 0.5) and Paccet (Tacepp = 0.7) on the validation set. The results show significant reduction in Expected Calibration Error (ECF) for both signals (21.53% + 12.16% for need, 13.19% + 7. 7.42% for accept). This confirms that the performance gains in Ablation D stem from reliable probability estimates, ensuring the model is merely "lucky" specific threshold but genuinely functionally calibrated.
Sensitivity Hyperpun perparameeter Drift. ensure our method not brittle, we evaluated performance under suboptimal calibration parameters. We perturbed the temperature I (scaling logits) and the decision bias (shifting thresholds) around the optimal grid. Table 14 demonstrates strong robustness:
1. Moderate Drift (Robust): Within reasonable range (T [0.75, 1.25]. e [-0.15, 0.15)), Fl scores remain consistently high (> (2.8%)..8))) The "Flip Rate" (percentage of decisions changed relative to baseline) remains moderate (< 15%). indicating the core policy is stable.
2. Extreme Scenarios (Failure Modes): Only under "super optimistic" (T 0.5. +0.3) or "super pessimistic" (T 1.5, -0.3) settings does performance degrade significantly. This confirms that
17

---

Published as a conference paper at ICLR 2026
Reliability (Need)
Rellability (Accept)
1.0
Empincal accuracy
0.6
0.4
Perfect
Perfect
Need (Before)
Accept (Before)
Need (After)
Accept (Ater)
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Predicted probability
Predicted probability
Figure 6: Reliability Diagrams. Visualization of calibration performance for Need (left) and Accept (right) probabilities. The green curves (After Scaling) align significantly closer the perfect diagonal than the orange curves (Before Scaling), visually confirming the ECE improvements.
Temperature
ECE
Brier
Task Metries
Setting
Taeed
Tace
Need
Acc
Need
Acc
F1-Score
False Alarm +
Before Scaling
1.0
1.0
21.53%
13.19%
17.95%
18.58%
84,85%
25.99%
After Scaling
0.5
0.7
12.16%
7.42%
14.71%
18.20%
85.70%
25.12%
Table 13: Calibration Quality Metrics. Post-hoc scaling significantly reduces Expected Calibration Error (ECE) and Brier scores for both gating signals, creating more reliable foundation for decision- making.
while calibration helps, the underlying RDC-SFT model provides stable enough signal that precise tuning is beneficial but not strictly mandatory for viability.
Scenario
T
Recall
Pre t
FA
F1 1
AUDBC t
Flip
Baseline
1.0
98.80%
74.77%
25.23%
85.12%
86.43%
0%
Moderate Drift (Robustness Region)
Moderate
0.75
0
98.95%
74.41 %
25.89%
84.78%
84.47%
6.25%
Moderate
1.25
97.89%
75.58%
24.42%
85.56%
87.02%
4.46%
Moderate
1.0
+0.15
99.98%
71.87%
28.13%
83.34%
81.57%
12.08%
Moderate
1.0
-0.15
96.67%
76.22%
23.78%
85.30%
87.34%
14.83%
Moderate
0.75
+0.15
99.14%
73.69%
26.31%
82.84%
80.58%
14.29%
Moderate
0.75
-0.15
97.63%
75.21%
24.79%
85.67%
85.83%
13.52%
Moderate
1.25
+0.15
98.73%
73.33%
26.67%
83.21%
83.24%
15.17%
Moderate
1.25
-0.15
95.58%
77.67%
22.33%
82.97%
80.32%
17.85%
Extreme Scenarios (Failure Modes)
Super Optimistic
0.5
+0.30
100%
53.18%
46.82%
68.67%
42.57%
68.34%
Super Pessimistic
1.5
-0.30
37.49%
88.74%
11.26%
38.89%
45.64%
72.95%
Table 14: Sensitivity Analysis under Parameter Drift. The model maintains high F1-Score (> 82%)
across a wide grid of calibration parameters. Flip denotes the percentage of decisions changed relative the baseline decision
18

---

Published as a conference paper at ICLR 2026
G TRAINING CONHIGURATION
--- Base model
2
Qwen3-83
crust_rrmote_code)
template: qwen
5
cutoff_len: 4096
6 dataset: proactive agenttteeeeeeett__eeoo rrc__dd__iiee distille
max max_samples: 100000
train srain_en_prrompt false
ver__eeiee_.tran_bacclliiees
10
graaleet_.ccummlatioostteeeo 4
11
learning_rate: 1.0e-5
12 num_train_epochs: 3
13 ir_scheduler_type: cosine
14 warmup_ratic: 0.1
15
pure_bf16: true
19


Reviews:
[{"review_id": "i4VcnPPGSd", "reviewer": "Reviewer_M8Sr", "summary": "This paper presents PRISM (Proactive Risk Sensitive Intervention with Slow mode Margin), a framework for training proactive agents, with the goal of minimizing false alarms. The method introduces two key features: \n\n(1) The probability that the user *needs* help, and that this help will be *accepted* are estimated separately and combined into a score to decide whether to act or not. Action is only allowed when the score is higher than a certain threshold.\n\n(2) These estimations are first computed with a *fast* model. If the score is close to the threshold, then the estimations are recomputed with a *slow* model, making decisions \"more thoughtful\" near the decision boundaries.\n\nThe paper evaluates PRISM against proprietary and open source LLMs without specific proactive training, and against agents from the ProactiveBench paper (ICLR 2024). In the experimental evaluation, the PRISM agents show improved performance with respect to the baselines.", "questions": "Q1. In Table 3 you show the ablation on slow-mode triggering. To my understanding, the expected trend should be that the more the slow-mode is used, the better the performance. However, this is not the case. Can you explain why?\n\nQ2. This is a request rather than a question: in lines 072 - 073, you mention that you will release the code and protocols upon acceptance through an anonymous repository. Once accepted, there is no need (or point) for anonymity, but an anonymous repository is the perfect tool to ensure reproducibility during the review process. Please release them now so they can be part of the review process.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. Principled solution to a relevant problem: separating need and acceptance probability, and allowing more compute budget for difficult decisions makes sense and is shown to empirically work. It also makes the decision more easily interpretable.\n\nS2. Ablation studies help see the contribution of each of the critical optimizations introduced by PRISM.", "weaknesses": "W1. Using LLM as judge is an important limitation, especially when measuring inherently human responses, as are the usefulness and acceptance of the agent's help.\n\nW2. The empirical evaluation lacks a discussion on statistical significance. Some of the reported small improvements may be explained by the inherent randomness of the problem. For a clear example, see lines 300-304.\n\nW3. Both in learning and deployment, considering a fixed cost of false alarm and false negative may be overly simplistic. In real situations, these costs are different for each scenario, and may also evolve over time. These nuances are not captured by the presented framework.\n\nW4. The paper does not include the code and data to reproduce the experiments.\n\nW5. The paper needs an editorial pass to polish certain presentation aspects. From more to less important:\n- The citation of (Barto 2021) is wrong in two different ways. The reference in SIAM Review 6(2):423 is not by Andrew Barto, but by Volker H. Schulz. Secondly, the reference in question is a review of the book \"Reinforcement Learning: An Introduction\", written by Richard Sutton and Andrew Barto. I guess what the author really intended originally is to cite the book, and not the review of the book.\n- It is a good practice to, whenever available, cite the peer-review version of a paper. This applies to Jimenez et al. 2023 [ICLR 2024], Lu et al. 2024 [ICLR 2024], Wang et al. 2022 [ACL 2023], Zhang et al. 2024 [EMNLP 2024].\n- There are missing references (indicated as ??) in line 542\n- Title of the appendix is \"EDeriving\" instead of \"Deriving\" (line 540)\n- Missing space after period in line 206.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47fH1dunDP", "reviewer": "Reviewer_PzXB", "summary": "This paper presents PRISM, a framework for proactive agent intervention that combines a cost-derived acceptance gate with margin-gated slow reasoning and calibration-aware distillation. The central contribution is formulating proactive assistance as a cost-sensitive selective decision problem where the agent estimates two calibrated probabilities, $p_{\\text{need}}$ (whether help is needed) and $p_{\\text{accept}}$ (whether help will be accepted), and intervenes only when $p_{\\text{accept}}$ exceeds a dynamic threshold derived from asymmetric costs.\n\nPRISM invokes resource-intensive, slow reasoning only near the decision boundary through a margin parameter $\\delta_{\\text{slow}}$, concentrating computation where it is most likely to change the outcome. Training employs Risk-Decision Consistent distillation (RDC-SFT) that aligns supervision with the deployment gate and costs. On ProactiveBench, PRISM achieves F1 of 86.61\\% (vs.\\ 83.28\\% for DeepSeek-R1), precision of 77.05\\% (vs.\\ 72.35\\%), and reduces false-alarm rate to 22.94\\% (vs.\\ 27.64\\%), all at near-saturated recall (98.88\\%).", "questions": "1. Can PRISM maintain its advantage over baselines across *all* judge pools if you re-score 100–200 events with two different judge pools (swap in/out frontier or open models) and report inter-judge agreement (Cohen kappa or Matthews correlation) plus F1 deltas?\n\n2. On a small human-labeled subset (50–100 events) with labels for y_need and y_accept, what do calibration metrics (ECE, Brier) for (p_need, p_accept) and task metrics (F1/precision/false-alarm rate) show? If miscalibration appears, after temperature scaling, how do those metrics change?\n\n3. If you sweep (C_FA, C_FN) over a 2D grid, how do AUDBC, false-alarm rate, and expected latency vary as a function of the slow-margin delta_slow, and does this trace a clear benefit–burden Pareto frontier?\n\n4. How much does each signal contribute under these ablations: (a) gate on p_accept only (set p_need = 1), (b) gate on p_need only (set p_accept = tau(p_need)), and (c) uncalibrated vs calibrated (p_need, p_accept) (pre vs post temperature scaling)?\n\n5. If you re-run the ProactiveAgent pipeline using PRISM’s gate and margin-gated slow reasoning but *without* RDC-SFT (i.e., keep their original model, replace only the decision policy), how much of the gain comes from the gate versus the distillation?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The decision-theoretic gate is properly derived from Bayes-risk minimization under asymmetric costs (Appendix, Proposition 1). The comparative statics are correct: the threshold tightens with higher false-alarm cost and loosens with higher need probability. This is the *right* way to control the benefit-burden tradeoff, and it connects cleanly to the classical selective prediction and reject-option literatures (Elkan 2001; Chow 2003; Geifman & El-Yaniv 2017). The paper appropriately positions its novelty: while cost-sensitive thresholding is well-established in classification, the *synthesis* (i.e., decomposing intervention into calibrated need/acceptance probabilities, deriving an adaptive threshold, and coupling it with margin-gated slow reasoning) is novel for proactive LLM agents and goes beyond heuristic triggers in prior systems (Lu et al. 2024). The false-alarm cost modeling is particularly well-motivated: recent HCI studies document that unsolicited AI assistance can “backfire,” threatening users’ self-perception and eroding trust (Liao et al. 2016; Zhang & Zhu 2025). PRISM’s risk-sensitive gate directly addresses this documented failure mode.\n\n2. Invoking slow reasoning only operationalizes Russell & Wefald’s (1991) value-of-computation framework; compute where it can change the action. This echoes recent dual-process architectures (e.g., Dualformer’s auto-mode that adaptively switches between fast and slow reasoning, Su et al. 2025) but tailors it to the intervention decision through counterfactual checks. Empirically, this improves F1 by 3.66 points over fast-only and 2.59 points over always-slow while achieving the highest AUDBC (87.73 vs. 79.84 and 76.63). This is consistent with selective classification theory and the need for well-calibrated probabilities.\n\n3. The RDC-SFT objective explicitly rewards calibrated estimates and penalizes squared errors on ( (p_{\\text{need}}, p_{\\text{accept}}) ), then filters training data by a ranking score (Eq. 3.2). This approach aligns with recent advances in training for calibration (e.g., Damani et al. 2025’s RLCR, which augments rewards with proper scoring rules like Brier score to jointly improve accuracy and confidence reliability). The schema-locked distillation that decouples the response policy from the intervention gate is a practical innovation that addresses deployment concerns: it enables post-training threshold tuning without retraining, yielding auditability. Ablations (Table 4) confirm that RDC-SFT dominates vanilla SFT (F1 86.61 vs. 76.09), weighted-SFT (86.61 vs. 80.59), and DFT (86.61 vs. 76.09), and that the policy improvement depends on calibration (Table 2: dynamic thresholds on uncalibrated probabilities underperform fixed thresholds). Temperature scaling (Guo et al. 2017) is acknowledged as a simple fix for miscalibration.\n\n4.  PRISM achieves large margins over public baselines: +20.14 F1 points and −22.78 % relative reduction in false alarms vs. strong proactive baselines. Crucially, the distilled student *beats* its teacher (DeepSeek-R1) on precision and false-alarm rate at similar recall, which is non-trivial and highlights the value of structured distillation and gating beyond raw reasoning capacity. The evaluation uses a consistent LLM-as-judge protocol with majority voting, and the paper commits to open-sourcing code and models on a public benchmark (ProactiveBench), supporting reproducibility.\n\n5. The framework exposes a compact set of knobs that move the benefit-burden frontier in predictable ways without retraining. This addresses a practical gap: in deployed systems, model outputs and product-level rules are often entangled, making it hard to audit or tune behavior post-deployment. By decoupling the intervention gate from the response policy, PRISM enables independent threshold adjustment—a design principle that echoes earlier mixed-initiative systems (e.g., Horvitz’s work on modeling interruption costs and user attention, 1999–2003) but brings it into the LLM era. This is valuable for product deployment and aligns with best practices from calibration and abstention literature.", "weaknesses": "The main weakness is using LLM-as-judge evaluation, because it's fragile. Labels for $(y_{\\text{need}}, y_{\\text{accept}})$ rely on DeepSeek-R1 with majority voting over three judges (including GPT-4o, Claude 3.5-Sonnet). This is vulnerable to bias, variance, and rubric drift. A recent comprehensive survey (Ye et al.\\ 2024) documents that LLM judges exhibit position bias, inconsistency, and can give overly assertive assessments with unjustified confidence; they are ``not yet ready to fully replace human evaluators, especially for complex, high-stakes judgments.'' While the paper acknowledges this limitation (Section 6) and notes that baselines face the same constraint—making PRISM's gains meaningful \\emph{within} the competitive evaluation context—the issue remains: PRISM's gains might partially reflect overfitting to the judge's biases rather than genuine alignment with user preferences. The paper would be substantially strengthened by robustness checks (see Questions below). This is a field-wide challenge, and the requested checks align with emerging best practices (multi-LLM consensus, human-in-the-loop validation).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kNH5ZJtPrU", "reviewer": "Reviewer_Y4dz", "summary": "This paper introduces PRISM, a decision-theoretic framework for cost-sensitive, selective intervention in proactive language agents—systems that must decide whether and when to offer help. Instead of always reasoning deeply or using fixed heuristics, PRISM dynamically balances benefit and burden by modeling both need and acceptance probabilities and gating interventions through an adaptive, cost-derived threshold.", "questions": "1. Have the authors validated PRISM’s benefits with real user judgments or pilot human studies? Since proactive timing and acceptance depend on user perception, it would be valuable to know how well the LLM-judge proxy aligns with human acceptance and satisfaction. Could the authors provide evidence of correlation between judge labels and human ratings?\n\n2. The approach relies heavily on calibrated $p_{need}$ and $p_{accept}$. How does PRISM handle calibration drift in deployment or across domains? Would an online or adaptive recalibration mechanism (e.g., temperature scaling on recent feedback) help sustain performance?\n\n3. How sensitive are the results to the chosen false-alarm and missed-help cost ratios and slow-margin width?\n\n4. Could the authors provide explicit measurements of runtime latency and token usage compared to always-slow or baseline proactive agents?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper offers a fresh formulation of proactivity as cost-sensitive selective intervention, introducing a principled gating mechanism that jointly models user need and acceptance under asymmetric costs. The dual-process “fast vs. slow” reasoning and the festina lentedesign elegantly merge decision theory, calibration, and selective computation.\n\n\nPRISM provides a practical and general framework for controllable, efficient proactive agents. Its decision-theoretic approach and selective reasoning strategy have broad relevance to risk-sensitive and resource-aware AI.", "weaknesses": "Real user studies or human acceptance feedback is not conducted, which is important in order to show that PRISM’s gating truly improves user experience and perceived helpfulness. \n\nThe framework assumes stable calibration of $p_{need}$ and $p_{accept}$, but no experiments examine how PRISM performs under domain shift or calibration drift.\n\nIn the ablation study, the precise effect of threshold width and cost ratios are not systematically explored.\n\nThe paper could engage more deeply with related concepts such as metareasoning and risk-sensitive RL to situate PRISM’s novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)