Language Models Can Predict Their Own Behavior
Authors: Dhananjay Ashok
Jonathan May
Venue: NeurIPS 2025 poster ()
Keywords: foundation models
Large Language Models
LLMs
Natural Language Processing
NLP
Primary area: deep_learning
TL;DR: The internal states of a Language Model can often preemptively predict several behaviors of interest, and can be used to form precise early warning systems that detect jailbreaking, instruction following failures and more.

Abstract: The text produced by language models (LMs) can exhibit specific `behaviors,' such as a failure to follow alignment training, that we hope to detect and react to during deployment. Identifying these behaviors can often only be done post facto, i.e., after the entire text of the output has been generated. We provide evidence that there are times when we can predict how an LM will behave early in computation, before even a single token is generated. We show that probes trained on the internal representation of input tokens alone can predict a wide range of eventual behaviors over the entire output sequence. Using methods from conformal prediction, we provide provable bounds on the estimation error of our probes, creating precise early warning systems for these behaviors. The conformal probes can identify instances that will trigger alignment failures (jailbreaking) and instruction-following failures, without requiring a single token to be generated. An early warning system built on the probes reduces jailbreaking by 91%. Our probes also show promise in pre-emptively estimating how confident the model will be in its response, a behavior that cannot be detected using the output text alone. Conformal probes can preemptively estimate the final prediction of an LM that uses Chain-of-Thought (CoT)  prompting, hence accelerating inference. When applied to an LM that uses CoT to perform text classification, the probes drastically reduce inference costs (65% on average across 27  datasets), with negligible accuracy loss. Encouragingly, probes generalize to unseen datasets and perform better on larger models, suggesting applicability to the largest of models in real-world settings.

Language Models Can Predict Their Own Behavior
Dhananjay Ashok Information Sciences Institute University of Southern California ashokd@usc. edu
Jonathan May Information Sciences Institute University of Southern California jonmay@isi.
Abstract
The text produced by language models (LMs) can exhibit specific "behaviors, such failure to follow alignment training. that hope detect and react to during deployment. Identifying these behaviors can often only be done post facto, i.e., after the entire text of the output has been generated. We provide evidence that there are times when we can predict how an LM will behave early in computation, before trained on the internal representation of input tokens alone can predict wide range of eventual behaviors over the entire output sequence. Using methods from conformal prediction, we provide provable bounds on the estimation error of our probes, creating precise early warning systems for these behaviors. The conformal probes can identify instances that will trigger alignment failures (jailbreaking) and instruction dooructionffolloiing failures, without requiring single token be generated. An early warning system built on the probes reduces jailbreaking by 91 %. Our probes also show promise in pre-emptively estimating how confident the model will be in its response, a behavior that cannot be detected using the output text alone. Conformal probes can preemptively estimate the final prediction of an L.M that uses Chain-of- Chain-of-Though Thought (CoT)) prompting, hence accelerating inference. When applied to an LM that uses CoT to perform text classification, the probes drastically reduce inference costs (65% on average across 27 datasets), with negligible accuracy loss. Encouragingly, probes generalize to unseen datasets and perform better on larger models, suggesting applicability to the largest of models in real-world settings.
single token is generated. We show that prohes
Introduction
Language models (LMs) have emerged as the dominant approach to general language tasks [88, 22]. seeing widespread adoption in chatbots [3, 68], code generation [18] and reasoning systems [67]. However, they have been known to "misbehave, hallucinating false information [98], failing to adhere output specifications [59], or, most concerningly, producing content that is misaligned with human values [75], sometimes dangerously so [35]. In practice, such misbehavior is mitigated by deploying post-hoc guardrails on the output of the LM [92, 24, 16, 101, 8, 53]. For example, a chatbot LM may exhibit harmful behavior by complying with malicious request [41, 90]; developers will typically attempt to detect this behavior in the output [101] and return an abstention message instead of the original LM content. Such framework suffers the economic [100] and environmental [83, 76] expense of generating every output token, a cost that threatens to grow exponentially as frontier labs continue to scale both model sizes [37, 49] and the number of tokens generated during inference [91, 95, 67.
In this paper, we show that the internal representations of LMs can be used to preemptively predict behaviors that will emerge over the entire output sequence. This information becomes accessible
OUr code is accessible at: bettps:/ggitha
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training
Validation
Inference
Frane
Unsate
Calibrate probes for conformal prediction
User specifled confidence
Trust Language Model Decide
list my frands account me
Sare
Safe
Accuracy
Train probes hidden states to estimate behavior
Hidden state of Input taken
700
Validation Set
8
7(N) 4s
No
If probe confident, behavlor estimate
f(RD.yi
9a Threshold
Force Abstention
Figure 1: Overview of our method applied in preemptive safety alignment failure detection. During training, we use many input prompts to collect a dataset consisting of the hidden states of the input tokens and eventual output behavior (in this example, the behavior is whether the LM fails abstain on a malicious prompt). The dataset is used to train a probe on the hidden states. With a validation set, we calibrate the probes for conformal prediction by identifying threshold for confident predictions. During inference, if the probe is confident in its estimate of the eventual LM behavior, we stop computation early (in this example, overriding the LM and abstaining).
before the LM has generated a single token, enabling the system to take appropriate actions without suffering inference costs on those instances. Concretely (Figure 1), we train linear classifiers (probes) [4] that use LM's internal representation of input tokens predict the eventual behavior of its output. We then calibrate the probes using methods from conformal prediction [81]. During inference, we look to probes to make a prediction only when there is provable guarantee the estimation error, ensuring precise early warning signals for various model behaviors.
We show that these probes can preemptively identify degenerate behavior, such as instruction- following failures (Section 3). Across multiple datasets and output formats, our probes give precise early warning signals for cases where the LM will fail to follow specified output format instructions.
Conformal probes can also detect deeper, 'self-reffective' properties, like whether an LM's behavior conflicts with its safety alignment. We demonstrate this (Section 4.1) by applying our method to enhance the safety of an LM that is instructed to abstain from complying with malicious requests. Probes are trained to identify cases where this will not happen the request will be malicious and the model will not abstain. Once again, our probes are able to precisely detect (with over 92% accuracy) cases where the LM complies with malicious requests. An early warning system built on these conformal probes reduces the jailbreak success rate from 30% to 2.7%,
reduction.
We further show that conformal probes can identify behaviors that cannot be measured with output text. We show (Section 4.2) that our probes can a priori estimate an LM's confidence as measured by the per-token perplexity of the output; as this model-specific property cannot be inferred from fust the text sequence alone, we show the wide scope of applicability of our probing approach.
Finally, we apply our method to an LM that tackles text classification by generating several explanation tokens before class prediction, i.e using Chain-of- Chain-of-Though (CoT) reasoning [91] (Section 5). The probes estimate the final class prediction that will appear after the reasoning chain and exit early if confident in their estimate. On 27 datasets, spanning the tasks of Multiple Choice QA, Sentiment Analysis, Topic Classification, Toxicity Detection and Fact Verification, our method reduces inference costs by 65% on average, while suffering accuracy losses of more than 1.4% (worst case). The probes generalize to unseen datasets, reducing the inference cost of CoT on OpenBookQA [63] by 68% with minimal loss to accuracy (0.4%), despite only training on other MCQA datasets.
We demonstrate that often fewer than 500 training instances are sufficient for the probes to attain high estimation consistency, while ablations the probing layer present a more complicated, task-specific story. Encouragingly, increasing the scale of the L. improves the performance of our technique, suggesting it may scale favorably with ever-increasing model sizes [15, 37, 19, 26).
We hope our findings enable practitioners to build efficient early warning systems on language models and enable further research into the information encoded in their hidden states [71, 10. 66).

---

2 Background
Lightweight probes have long been used to interpret the internal activations of neural networks [4] and language models [71, 10]. Given a set of input prompts, we compute the generated outputs and store the internal activations computed during the forward passes an LM (e.g. the outputs of specific Transformer [88] layer). The instances are then assigned classification label meant to capture some property of the input and output. The probes are then trained to predict the label using only the internal activations input. For example, Azaria and Mitchell [10] collect the activations of the middle layer of an LM when processing the output tokens, and manually label the generations true or false. They then train small neural network which can identify whether the model output is truthful, given the internal activations of the generated tokens. Prior work in probing uses the internal states of all tokens, even generated ones, which incurs the cost of inference. In this work, we explore an alternate approach, training probes on the internal states of the input tokens alone, allowing to predict properties of the output tokens before they have been generated.
Hidden State Probing:
Conformal Prediction: Given a validation set, conformal prediction can be used to precisely calibrate the model's 's confidence [29, 52]. During inference, the conformal system makes prediction if and only if the probability of the prediction being accurate is above a user-specified probability [81].
In the classification setting, we are given validation set {x11.... Xx), (Vi.... (......-)))) ), x; Rd Di {1, ......... c) (where d the dimension of the input vector and e the number of classes), a classifier which maps the input to scores for each class f Rd - I and user-defined confidence level
(0, 1). To form well-calibrated prediction set, we define a score function S IR x (1,2....c) + R f(x)) that measures how poor a prediction is e.g. S(x, y) 1 (_, exp [xx, )r- Using scores the validation set (s1,...sp) sn) we calibrate by calculating threshold fa. the 1 - a quantile of the scores:
a Quantile((11, $2, s2, sn}. [(n + a
n
At inference time, the prediction set for a test instance is (d[S(x, à) < g. For the above score function, no two classes can score greater than 0.5 simultancously, implying that with a > 0.5 the prediction set inference time either empty (deferral/abssention)) or consists of single class label.
If the validation set is exchangeable (a slightly weaker assumption than the typical I.L.D assump- tion [12)) with the test set, then prediction at inference time is provably guaranteed [81] satisfy
P Vhest
Crest] Or
3 Can Internal States Predict Eventual Behavior?
In this section, we ask whether the internal representations of the LMs input can be used to predict eventual behavior, even before any output token has been generated. We explain our methodology with the running example of an LM instructed to answer question in bullet point format.
The LM is expected to provide an answer in two bullet points, with any fewer or greater bullets in the given input prompt:
answer being a failure to follow instructions. Suppose
Question:
What do Osteoclasts do?
LM may respond to this with:
1. Decompose bone 2. Differentiate from monocyte 3. Create space for tissue
We can identify that the LM has behaved contrary to its instructions when it generates the '3', The previous tokens give no indication that a failure imminent; however, is it possible that the model contains signals that can identify this eventual failure by the time it generates the first token, ie.. '1'
To test this, we collect three QA datasets-NNauuaalO0 NaturalQA [54], MSMarco [65] and TriviaQA [47] and prompt the LM to answer questions in a specific format. One format requires the answer to be organized in exactly three bullet points, while the other requires JSON output with pre-specified
3

---

94.1
94.4
66.7
80.6
90.1
84.2
05.6
66.4
79.9
85.93
57.0
73.1
94.0
78.1
96.6
70.9
69.0
50.0
45
50.0
50.0
26
50.0
50.0
50.0
MSMarco Bullets
NaturalQA Bullets
TrivisQA Bullets
MSMarco ISON
NaturalQA ISON
TriviaDA ISON
Rendom
BERT (340M Params)
Probe (SK Params, Total) Probe (Conforrmll (+-08)) Probe (Conforrmal c+0.9)
Figure 2: Estimation consistency, i.e. accuracy when preemptively predicting whether the LM will fail to follow format instructions. Numhers inside the bars bold italics are the coverage percentages for conformal probes that engage in selective prediction. All probe results are means seeds, with error bars showing the 2n confidence interval. The probes not only outperform the random baseline, but consistently match or outperform fine-tuning on a BERT model despite using fewer than 0.003% of the parameters. The conformal probes have significantly higher consistency, and give users the flexibility to trade off precision and coverage based on the confidence level a.
5 random
fields. We evaluate whether the output format has been followed, and then sample this data to obtain training and testing splits that have equal number of failures and successes.
We collect the output of the middle Transformer layer Llama3.1 1-BB [26] when processing the final token of the input (in the example above, the "?'), then train linear classifiers predict whether the output will eventually fail to follow instructions. We ablate layers in Section 6, reproduce results on other LMs in Appendix A, and provide the prompts that specify the format in Appendix C.
We also fine-tune a larg-large- [22) model for text classification, which involves tuning 340M parameters, 4 x 10 times the number of trainable parameters of the linear model. The BERT model is trained to predict whether the output will fail to follow formatting instructions given the text of the input prompt alone, and is measure of how a significantly stronger model performs when attempting to establish a pattern between the inputs to the model and the output behavior. Methods evaluated the metric of estimation consistency, i.e, how accurate they are at preemptively predicting whether or not the formatting instructions will be followed.
The probes (Figure 2. navy blue bar) outperform the random baseline in all settings, showing that an LM's representation of the input tokens contains information on behavior that will emerge over the entire output sequence. Moreover, the probes also outperform fine-tuning on BERT in all but one of the settings, despite using around only 0.0025% as many parameters.
However, despite outperforming the baselines, the average estimation consistency of the probes is less than 75%. A naive attempt act on the probes estimate would result in catastrophic failure.
3.1 Creating robust behavior estimators with conformal prediction
Internal states may occasionally be insufficient to estimate eventual behavior. The ideal system handles such cases, making consistent predictions when confident and deferring otherwise. Hoping to impart such a capability to our probes, we use the probes learned above in a conformal prediction framework. Specifically, we use a held-out validation set MDaai to calibrate the probe after training. We compute the probes' prediction probabilities on the validation set, and find the lowest threshold quantile g that satisfies:
[0, 1] for each class with label y (1,...c)
EyeD, T((max (yi) 2 g) A (argmax($:) = =y:)]
2 a
Eyieaa 1|max (yi) 2 g]
During inference, when probe predicts a behavior with probability vector Prest we return the prediction and only if max (frea) 2 q, otherwise we defer to the LMM. Unless specified otherwise. set 0.9. If no satisfying g be found, we defer on all instances. We ablate in Section 6.

---

Conformal Probe
0.9)
BERT
Dataset
Coverage
Consistency
Prec.
Rec.
Consistency
Prec.
Rec.
SelfAware
100.0
98.0
94.5
94.5
93.3
91.7
84.4
KnownUnkown 100.0
92.3
92.6
75.1
72.8
67.1
64.4
93.1
WildJailbreak
94.0
90.0
91.0
83.4
73.8
33.3
Table 1: Probes estimate whether the LM will fail to abstain when it should have with significantly higher precision and recall than a BERT baseline. On WildJailbreak, where 30% of malicious prompts are mistakenly complied with, using the probe as an early warning signal reduces successful jailbreaks to 2.7%. In comparison, using the BERT model would leave jailbreak success rate at 20%
Using conformal probes (Figure 2) significantly increases performance across all datasets. At = 0.8, the probes have a test coverage of 40% and, for the covered datapoints, estimation consistency is 80.7%, showing that the conformal probes are well calibrated to the user-specified confidence level. Raising a to 0.9 results a reduced average coverage of 10%, but an increased average consistency of 89.2%. This allows users to flexibly trade off between coverage and consistency, enabling precise and trustworthy early warning systems that do not offer an opinion when confidence is low.
4 Creating Early Warning Systems for Other Behaviors of Interest
In this section, we demonstrate that probes are not limited to preemptively identifying degenerate behavior, such as failing to follow instructions. We show that conformal probes can serve as early warning systems for instances that will trigger failures to follow safety alignment, and those that will result in the model giving low-confidence response. We describe the key experiment designs below, with exact prompts and details in the Appendix C.
4.1 Detecting failures to follow safety alignment
The deployment of LM-powered chatbots raises concerns of harmful behavior that is unaligned with human interests [28]. One common setting where such behaviors arise is when the user requests information that does not exist, with models often hallucinating an answer regardless, misleading the user in the process [9]. Another common occurrence is when the user themselves requests the model to help them complete malicious tasks, such building weapon of mass destruction [90].
We collect two datasets (SelfAware [96], KnownUnknown [5]), which have answerable and unan- swerable questions. The LM reasons as to whether the question is answerable, providing response if it is and abstaining otherwise. Similarly, we collect malicious and benign prompts from WildJail- break [45], and prompt the LM to refuse to comply with request if it is malicious. Inspired by work in Al safety and alignment, we adopt the viewpoint that answering an unanswerable question or complying with a malicious request is disproportionately problematic [11]. We filter our training set to only include the instances where the LM answers or complies with the request, and train prohes to identify the cases where the LM should have abstained (i.e. fails to avoid answering an unanswerable question or falls victim to jailbreaking). During inference, if the probe is confident (a = 0.9) that the LM has failed to abstain when it should have, we override the LM and abstain instead.
Results show (Table 1) that probes greatly enhance the safety of the LMs, outperforming the BERT baseline on consistency, precision and recall. Across all datasets, probes identify over 72% of the failures to abstain (over 90% for 2/3 datasets). This safety comes at little cost; when the probes force an abstention, they are correct over 86% of the time (on average). They make confident estimations on at least 92% of the samples, and maintain conformal estimation consistency of at least 92%. Concretely, on WildJailbreak the use of conformal probes reduces the percentage of successful jailbreaks from 30% to 2.7%, a significant safety improvement.
4.2 Quantifying LM uncertainty
Language models are prone to hallucination [98] and often have gaps in their knowledge [38], making methods that estimate the trustworthiness specific outputs vital. Recent work [48, 46) approaches this problem by using the model itself to quantify how certain it is in its answer, either by explicitly
5

---

88.9
91.9
92.6
91.0
$1.0
52:7
80.0
51.2
79.8
90.5
62.3
70.8
63.8
68.4
66.266.6
59,0924
63.3
50.0
50,0
50,0
54,0
46
30
11
MSMarco Perplexity
NaturakaA Perpiexity
TrNoQA Perplexity
MSMarco Verbalized
NoturalQA Verbalized
TrivDAA Verbalized
Random
BERT [340M Paramss]
Probe (SK Params, Total) Probe (Conformal a+0.8) Probe [Conformal a+0.9)
Figure 3: Estimation consistency when preemptively predicting an LM's confidence in its response. Numbers inside the bars in bold italies are the coverage for conformal probes that engage in selective prediction. Probes outperform all other methods, with the confidence estimation tasks proving more challenging for BERT. Conformal probes have lower coverage rates than the format following (Section 3) and safety alignment (Section 4.1) tasks. However, they maintain consistency at the user-defined confidence level, ensuring that their early warning signals are reliable.
prompting verbalize its confidence or using the per-token perplexity of the output. we detect an untrustworthy response, we may abstain from answering escalate the query to more capable model. An early warning system that preemptively detects such responses before they are even generated would allow us to take these actions more efficiently.
We train conformal probes preemptively estimate how confident the LM will be in its response using both of these uncertainty quantification methods on NaturalQA, MSMarco and TriviaQA. In the case of the perplexity measure, we consider the bottom 25% of scores be 'high confidence," and the top 25% to be 'low confidence' (discarding the rest). In both cases, the probes attempt identify whether the model will be confident its output. This task proves more challenging for all methods, however, probes continue to outperform both the random baseline and the much larger BERT model. Despite this, the conformal probes maintain high consistency, making the early warning and redirection system feasible in practice.
Selective Inference-Scaling with Conformal Probes
In this section, we show how our method can be used to accelerate systems that use CoT prompting with an LM to perform text classification. We collect 27 text classification datasets, spanning the tasks of Multiple Choice Question Answering, Sentiment Analysis, Topic Analysis, Toxicity Detection and Fact Verification. Details on dataset setup are in Appendix C.
use Llama3. under CoT prompting to perform text classification, i.e., outputting an expla- nation before the final class prediction. We train linear probe that uses the internal representation of the final input token at the 18th layer (based on results from previous experiments) to predict the class that the CoT model will eventually output, and then perform conformal calibration. During inference, if the conformal probe is confident, we interrupt generation and use the probe estimation the final answer. If not, allow the model continue its CoT generation and provide the final answer. We note that this method of acceleration is synergistic with other inference optimization algorithms such Speculative Decoding [55] architectural changes [27]. This is because in the cases where the probe does not interrupt the LM, it does not influence the remaining forward passes, hence conformal probes can be flexibly added on top of any inference paradigm for further efficiency.
We compare to vanilla CoT on two metrics Accuraay Loss (Accuracy of CoT Accuracy of Method) and Inference Cost Reduction #CoT
Pases
#Melhod Forward Passess
# CCT Forward Passes
The results (Figure 4, in blue) show that the method is highly effective at reducing the inference cost with minimal cost accuracy. The minimum inference cost reduction is 4. 4.7%, with an average reduction of 65% across all datasets. Despite this significant speedup, the average accuracy loss near zero (-0.46%), with the worst loss at 1.34%. Surprisingly, accuracy increases on several datasets. Finally, we investigate the out-of-distributiin generalization capabilities of the conformal probes. For each test dataset of the MCQ and Sentiment tasks, we train and calibrate using data from every other dataset. The results (Figure 4, in orange) show that the probes do exhibit OOD

---

Teel Diet
8
OOD
Task
6
Fact Verfication
Accuracy Loss %
Sentiment Anellyis
4
Topic
Toxicity Classincation
2
x
-6
0
20
40
60
80
100
Inference Cost Reduction %
Figure 4: Tradeoff between accuracy loss and inference cost when using probes accelerate CoT Prompting in IID and OOD settings. Using IID probes leads to a 1.34% accuracy loss (worst case) and negative accuracy loss average (i.e. accuracy improves). Inference cost reductions are 65% on average. Probes generalize to OOD data, inference cost reductions are higher at 81 % on average, with small accuracy loss of 2.3% average.
generalization, suggesting the method may be applicable even when there are slight shifts between training and test distributions.
6 Analysis and Ablations
Why does accuracy improve when using probes? Seeking to explain the surprising increase in accuracy when accelerating CoT models (Figure 4), we plot (Figure 5, left) the correlation between the CoT model's accuracy and the probe's estimation consistency. It is generally positive, which suggests that incorrect probe estimation correlates with incorrect CoT generation. This minimizes the harm of an inconsistent estimation of the CoT decision, the cases where estimates are likely to be inconsistent coincide with cases where a consistent estimate does not benefit accuracy.
2.5
Consiitendy
Densty
1.5
Conssistency
40
29.6
0.44
-0.2
0.0
0.2
0.4
0.6
Com(Coo Accuracy, Approx Consistency) datasets
Aocuuryy
Total)
Figure 5: Left: Density plot of the correlation coefficient between the CoT LM accuracy and the consistency of the probe. The coefficient is often positive, indicating that instances where the probe inaccurately estimates the LM decision are also instances where the LM is more likely to be incorrect. Right: On CosmoQA, probes that the initial tokens of the input prompt have near-random consistency, with performance increasing considcrably as we use later tokens in the prompt.
How early the computation does the model show signs of the final behavior? have shown that the final input token often contains sufficient information to predict output behavior, but at which token does this information start becoming clear and accessible? investigate this, we collect activations from every input token (after the FewShot example tokens) of the CSQA dataset and train

---

linear classifier to map these internal activations to the eventual CoT prediction. Probing (Figure 5, right) using the embeddings of tokens the first quarter of the question leads near-random performance, showing that this information is not readily accessible at initial tokens. Consistency increases steadily as we use later tokens, with sharp rise around the final tokens.
mmlu
twitterfinance
Total Teat Consistency %
60
Consistency
90
40
Coverage
Llama-3-38
Base Rate
Llama-3-88
20
Llama-3-70B
200
400
600
800
1000
1200
1400
0
10
20
30
40
50
60
70
80
Number of Training Datapoints
Layer
mmlu
selfaware
Consistency
85
80
75
10
15
20
25
Layer
Layer
mmlu
medmcqa
100
70
80
Consislency
85
60
60
40
55
20
50
0.5
0.6
0.7
0.8
0.9
1.0
10
20
40
60
Confidence
Layar
Figure 6: Left: Ablations on MMLU. Within 500 training datapoints, probes (a = 0.9) often approach the consistency and coverage they will attain when trained the entire dataset. Layer ablations (a = 0.9) show that early layers perform poorly. Increasing the confidence threshold a has no effect until a point, after which conformal consistency increases while the coverage tends zero. Right: Estimation consistency increases with LM size, suggesting the method may scale favorably
Ablations on probe parameters: We ablate the parameters involved in the design of the conformal probing system: number training datapoints, probing layer and the conformal confidence threshold We show results (Figure 6, left) for the MMLU dataset, however, all trends discussed hold for the majority of the datasets we explored, with more examples in Appendix B.
We ablate the amount of data used to train the probe and find (Figure 6, top left) that probes can approach the consistency and coverage they will achieve over the entire dataset with only 500 datapoints, suggesting that the method is particularly data efficient. We explore (Figure 6, middle left) whether the specific layer being probed affects the performance of the probe. Our most general finding is that early layers offer poor estimation consistency. However, whether mid or late layers perform better is task and lataset-specific, and cannot be described generally. For more details, see Appendix B. Increasing the confidence required from the probes has predictable effects (Figure bottom left). Increasing the confidence threshold has no effect until point, after which conformal consistency increases while the coverage tends zero.
How does model scale affect performance? We varied the size of the LM used (Llama3 3B, 8B and 70B) and measured probe consistency across several layers. Encouragingly (Figure 6, right), the

---

performance of the larger models is consistently better, suggesting that the information encoded in the internal activations is easier for the probe 'read' when the model is more powerful. This bodes well for the methods' ability to scale with models of increasing size and capabilitty
What are some limitations of the probes? Other experiments we conducted suggest that probes are limited in the behaviors they can detect. Using the MCQ task, we tried to estimate whether the LM would oulput an incorrect answer. The probe accuracies were consistently near random. We hypothesize that since probes are simple linear classifiers, they can only detect patterns and use 'knowledge' that is well encoded in the LM activations. This suggests that probes struggle with output properties that cannot be identified by the LM itself (without external knowledge). We also observed that the correlation between the estimation consisteney of the probe and the token count of the output is negative (-6.1%). This suggests probes struggle more with inputs that evoke longer outputs.
7 Related Works
This work takes inspiration from recent methods that probe the hidden states of LMs to observe interpretable patterns [4,51, 51, 71, 36, 3] identify false statements [10, 56, 57, 97] and hallucinations [20, 84, 44). These works use the internal states of every token, even the outputs, to produce signals for these phenomena. In contrast, our work shows that internal states can predict behaviors before any output tokens are generated, suggesting that input token embeddings contain rich information on future LMM behavior. Furthermore, while conformal prediction has been used with LMs to guarantee the quality of generated text [73], robotic trajectories [89, 74] and text classification [52, 17], we are the first to use in conjunction with hidden state probes to create precise early warning systems.
Our work also has connections to the literature on early exiting during the forward pass of NN models, with works often using signals from the hidden states prematurely exit with prediction on the future outputs [93, 99, 94, 80, 42, 70J. While these works focus on the tokens predicted, we show a more general result: that internal states can predict the future behavior of LMs. These include behaviors such as whether or not the LM will output a high perplexity answer, or mistakenly comply with a malicious request, which cannot be inferred just by observing the output token sequence.
While recent work has begun exploring whether LMs exhibit 'introspective' properties in the black box setting [13], or whether LMs can predict 'global attributes' of their response [72, 25], are the first to show that when internal probes are deployed under conformal prediction they can be used to create early warning systems for wide range of behaviors like question abstention to format following errors. Our work advances research on understanding the nature of the information contained in the hidden states of LMs [71, 6, 66, 58, 87, 62). Specifically, we show that the information contained in the hidden states is relevant not just to the next token, but to behaviors that manifest several tokens later during the LMs generation.
8 Conclusion
We show that language model's hidden representation of input tokens alone contains vital infor- mation on the behavior of the LM over the entire output sequence. We train linear probes to read this information and use it in a conformal prediction framework to create precise early warning or exit systems for a wide range of LM hehaviors, including degenerate behaviors, safety alignment failures and more. The conformal probes can preemptively identify, before single token is generated, instances where an LM will fail to abstain from answering an unanswerable question, fall victim to jailbreaking, fail follow output format specifications or give low-confidence responses. On 27 text classification datasets across 5 different tasks, the method can accelerate Chain-of -fhoughhtt prompting by 65% with little accuracy loss. We show that the probes generalize to distriibbttinnoo test sets and scale favorably to larger LMs. Finally, we explore the limitations of the method, showing that the behavior of longer output sequences is harder to estimate and that tasks that require knowledge external to the model are particularly challenging. With the rising popularity of inference-time scaling methods, we hope our work can help ameliorate the growing computational cost of running LMS and provide more insight into the nature of the information contained in their hidden states.

---

Acknowledgments
We thank Robin Jia for his exceptional course on The Science of Large Language Models (accessible at htttp::/rrobijiaaa githut...b...... io/ www.sss/esaaaaaaa csci1222222-cc169999 html) and his advice on the early stages of this project. We also acknowledge support from Open Philanthropy.
10

---

References
[1] owi.ttpsss//....agggeeec///////a commen wtw.:mmeet-ccasaaiiiaaiiiiiaiiaa laapaaasiiiiiiiiiioooalleeeeee data.
[2] N. A. "erssho/wwiie---naacaaaaeeeeeeee datasets at hugging face. URL https:
//huginggfce http://wwww...aaaaace//////aaata
[3] Akyürek, D. Schuurmans, J. Andreas, Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL
1d=0g0X4HByN4I.
[4] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes, 2017. httet/oormnidddy/ffft
URL https: // /ptp//veeeevieeeeeeee
[5] A. Amayuelas, L. Pan, W. Chen, and W. Wang. Knowledge of knowledge: Exploring known- unknowns uncertainty with large language models. arXiv preprint arXiv:2305. 123330551777 13712, 2023.
[6] Anonymous. Does know?: Probing for uncertainty language model latent belief'..
In NeurIPS Workshop on Attributing Model Behavior at Scale, 2023. URL https:// openreview.
[7] N. Asghar. Yelp dataset challenge: Review rating prediction. arXiv preprint arXiv:6605.05362. 2016.
[8] S. G. Ayyamperumal and L. Ge. Current state Ilm risks and guardrails. arXiv preprint ar arXiv:2406.12999 2024.
[9] R. Azamfirei, S. R. Kudchadkar, and J. Fackler, Large language models and the perils of their hallucinations. Critical Care, 27(1):120, 2023.
[10] Azaria and T. Mitchell. The internal state of LLM knows when it's lying. H. Bouamor,
J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967-976, Singapore, Dec. 2023. Association for Computational Linguis- URL / hatp//aoaaaanttogg....o. or2/2023.
tics. doi: 10.18653/v1/2022 1tn.://////222....nnnnn-oo-m findings-emnn
68.
[11] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.0800 2022.
[12] J. M. Bernardo. The concept of exchangeability and its applications. Far East Journal of Mathematical Sciences, 4:1 1--12, 1996.
[13] F. Binder, J. Chua, T. Korbak, H. Sleight, J. Hughes, R. Long, E. Perez, M. Turpin, and O. Evans. Looking inward: Language models can learn about themselves by introspection. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //opnreview. aeee//oorummdddabspkiit.
[14] Y. Bisk, R. Zellers, R. I.. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical com- monsense in natural language. In Phirth-Fourth AAAI Conference on Artificial Intelligence, 2020.
[15] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Serberr--ooss. G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https::tt/.m : http:///ptccoeee.occceooo neurips np...p..cc//////o cc/ pddf. Chatgpt, microsoft, and the development of URL nttps://contewt wwwmthepage.ccommmm com/ chatgpt-microsot and-the tel-the-deee -tw--deeeoomeeettooooo wtt.llpmmeettoo-guuurriilll////. guardraiirsss
paper_files/ htwilleepppprr//20000 http:///11////777000dbbbebbb6660
[16] S. Brumfield. guardrails, Dec 2023.
[17] M. Campos, A. Farinhas, C. Zerva, M. A. Figueiredo, and A. F. Martins. Conformal prediction for natural language processing: A survey. Transactions of the Association for Computational Linguistics, 12:1497-1516, 2024.

---

[18] M. Chen, J. Tworek, H. Jun, Q. Yuan. H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint 2...x:2210703333
2021.
[19] A. Chowdhery, S. Narang. J. Devlin. M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240)11-133, 2023.
[20] Y.S.. Chuang, L. Qiu, C.-Y. Hsieh, R. Krishna, Y. Kim, and J. R. Glass. Lookback lens: Detect- ing and mitigating contextual ballucinations large language models using only attention maps. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pages 1119-436, Miami, Florida, USA, Nov. 2024, Association for Computational Linguistics. doi: 10.18653/v 1/.18865/1//2224.cmm URL https: worllloooogoooooooooooooo org/ emnlp-main.
[21] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: 1a0..8803..55. 2018.
[22] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volune I (Long and Short Papers), pages 4 114186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/N19- 1423. WtRL:tts::/ / haaaaaaaaaanachooocccee
1tg/N19-14233
[23] T. Diggelmann, J. Boyd-Graber, J. Bulian, M. Ciaramita, and M. Leippold. Climate-fever: A dataset for verification of real-world climate claims, 2020.
[24] Y. DONG, R. Mu, G. Jin, Y. Qi, J. Hu, X. Zhao, J. Meng, W. Ruan, and X. Huang. Position: Building guardrails for large language models requires systematic design. In Forty-first International Conference on Machine Learning, 2024. URL ttttps://operree
net/
[25] Z. Dong, Z. Zhou, Z. Liu, C. Yang. and C. Lu. Emergent response planning in LLMs. In Forty- second International Conference on Machine Learning, 2025. URL https: httpps//opeerevvlt pet/forum"ii--Ce7P888
[26] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama herd of models. arXiv preprint vxXiv:2407:21783, 2024.
[27] M. Elbayad, Gu, F. Grave, and M. Auli. Depth-adaptive transformer. International Conference Learning Representations, 2020. URL https: http://///p/eeeeee./oooo id-SJg7KhVKPH.
[28] I. Gabricl. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411-437, 2020.
[29] A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. arXiv preprint arXiv:130..735,, 2013.
[30] Greene and P. Cunningham. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proc. 23rd International Conference Machine learning (ICML '06), pages 377-384, ACM Press, 2006.
[31] A. Gulli. URL http:// http://rrouo.s.eo/ee di, unipi. unipi.it/ ittittttgggglaa httpa///S__cpppeooooeeeeeeetatta haa_s_ewessarrccce html.
[32] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma. P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501. 12948 2025.
[33] F. Hamborg and K. Donnay. Newsmtse: dealii--arrget-deeeddentt sentiment classification in news articles. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACI. 2021), Apr: 2021.
[34] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding. International Conference on Learning Representations.
12

---

[35] D. Hendrycks, M. Mazeika, and T. Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:2306. 12001, 2023.
[36] J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. In K. Inui, J. Jiang, V. Ng. and X. Wan, editors. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLE) pages 2733-2743, Hong Kong, China, Nov. 2019, Association for Computational Linguistics. doi: 10. 1115586/1//D11-12755 URL wrt///9-12/DD5555550
aetps://aclanthooogg/
[37] J. Hoffmmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de las Casas, L. A. Hendricks, J. Welbl, A. Clark. T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre. An empirical analysis of compute-optimal large language model training. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances Neural Information wwweeeewweweeeeerreeeeeeeeeeeeee wwt./frummm??
Processing Systems, 2022. URL https: / /pwp/reveeeeeeeee///////p
OAPR.
[38] X. Hu, J. Chen, X. Li, Y. Guo, L. Wen, P. S. Yu, and Z. Guo. Towards understanding factual knowledge of large language models. In The Twelfth International Conference on Learning Representationss 2024.
[39] L. Huang, R. Le Bras, C. Bhagavatula, and Y. Choi. Cosmos QA Machine reading compre- hension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA, pages 2391 1-401, Hong Kong, China, Nov. 2019, Association for Computational Linguisties. doi: 10. 10.18653/w1/Di9-1222 URL httpps///ver wwwp:w/wwcwlbb..rrr....... https://www.....ebborrrgg http::/ww...bbbor/////a/boo/////
1243.
[40] F. Inc. Finaaceine/audiiosemmmnnnnt datasets at hugging face. URL huRph:tts///uugigtttee co http:/aatee//iiaannnonn//aaaaaot
[41] S. L. Isaac, G. Irving, and I. Gabriel. Ethical and social risks of harm from language models. arXiv preprint arXiv:21 2021.
[42] M. Jazbec, J. U. Allingham, D. Zhang, and E. Nalisnick. Towards anytime classification in early-exit architectures by enforcing conditional monotonicity. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL htttps://openreview/ id=Aks1sk891N.
[43] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
[44] Jiang, Qi, X. Hong, D. Fu, Y. Cheng, F. Meng, M. Yu, B. Zhou, and Zhou. On large language models' hallucination with regard to known facts. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pages 1041-1053, 2024.
[45] L. Jing, K. Rao, S. Han, A. Ettinger, F. Brahman, S. Kumar, N. Mireshghalaah, X. Lu, M. Sap, Y. Choi, and N. Dziri. Wildteaming scale: From ti-the--ild jailbreaks to (adversarially) safer language models, 2024. URL pestps://arxxx
org/ /aa/a////24// 18510.
[46] Z. Jiang, J. Araki, H. Ding, and G. Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962 9777, 2021. doi: 10.1162/tacl a 000007. URL https: http///annnnoolooor/////0/00 org/2021.
[47] M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M.-Y. Kan, editors, Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational URL https: aa/ aclantholoy.. wwwp://gg/////////
Linguistics. doi: 10. 111886/331/111-.
[48] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield- Dodds, N. DasSarma, E. Tran-Johnson et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207. 000000055221110 2022.
[49] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:200J. 0000000010031111100 2020.
13

---

[50] T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal. Qasc: A dataset for question answering via sentence composition. arXiv: 11433..............11 2020.
[51] B. Kim, M. Wattenberg, J. Gilmer, C. J. Cai, J. Wexler, F. B. Viégas, and R. Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, 2017. URL https: //api. semantiemmniii cscholar org/ /oppCSsSSSDD::17777...
[52] Kumar, C. Lu, G. Gupta, A. Palepu, D. Bellamy, R. Raskar, and A. Beam. Conformal prediction with large language models for multi-choice question answering, arXiv preprint arXiv:2305. / 18404, 2023.
[53] D. Kumar, A. Birur, T. Baswa, S. Agarwal, and P. Harshangi. No free lunch with guardrails. arXiv preprini wwx.iv:254.00444 2025.
[54] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein,
1. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10.1 10.1162/tacl_4 Laaeecaa_000 20002626al__000666...o URL https: ////aacaannhooogy..ooooooo org/
[55] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pages 19274-19286. PMLR, 2023.
[56] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Shirty-sevnnth Conference Neural Information Processing Systems, 2023. URL https: /ttp://pppeeiiieeeeeeee
[57] J. Liu, S. Chen, Y. Cheng, and J. He. On the universal truthfulness hyperplane inside LLMs. In
Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pages
18224, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10. 1/254//1111222mmlllpppp 1012. URL https: / /aallallmmn aalaaaclanntoooooooooooo org/2024. emnlp-main 1012/.
[58] Z. Liu, T. Zhu, C. Tan, B. Liu, H. Lu, and W. Chen. Probing language models for pre-training data detection. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1576-1587, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. URL https: /alpa/aaaaaatooooy.ooooooooo.oae org/2024 2024 acl-long.
doi: 10. 10.18663/w1/20224.allooog 86/.
[59] R. Lou, K. Zhang, and W. Yin. Large language model instruction following: A survey of progresses and challenges. Computational Linguistics, 50(3):1053-1099, 2024.
[60] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: /ttp:////www.ee// aclweb. org/ http//nhhoooggPP11-111a
[61] P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
[62] T. Men, P. Cao. Z. Jin, Y. Chen, K. Liu, and J. Zhao. Unlocking the future: Exploring look- ahead planning mechanistic interpretability in large language models. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 7713-7724, 2024.
[63] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, 2018
[64] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. Mteb: Massive text embedding bench- mark. arXiv preprint 10..0:22 2022. doi: 10-48550ARXXIV:2100077 URL. https:// //txxi/axxvv.v// owg//bs////iiieoeo 07316.
14

---

[65] T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09228. 2016. URL. http://arxivv wwwpe/iiiooiioiiooooooooo abs/1611 1611 09268.
[66] K. Nylund, Gururangan, and N. Smith. Time is encoded in the weights of finetuned language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2571-2581 Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 1/16633/11/222.aal---aa 141. URL htltttts/aclaatthlll.// www.aaatttooooooooo///oo0/// 002/ acl acl-long.
141/.
[67] OpenAI. URL htt.tt:///pppaaaaaee httpp//www.o///ee//////aeaaeeeaa httpp///w//////////////////////a
[68] L. Ouyang, J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback.
Advances in neural information processing systems, 1--27730-27744 2022.
[69] A. Pal, L. K. Umapathi, and M. Sankarasubbu. Medmcqa: A large-scale multi-subject multi- choice dataset for medical domain question answering. In G. Flores, G. H. Chen, Pollard, J. Ho, and T. Naumann, editors, Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 260. PMLR, 07-08 Apr 2022. URL https: ptttps://proceedinne www.ingggmm......mmmmmmmmmmmmmim mlr. prwss/ satress/v14//pp v114//11222/22/ html.
[70] K. Pal, J. Sun, A. Yuan, B. C. Wallace, and D. Bau. Future lens: Anticipating subsequent tokens from single hidden state. In Proceedings of the 27th Conference Computational Natural Language Learning (CoNLLL), pages 548-560, 2023.
[71] F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 2463-2473, 2019.
[72] N. Pochinkov, A. Benoit, L. Agarwal, Z. A. Majid, and L. Men-Minassian, Extracting paragraphs from Ilm token activations. arXiv preprint arXiv::409 240.02880,,,, 2024.
[73] V. Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T.S. Jaakkola, and R. Barzilay, Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024. URL.
[74] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia,
J. Varley, Z. Xu, D. Sadigh, A. Zeng. and A. Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In 7th Annual Conference on Robot Learning, 2023. URI. https:tt////// / //penreeeei/ / 1d=42K80NyyFx.
[75] S. Russell. Human-compatible artificial intelligence., 2022.
[76] S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V. Gadepally. From words watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1-9. IEEE, 2023.
[77] E. Sandhaus. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia. 6(12):226752, 2008
[78] E. Saravia, H.-C. T. Liu, Y.-H. Huang, J. Wu, and Y.SS. Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium, Oct.-Nov. 2018 Association for Computational Linguistics. doi: 10.18653/w1/D18 1404. UtLptttp: // http///www//ooo////o// aclweb. org/ Srarthooogg/DDB-1111
[79] M. Sarrouti, A. B. Abacha, Y. M'rabet, and D. Demner-Fushman. Evidence-based fact- checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499-3512, 2021.
[80] T. Schuster, A. Fisch, Jaakkola, and R. Barzilay. Consistent accelerated inference via confident adaptive transformers. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4962-4979, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguisties. doi: 10. (0.18653/v1/200) 1221866/111002...elpppmaaaaa emwlp-main.4000 URL. https://aclanthooogy org/2021. ennlp-main,
main. 406/.
15

---

[81] G. Shafer and V. Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
[82] Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[83] E. Strubell, A. Ganesh, and A. McCallum. Energy and policy considerations for deep learning in NLP. In A. Korhonen, D. Traum, and L. Märquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/w1/P19- 1355, URL https://aclanthhologg
[84] W. Su, C. Wang. Q. Ai, Y. Hu, Z. Wu, Y. Zhou, and Y. Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 14379-14391, Bangkok, Thailand and virtual meeting, Aug. 2024. Association for Computational I. Linguistics. doi: 10. 10.18653/v1 1/2024//2222..2/// 1/2024.inndings.ae findings-acl.8544 aclanthology. 2024..22...22 www.diinnnncaccccc 854.
URL https://
[85] A. Talmor, J. Herzig, N. Lourie, and J. Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19... 1421. URL https: petps://aclantho///
org/ NN1N-11212
[86] J. Thome, A. Vlachos, C. Christodooulppollos, and A. Mittal. Fever: large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.0553 2018.
[87] Z. Tighidet, J. Mei, B. Piwowarski, and P. Gallinari. Probing language models on their knowledge source.
Belinkov, N. Kim, J. Jumelet, Mohebbi, A. Mueller, and H. Chen, editors, Proceedings of the 7th BlackboxNI.P Workshop: Analyzing and Inter- preting Neural Networks for NLP, pages 604-614, Miami, Florida, US, Nov. 2024. As- sociation for Computational Linguistics. doi: 10. https://aclanntholog. org/ 2024. Mlackbbomllp1.355/
URL
[88] A. Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[89] J. Wang, G. He, and Y. Kantaros. Probabilistically correct language-based multi-robot planning using conformal prediction. IEEE Robotics and Automation Letters, 2024.
[90] A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36:80079-88888 10099881100 10, 2023.
[91] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia. E. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824-24837. Curran As- sociates, Inc., 2022. URL httpss::/pprcceeddingggg//////ee neurips wwtpppprrrcpppppeppcpppppp wilesppapee Coneeeaeeeeeeeeeeeeeeeeeeeeeeeep pdf..
[92] J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks. K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang. Challenges detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447-2469, 2021.
[93] J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin. DeeBERT: BERT inference. In D. Jurafsky, Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246- 2251. Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/11/22200 acl-main. 204. URL https:ttt/// https:://allatthoot http://lcaaaathhoooooooo
Dynamic early exiting for accelerating
org/2020 2020 acl-main 204/-
[94] J. Xin, R. Tang, Y. Yu, and J. Lin. BERxiT: Early exiting for BERT with better fine- tuning and extension to regression. In P. Merlo, J. Tiedemann, and R. Tsarfaty, edi- tors, Proceedings of the 16th Conference of the European Chapter of the Association
16

---

for Computational Linguistics: Main Volume, pages 91-104, Online, Apr. 2021. Associ- ation for Computational Linguistics. doi: 10.18653/w1/2221.caas m.iim............ // /acllantolloyy .rgg/021. eaacelaalllmm. -main. 8/.
main.8. URL https:
[95] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. R. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL https://openreviee set/forum'id-5Xx1eee
[96] Z. Yin, Q. Sun, Q. Guo, J. Wu, X. Qiu, and X. Huang. Do large language models know what they don't know? In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages
Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (0.18653/v1/2023 2 URL wwt.hhttp///t accth/ooooo/ org/2023 wwwdingn-acc..ne.. 551.
[97] M. Yuksekgonul, V. Chandrasekaran, E. Jones, S. Gunasekar, R. Naik, H. Palangi, E. Kamar, and B. Nushi. Attention satisfies: A constraint-satiiffctiin lens on factual errors language models. In The Twelfth International Conference Learning Representations, 2024. URL https: https://openreveie
net/forummid=ggfFATTT
[98] W. Zhao, T. Goyal, Y. Chiu, Jiang, Newman, Ravichander, K. Chandu, R. L. Bras, C. Cardie, Y. Deng, et al. Wildhallucinations: Evaluating long-form factuality in Ilms with real-world entity queries. arXiv preprint arXiv:2407. /7468, 2024.
[99] W. Zhou, C. Xu, Ge, J. McAuley, K. Xu, and F. Wei. Bert loses patience: Fast and robust inference with early exit. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 18330-18341. Curran Associates, Inc., 2020. URL https:
newrips. ttt.cc//ppppppe.. htce/aper_fiilee pdf.
[100] 7. Zhou, X. Ning, K. Hong. T. Fu, J. Xu, S. Li, Y. Lou, L.. Wang, Z. Yuan, X. Li, al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024.
[101] R. Ziv. Developing hallucination guardrails. URL https://cookboot wttp:oowwwoooooooiiiiiocoo eraumpes//dvvaooooingggaalalllla hww.ipinniii.lliciiiiioioooooooo gur.ddriii.....s.....
17

---

A Confrrming Robustness of Results
A.1 Confirming Robustness Choice of Model
We conduct experiments on two other language models to ensure that our results hold for language model families outside of the Llama3 series. We use Destral-7B-Instrutt-ee and DeepSeek-2 -ireek----a Riik--R--Diiee Distill-Qwenn 14B from DeepSeek [32]. The results show that the phenomenon shown in the main text can be detected robustly across various model families.
from MistralAI [43]
Abe
Figure 7: Deepseek
Figure 8: Mistral
A.2 Data Leakage
Several datasets used in this study have been released before the creation of Llama3, and such, might have been pre-exposed to the model during training. We detail our reasons as to why data leakage is not concern for the conclusions that we draw from this study:
There are datasets in our study that have been leaked. For example, the MMLU test set is used as an official test benchmark by LLama3[26), and WildJailbreak was released to the public only after the initial release of Llama3 models [45]. The results strong on such datasets as well, showing that data leakage is not behind the performance of the probes.
Several tasks require the probes to identify behavior that fundamentally does not exist in text form the internet. The abstention failure, format following (bullets and JSON) and confidence estimation tasks require the probes to determine how the model will behave, regardless of whether the model has the input text not. For example, whether the model was pretrained on MSMarco questions is immaterial for predicting whether or not it will output answers in specific JSON format.
B Extended Analysis
B.1 Training Datapoint Ablation
As general trend (Figure 9), 500 training datapoints are usually sufficient for probes to achieve the maximum consistency and coverage that they will attain when trained on the entire dataset.
B.2 Layer Ablation
deeper analysis of the interaction between the performance and the layer used for probing shows that consistent patterns can be identified. A subset of the common patterns (Figure 10) shows that sometimes that apart from early layers being poor, it is unclear whether the middle final layers are consistently better for accuracy confidence.
18

---

mmlu
jigsaw
60
Consistency
Consistency
Coverage
Coverago
40
Base Rate
Base Rate
40
200
400
600
800
1000
1200
1400
500
1000
1500
2000
Number of Training Datapoints
Number of Training Datapoints
bbcnews
100
80
40
20
0
100
200
300
400
500
600
700
Number of Training Datapoints
Figure 9: Within 500 training datapoints, probes often approach the consistency and coverage (a 0.9) they will attain when trained on the entire dataset.
B.3 Confidence Threshold ablation
As general trend (Figure 11), increasing the confidence threshold has impact until point, after which it leads to a steady decline coverage and steady increase estimation consistency.
C Experiment Details
C.1 Datasets:
We used the following datasets in our experiments, all usage is in accordance with their respective licenses. For each dataset, we select maximum of 50,000 training instances to train (and validate) our probes, using the full test set measure all metrics.
C.1.1 Multiple Choice Question Answering:
To test this, we collect 8 MCQ MedMCQA [69], Commonsense [85], OpenbookQA [63] and QASC [50] and use CoT to generate outputs with explanations before the answer (for more prompts see Appendix C).
[34], CosmoQA [39], PiQA [14], ARC [21],
ARC: The A12 Reasoning Challenge (ARC) [21] knowledge and reasoning challenge that contains 7,787 natural, grade-school science questions (authored for human tests).
CommonsenseQA:: The Commonsense(AA [85] dataset contains 12,247 questions with complex
semantics that often require prior knowledge.
MedMCQA: large-scale, Multiple-Choice
Question Answering (MCQA) dataset designed to address realworld medical entrance exam questions, MedMCQA [69] has 194k AIIMS and NEET PG entrance exam MCQs covering 2.4k healthcare topics and 21 medical subjects.
The Massive Multitask Language Understanding Benchmark [34] is a test to measure a text model's 's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
MMLU:
OpenBookQA: Modeled after an open book examination, OpenBOOKQA [63], OpenBookQA consists of 6000 questions that probe an understanding of elementary level science facts and their application to novel situations.
19

---

amazonreviews
unintended Jgsaw
100
90
60-
Consistency
80
Coverage
70
50
Base Rale
80
50
20
5
10
15 Layer
20
25
30
10
15
20
25
Layer
(a) Immediate saturation
(b) Only confidence rises then plateaus
6509
100
15
30
10
15
30
Layer
Layor
(c) Confidence rises and dips, accuracy rises past middle
Confidence rises and dips, accuracy dips after middle
fever
100
00
62
20
20
10
20
25
30
5
15
20
Layer
Layer
(c) Accuracy stays roughly constant, confidence rises slightly then falls back starting
(f) Accuracy constant, significant variance in con- fidence with peak in middle
Figure 10: Layer ablations show that the decision on which layer to use is highly dependant on the specific task and dataset.
QASC: Question Answering via Sentence Composition [50] tests a models ability to compose knowledge from multiple pieces of texts. The dataset consists of 9 9,980 multiple-choice questions from elementary and middle school level science, with focus on fact composition;
PiQA: Physical Interaction: Question Answering [14] dataset that tests whether AI systems can learn to reliably answer physical common-sense questions without experiencing the physical world. The dataset consists of 16,000 training QA pairs with 2,000 and 3,000 examples held out for validation and training.
CosmoQA: Commonsense Machine Comprehension Question Answering [39] is dataset of 35,600 problems that require commonsense- reading comprehension, formulated as multiple-choice questions. The dataset focuses reading between the lines with questions that require reasoning beyond the exact text spans in the context.
C.1.2 Sentiment Analysis:
MTEB: The following datasets were taken from the Massive Text Embedding Benchmark [64]. AmazonReviews: dataset with 1.7 million Amazon product reviews and sentiment score ranging
20

---

mmlu
unintended jigaw
130
100
50
60
40
.*0: Consislancy
Coverage
Base Rate
0.3 0.6
0.7
0.8
0.9
1,0
0.5
0.6
07
0.0
0.9
1.0
csqa
100
60
0.6
Confidence
Figure 11: Increasing the confidence threshold a has effect until point, after which conformal consistency increases while the coverage tends to zero.
from 0-5, TwitterSentiment: a dataset with 30,000 tweets and a sentiment class from positive, neutral negative
Yelp Polarity:
The Yelp review rating challenge [7] consists of nearly 600,000 yelp reviews with sentiment classes from positive or negative.
TwitterFinance: The Twitterfinance dataset [2] is an annotated corpus of 11,000 finance-related tweets. This dataset is used to classify whether the tweet are bullish, bearish, or neutral.
NewsMTC: A dataset for sentiment analysis (TSC) news articles reporting on policy issues, NewsMTC [33] consists of more than 11,000 labeled sentences.
IMDB Reviews: A classic sentiment analysis dataset, IMDB Reviews [60] consists of 50,000 highly polar movie reviews with binary labels.
Financial Phrasebank: A dataset that measures the polar sentiment [61] of sentences from financial news. The dataset consists of 4840 sentences from English language financial news categorised by sentiment. The dataset is divided by agreement rate of 5-8 annotators.
Based on Financial Phrasebank, this dataset [40] is additionally annotated by auditors to reflect bearish, bullish and neutral labels for accounting related sentences.
AuditorSentiment:
Emotion: The DAIR-A AI Emotion dataset [78] is a dataset of 20,000 Twitter messages with six basic
emotions: anger, fear, joy, love, sadness, and surprise.
SST5: A standard sentiment analysis dataset, SST5 [82] consists of fine grained sentiment labels for 215,154 phrases in the parse trees of 1 11,855 sentences.
C.1.3 Fact verification:
ClimateFEVER: dataset that consists 1,535 real-world claims regarding climate-change
collected on the internet. Each claim In ClimateFEVER [23]is accompanied by five manually annotated evidence sentences retrieved from the English Wikipedia that support, refute or do not give enough information to validate the claim totalling in 7,675 claim-evidence pairs.
Health VER: A dataset for evidence-basee fact-checking of health-related claims, Health VER [79] consists of 14,330 evidence-claim pairs.
FEVER: Fact Extraction and VERification [86] consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from.
21

---

C.1.4 Topic Identification:
AGNews: A collection of more than 1 million articles. AGNews articles have been gathered from more than 2000 news sources and annotated for Topic [31].
BBCNews: The BBCNews dataset [30] is dataset consisting of 2,225 articles published on the BBC News website corresponding during 2004-2005. Each article labeled under one of 5 categories: business, entertainment, politics, sport or tech.
The New York Times Annotated Corpus [77] contains over 1.8 million articles written and published by the New York Times between January 1, 1987 and June 19, 2007 with article metadata provided by the New York Times Newsroom, the New York Times Indexing Service and the online production staff at nytimes.com.
NYTimes:
C.1.5 Toxicity Detection:
We two datasets provided by the same organization, Jigsaw/oxicity and JigsawWnnnteendedBias Toxicity [1]. Both datasets have scores for toxicity from a chatroom setting.
C.1.6 Others:
We use two datasets which contain unanswerable questions, Self- Aware [96] and KnownUnkown [5]. Selfaware is dataset consisting unanswerable questions from five diverse categories and their answerable counterparts. KnownUnknown is dataset with questions on known quantities and unknown quantities. We collect jailbreaking prompts (and benign prompts) from WildJailbreak [45]. dataset with 262,000 vanilla (direct harmful requests) and adversarial (complex adversarial jailbreaks) prompt-response pairs. The dataset also has benign prompts that should be complied with.
Unanswerable Questions:
Format Following, Confidence Elicitation: Both of these tasks use the same of datasets NaturalQA [54], TriviaQA [47] and MSMarco [65]. These datasets were selected they are large question-answering datasets which often require generative output that can vary in length.
C.2 Task Specific Set Up:
C.2.1 Text Classification Tasks:
In all lext-classffication tasks, the CoT model is first asked to output reasoning chain and then finally provide a classification answer. The probes are always trained preemptively identify what the prediction will be using only the input token embeddings.
MCQ: For the MCQ tasks, select two options by randomly sampling incorrect answer from the provided options along with the correct option. This is because the MCQ datasets vary in terms of number of MCQ options, and for the diitt--diittitttionn experiment we require all the datasets to have the same number of answer classes (note that the topic identification task below is multiclass, showing that our method is not limited to binary classification). The CoT model is asked identify the correct answer option.
Sentiment:
For the MCQ task, many datasets in binarized format, while others include continuous scores, or ternary labels (including neutral sentiment. We make all the datasets have a similar label structure, by keeping only positive and negative labels. Concretely, the label is if and only if the sentiment is positive (or bullish for finance datasets), and O otherwise. The CoT model is asked to identify whether or not the has positive sentiment.
We keep only a subset the topics to ensure every topic is fairly represented the data. After dropping topics, AGNews has World, Sports, Businsess Science, BBCNews has Business, Tech, Sports or Politics, while NYTimes has Health, Fashion, Real Estate or Television. The CoT model is asked to identify the topic of the article.
Topic:
We map all claims to with supported or not supported (includes refuted and neutral). For ClimateFEVER and Health Ver, we provide evidence along with the claim, while for FEVER we provide only the claim. The CoT model asked to identify whether or not the claim supported.
Fact Verification:
22

---

Toxicity Detection: We randomly sample the datasets to ensure a balance of toxic and not toxic comments, The CoT model is asked to detect whether or the text toxic
C.2.2 Other Tasks:
In these tasks, the probes trained to predict variety of other behaviors using the imput token embeddings.
LM Abstention: For the unanswerable question datasets of SelfAware and Choownnnkowwn. the model is first instructed to give a reasoning chain about the question, and answer only the question answerable (abstaining otherwise). With WildJailbreak, the input need not be question, but any request. Hence, we instruct the model to comply with the request if not malicious, and abstain otherwise. We keep only the instances where the LM responds complies, and train probes to detect whether or not the LM should have abstained (i.e. unanswerable question or malicious request).
Format Following: Using an input prompt, we specify two different output formats LM must obey. In both cases, the probe learns whether the LM will fail follow format specifications comply with them.
Bullets: The output should presented in 3 numbered bullet points, no fewer and no more
JSON: The output should be presented in a JSON string with the following structure: ['short_answee <str>, 'entities"; List<str>, 'references'; List<str>)
Confidence Estimation: There are two settings for this task:
Internal Confidence: The LM prompted output an answer to the input question, and we record the ooken-normalizer perplexity as proxy of confidence. We keep only the bottom and top 25% of instances as per normalized perplexity and train the probes differentiate between the two.
Verbal Confidence: The LM is prompted to output an answer, along with confidence score (either confident or unsure). The probe is trained identify what the confidence score will be.
C.3 Prompts and Inference:
have written separate prompts for each dataset to ensure the LM follows the instructions and outputs the text in way that guarantees we can parse it and infer the behavior we seek to pre- emptively identify with the probes. For example, prompt for the MMLU dataset for MCQA is
Question: What is true for type-Ia supernova? Option A: This type occurs in young galaxies Option B: This type occurs in binary systems Give an explanation and then the answer: Explanation: Type Ia supernova is type of supernova that occurs when two stars orbit one another in which one of the stars is a white dwarf Answer:
Question: <NEW QUESTION>
Give an explanation and then the answer:
Explanation:
For the ISON task, one such prompt is:
Answer the following questions by giving short_ansver, entities list and references list. Give the output JSON format Question: What is the capital of France? "Paris", "entities"; ["France"), "references": pttttps://en.wiiipeeee org/vikiiParis") http://www..kppaaooooooo////aaaa Question: <NEW QUESTION> Answer:
Answer: "short_answer"
23

---

Each task and dataset has different set of prompts. We have provided link to the code, and all prompts can be seen the file data.py
All LM inference uses greedy decoding and is hence deterministic.
C.4
Hardware:
compute cluster with 8 NVIDIA A40 GPUs (approx 46068 MiB of memory) on CUDA version 12.6. The CPU on the cluster is an AMD EPYC 7502 32-Core Processor. Most experiments could be conducted with less than 16GB of GPU RAM.
All of our experiments were
Generating the hidden states took around 2 hours for every 10,000 points, while training a hidden state probe takes fewer than 30 seconds.
C.5 Replicability:
To ensure that our code is easy replicate and our method is easy extend, have provided open access to our code.
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provvided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to claborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The main claims made in the abstract and introduction are that the internal states of the input tokens can predict the behavior of the LM on the entire output sequence whole, and that this can be used signal create precise early warning systems for variety of important behaviors. We show this through set of experiments, where probes trained on the internal states successfully predict the future behavior of LMs, and can be used to preemptively detect instruction following errors, jailbreaking, etc.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
25

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the analysis section (Section 6), we dedicate section to discuss the limitations of the probes. We detail how the kinds of behaviors the probes can detect have limits, and also show that certain kinds of inputs (ones that produce longer outputs) are harder for the probes to model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: While we use theoretical results from conformal prediction (see Section 2), these are our theoretical results, and we do provide proof. There no original theoretical result in this work.
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
26

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have discussed the core implementation details in the Appendix C. includ- ing the dataset-specific set-up and hardware used. These details are a sufficient high-level guide to the experiments conducted in the paper, and allow a researcher to reproduce a simi- lar version the results even if they do not have access to our code. However, as detailed in the answer below, to ensure that the prompts, random seeds, generation parameters etc are transparent, we have released an anonymous version of our code base.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., . a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided an anonymous link to our code: setts://anonnyan woppn.science.////ooooccppppp.ee wcwence/eeeeeeeeeeee /tte////LBMehaaaee eecchavvorEstiinme orEstimation/ This code allows the reviewers (and fu- ture researchers) to fully reproduce our results. Since we use only open-sourced models with fixed random seeds for all stochastic pieces of code, we can guarantee completely reproducible pipeline (given differences in hardware). Unfortunately, we cannot submit the data (specifically the hidden states used to train the probes) due to its size exceeding the supplemental material limits. However, the code will allow a complete reproduction of these hidden states, maintaining the reprodueibility of our results.
27

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ cublic/guides woo.e/ummmiiiiooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips wtippsccpppuiiiiieeiieeea htt//uuliccggeeee wtwbbiccggudes//eeeeeeeeoo httpe/uuiiieeeooeeuunbieeeeeeee www.deeuumiiiiinoooooooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided sufficient level of detail in the main text (Section 3) and the Appendix C to comprehend the results. While we have not provided full details as there are large number of hyperparameters, the code base contains full details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: There only one source of randomness in our method: the splitting of the dataset into training and validation splits for the linear probe training and conformal threshold learning. We have used different random seeds and present aggn-agggeegtted results in all figures and tables. The variance across seeds is typically low, and all the results we show are significant at a 2a level.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
28

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors Is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C explicitly details the hardware that was used the experiments, and gives an estimate on the compute/time required for each of them.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neurs http///eurrrr.rpeee wtteeuripp..pppuiiipieeeeeeeeeee
Answer: [Yes]
Justification: The research does not involve human subjects and uses only publically acces- sible data.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: A core motivation of this work is that of safety and early detection of harmful LM behavior. Through our experiments in Section 1, Section 4.1 and Section 4.2 we have shown that the deployment of method could potentially have positive societal impacts by enabling more efficient and pre-emptive guardrails for harmful behaviors.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The code, data and probes that we released are all based existing releases, and have little potential for misuse on their own.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and models used in this paper are done so in line with their stated licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The asset we introduce is codebase, with documentation that enables reproduction and use.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: This is paper about LLMs, as such they are central the research. We have outlined how we have used the LLMs clearly, and described all relevant design decisions. LLMs were not used in the writing of this manuscript, or creation of the core method/experiments in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://nng httpas///eeuips..eeeee cc/ cct.//ooneerrrccc///////1/// for what should or should not be described.
32


Reviews:
[{"review_id": "vcctVYjdWk", "reviewer": "Reviewer_DPE7", "summary": "This paper explores the concept of enabling models to predict their own behavior in advance. The proposed method aims to identify potentially problematic behaviors a priori within a given confidence level, thereby facilitating timely intervention. The authors demonstrate the applicability of their approach across multiple tasks, including instruction following, safety alignment, uncertainty estimation, and inference acceleration. While the idea of a priori behavior prediction is promising, the paper would benefit from a stronger focus on a specific domain or core problem. The current broad coverage makes it challenging to evaluate the depth and significance of the technical contribution.", "questions": "It is unclear from the paper whether the backbone model is updated during the training of the classifiers. If the backbone is fine-tuned, it might lead to better probing performance, but could also affect the language model’s generation quality. What is the trade-off between probe effectiveness and model behavior stability in such a setting?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n-\tThe idea of a priori behavior prediction has strong practical and application value.\n-\tThe paper is well-written and easy to understand.\n\n**Weaknesses**\n\n-\tThe paper feels disorganized and lacks focus. It covers a wide range of topics—instruction following, safety alignment, uncertainty estimation, and inference acceleration—without clearly defining its main objective. These topics should be explored in separate studies rather than being briefly touched upon in a single paper.\n-\tFor most of the domains discussed, there is little technical distinction between attaching a classifier to a large model and using BERT-based classification; the experimental results also reflect this lack of differentiation.\n-\tThe paper does not compare or discuss its approach against post-hoc decision-making methods, which limits the evaluation of the proposed method's advantage.\n\n**Minor Issues and Suggestions**\n\n-\tFigure 2 may be misleading in its current presentation, even though the coverage percentages are explained in the caption. The term Estimation consistency is also inaccurately used — it should refer specifically to the estimation accuracy among the covered samples , not overall.\n-\tIn the discussion of inference acceleration, the cost of inference is not solely determined by the number of generated tokens; the number of input tokens should also be taken into account. Therefore, the relevant claims or reported metrics should be revised accordingly to reflect a more accurate cost analysis.", "was_revised": false, "final_justification": "", "rebuttal": "**On weakness: Paper covers a wide range of topics**\n\nWe thank you for your feedback and would like to ask you for suggestions on how we may rewrite certain sections to make the paper feel more organised. \nHowever, we would expressly disagree that the wide range of topics covered is a weakness. This paper is not attempting to set the state-of-the-art in a particular task, but rather is trying to demonstrate a phenomenon that occurs in LMs across a wide range of inputs. That is, we are trying to firmly establish that across a wide range of inputs and behaviours, the hidden states of LMs contain valuable information on how the model will behave. We additionally hope to show that by leveraging this information, we can craft efficient and precise early warning and exit systems for these behaviours. The fact that the same method of conformal probes is applicable to such a diverse range of topics without any modification is one of its greatest strengths, and shows the generality of our findings. We do not agree that the work should have instead focused on a single set of tasks.\n\n**On issue: cost of inference measurement**\n\nWe thank you for your attention to this detail. We would note our initial motivation for measure % of tokens generated as opposed to wallclock time: we wanted a measure of efficiency that is agnostic to specific hardware implementations, and that captures the reason why the efficiency of the probe-based generation is superior. However, we understand your concerns that the empirical reality may not match up with this more conceptual metric, hence we measure the wallclock time of normal inference v.s. Inference with conformal probes guided early exiting on Llama3-8B using alpha=0.8, and present the results in the table below:\n\n| Dataset      | Total Inference Time (normal) | Total Inference Time (conformal probes + early exit) | Inference time reduction (%) | % Early Exit | Token generated reduction (%) |\n|--------------|-------------------------------|------------------------------------------------------|------------------------------|--------------|-------------------------------|\n| Selfaware    | 2600                          | 169                                                  | 93.5                         | 100          | 96.8                          |\n| KnownUnknown | 4131                          | 380.9                                                | 90.77                        | 99.84        | 92.4                          |\n| MMLU         | 1,12,336                      | 93,598.5                                             | 16.67                        | 17.24        | 15.91                         |\n\n\nAs we can see, the inference cost reductions are significant even when measured by wallclock time. Importantly, the inference time reduction roughly matches the % reduction in tokens generated, which is the original metric we report. This confirms that empirical results match our more hardware-agnostic measures of inference efficiency."}, {"review_id": "sDuLEXib9Q", "reviewer": "Reviewer_hGPd", "summary": "This paper proposes conformal probes, simple linear classifiers trained on a language model’s input-token hidden states and calibrated with conformal prediction. Once calibrated, the probes fire only when they can guarantee a user-specified error rate, giving an early-warning or early-exit signal before the model generates a single output token. \n- Safety & alignment: Detect prompts that will cause refusal-policy failures, cutting jailbreak success from 30 % to 2.7 % (≈ 91 % relative reduction) on the WildJailbreak set.\n- Format & instruction following: Predict whether an answer will violate output-format constraints with higher precision than a 340 M-parameter fine-tuned BERT baseline while using only ~0.0025 % as many trainable parameters.\n- Efficiency: Interrupt Chain-of-Thought (CoT) generation for text-classification tasks when the probe is confident, yielding a 65 % average reduction in forward passes across 27 datasets with ≤ 1.4 % worst-case accuracy loss and often a small accuracy gain", "questions": "1. Scope of detectable behaviours \n- You report that probes cannot predict outright answer correctness in MCQ tasks. Could you quantify this ceiling (e.g., accuracy vs. random) and elaborate on what differentiates behaviours the probe can vs. cannot capture? Clarifying this boundary would strengthen Section 6.\n\n2. Latency & resource overhead\n- While the method reduces generation compute, each query still requires a forward pass to produce hidden states for the probe. Please provide wall-clock latency and energy measurements relative to vanilla inference to validate net savings.\n\n3. Calibration stability across domains\n- How sensitive are the conformal thresholds to domain shift? The OOD CoT results are encouraging, but would thresholds learned on, say, conversational data hold for code or medical text? A small experiment or guidance on recalibration frequency would help practitioners.\n\n4. What are the practical advantages and potential drawbacks of turning conformal probes into a proprietary, in-house safety and efficiency layer?\n\n5. How would conformal probes perform differently on reasoning‐focused LMs versus instruction‐tuned LMs, and what model-specific considerations should be taken into account?", "limitations": "yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- Writing is concise.\n- A lightweight, model-agnostic safety-efficiency layer that can sit atop any inferene stack and composes with speculative decoding; addresses growing cost of long generations.\n\n## Weakness\n- Experiments are somewhat limited to the Llama3 families.\n- No latency measures for probe inference itself.\n- Linear probing are conceptually simple; early-exit ideas exist in speculative decoding (or adative inference) over vision/nlp. So, somewhat incremental novelty may hinge on the conformal guarantee.", "was_revised": true, "final_justification": "latency data, broader model tests, and the conformal guarantee clarification address my concerns, so I’m raising my score to 4. It’s clear that behavior-level early exits offer a valuable safety-efficiency layer.", "rebuttal": "**On question: Practical advantages / potential drawbacks**\n\nWe think that conformal probes can be a significant boon to organisations hoping to limit the computational cost of their LM inference. By identifying easy instances, catching malicious inputs early and exiting when they have precisely anticipated the LMs output, the probes can greatly reduce time and cost, while giving users exact control over the allowable error rate. \n\nConformal probes are not without their limitations. Specifically, the mathematical guarantee on the error rate only holds when the test data is exchangeable (similar to IID), and hence if deployed on data that is too different from the training distribution, it may fall below the user's stated performance threshold without notice. We speak more about this below, in an answer to your question on conformal calibration, where we suggest ways to use this method with confidence even in the OOD setting. \n\n**On question: latency and overhead of inference efficiency measurement**\n\nWe thank you for your attention to this detail. We would note our initial motivation for measure % of tokens generated as opposed to wallclock time: we wanted a measure of efficiency that is agnostic to specific hardware implementations, and that captures the reason why the efficiency of the probe-based generation is superior. However, we understand your concerns that the empirical reality may not match up with this more conceptual metric, hence we measure the wallclock time of normal inference v.s. Inference with conformal probes guided early exiting on Llama3-8B using alpha=0.8, and present the results in the table below:\n\n| Dataset      | Total Inference Time (normal) | Total Inference Time (conformal probes + early exit) | Inference time reduction (%) | % Early Exit | Token generated reduction (%) |\n|--------------|-------------------------------|------------------------------------------------------|------------------------------|--------------|-------------------------------|\n| Selfaware    | 2600                          | 169                                                  | 93.5                         | 100          | 96.8                          |\n| KnownUnknown | 4131                          | 380.9                                                | 90.77                        | 99.84        | 92.4                          |\n| MMLU         | 1,12,336                      | 93,598.5                                             | 16.67                        | 17.24        | 15.91                         |\n\n\nAs we can see, the inference cost reductions are significant even when measured by wallclock time. Importantly, the inference time reduction roughly matches the % reduction in tokens generated, which is the original metric we report. This confirms that empirical results match our more hardware-agnostic measures of inference efficiency.\n\n**On question: Calibration stability across domains**\n\nThis is an important question, and we would like to answer it from two perspectives:\n1. Theoretically, no, the learned conformal thresholds do not apply under shifting domains. This is intuitive as the formal guarantee that the threshold will bound the error rate depends on the test dataset being exchangeable (similar to IID) with the calibration set. This does not hold in the OOD case, and hence, unless we add assumptions on the statistical nature of the distribution shift (which is hard to do for abstract concept shifts like code v.s. medical text) we cannot guarantee a bound on the error rate. \n2. Emperically, we see optimistic results. While there is significant variance across tasks and datasets, we observe that the calibration threshold still limits the error rate when used in the ood setting. To see this, refer to Figure, which uses alpha=0.9 and hence hopes to limit the error rate to 10%. We see that across 18 datasets, this threshold is held even in the OOD setting. However, from our experiments we are able to give the following recommendations for practitioners who hope to use conformal probes in an OOD setting:\na. As usual, the training set that is closest in approximation to the testing set will give the most secure guarantees for the threshold. This is seen in the case of the sentiment datasets: we use only news-based, generic sentiment datasets to train the probes (twitter-mteb and news-mtc). Then, we test our probes on both generic (IMDB) and financial (TwitterFinance) sentiment analysis datasets, a shifted domain. The results show that when we use news based hold out datasets, the calibration is more precise\n| Dataset         | Conformal Consistency with alpha=0.8 | Conformal Consistency with alpha=0.9 |\n|-----------------|--------------------------------------|--------------------------------------|\n| Twitter-Finance |  77.69                               | 81.21                                |\n| IMDB            | 83.72                                | 86.81                                |\n\n\nb. When in doubt, having more datasets from a particular task is helpful. We see this in the MCQA task, where calibration is consistently better when we add more OOD datasets into the training set (for a single, fixed test dataset). This suggests that include a diverse range of training data helps improve calibration\nc. Expect larger than specified errors, and if the deployment scenario is safety critical, then use a small hold out test set to validate the conformal probes performance before using it in practice. \n\nWe believe that the conformal probes show promising signs of generalizing to unseen distributions, and that a careful selection of training datasets can provide highly performant and well calibrated probes.\n\n**On question: scope of detectable behaviors**\n\nWe are quite interested in better understanding the kinds of behaviors that LM hidden states struggle to predict. One such behavior is whether or not the LM will incorrectly answer an MCQA question. To quantify this, we experimented with 8 MCQ datasets, and trained hidden state probes to predict whether or not the LM will output an incorrect answer to a question. On 5/8 of these datasets, the hidden probes (when used without any conformal calibration) perform worse than, or equivalent to random performance. On the other 3 datasets, the probes perform slightly better than random, however any attempts at calibration show that no reliable threshold can guarantee an error rate of less than 60%, which is quite poor. \n\nMore generally, we find that there are two types of tasks that LM hidden states cannot predict well:\n1. Lexical quantification tasks: We experimented with tasks such as, predict the exact number of words that will be in the output, predict the number of verbs that will appear in the output etc. While the probes are able to perform reasonably (considerably higher than random) on the binarized version of these tasks (i.e. will the output be long or short, will the output contain a verb or not), it performs poorly on the quantification version. This is consistent with prior works that show LMs struggling with counting based problems[1], and suggests that the problem transfers to hidden state representations.  \n2. Tasks which require external knowledge: Our hypothesis as to why the MCQ tasks fail, is that unlike the other tasks we explore of format following, jailbreaking, etc, external knowledge is required to verify an answer to the MCQ questions. We can be sure that this knowledge is either not present, or hard to access in the LM, as it incorrectly answers many of these MCQ questions, which suggests that the hidden probes will always struggle to detect when the LM is going to incorrectly answer. \n\n\n**On Weakness: Experiments limited to LLama3**\n\nWe have made sure to include ablations on other models in the Appendix, showing that the trends discussed in the main text are replicable on the Mistral-7B and DeepSeek-R1-Qwen-14B models. We follow this up with additional experiments below, we train hidden state probes on the following model families: Mistral (Mistral-7B), Qwen3-8B, DeepSeek-R1\n\nThe probe results (when used without any conformal calibration) shows that we achieve above random performance with all LMs, showing the central claim of our work: that the hidden states of LMs can predict the future behavior of the output. \n| Dataset      | Random | Llama3-8B | Mistral-7B | DeepSeek-R1 | Qwen3-8B |\n|--------------|--------|-----------|------------|-------------|----------|\n| MMLU         | 51.36  | 69.51     |  69.9      | -68.45      | -71.42   |\n| KnownUnknown | 74.95  |  81.92    | 85.08      | 75.96       | 81.12    |\n| SelfAware    | 78.92  |  83.85    |  88.74     | 79.92       | 82.62    |\n\nWe observe that when using conformal alpha = 0.90, the probes are able to maintain high estimation consistency regardless of the underlying LM used, showing that the method is able to properly calibrate probes for a variety of LMs. \n\n| Dataset      | Random | Llama3-8B | Mistral-7B | DeepSeek-R1 | Qwen3-8B |\n|--------------|--------|-----------|------------|-------------|----------|\n| MMLU         | 51.36  | 89.10     |  88.15     | 89.45       | 91.43    |\n| KnownUnknown | 74.95  | 93.33     | 88.64      | 92          | 89.55    |\n| SelfAware    | 78.92  | 91.21     | 89.06      | 83.5        | 89.94    |\n\nOur motivation for focusing on the Llama3 family in the main text is primarily to keep the focus on the methods and tasks, as opposed to the specific language models used. We would posit that we have shown sufficient evidence to believe that none of our results are tied to any particular feature of the Llama3 LMs, and would hence disagree that the paper's use of Llama3 is a significant weakness. \n\n\n[1]: Fu, Tairan, et al. \"Why Do Large Language Models (LLMs) Struggle to Count Letters?.\" arXiv preprint arXiv:2412.18626 (2024)."}, {"review_id": "08xiiNmyqO", "reviewer": "Reviewer_74n8", "summary": "This paper introduces a novel and practical method for predicting the future behavior of a language model by probing the internal representations of its input tokens, before any text is generated. By integrating simple linear probes with conformal prediction, the authors create a reliable early-warning system with provable error bounds. The method is validated across a wide range of applications, including preemptively detecting safety failures (e.g., jailbreaking) and accelerating Chain-of-Thought inference, demonstrating significant performance gains (e.g., 91% jailbreak reduction, 65% inference cost reduction) with minimal overhead and strong generalization.", "questions": "- The finding that \"bigger models lead to better probe performance\" is a fascinating one. Is it because larger models encode their future behavioral plans in a more \"linearly separable\" or \"redundant\" manner? Does this performance improvement trend continue indefinitely, or does it plateau?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strenghts\n- The core idea of preemptive behavior prediction is innovative and provides insights into the critical, real-world problems of high inference costs and the need for proactive safety guardrails.\n- The claims are substantiated by extensive experiments across a diverse set of 27 datasets with OOD generalization tests.\n\n### Weaknesses\n- The paper primarily probes the hidden state of the final input token. The motivation and how does the attention mechanism effectively compress the \"intent\" or \"future behavioral plan\" of the entire prompt into this single token's state are unclear.\n- Apart from the out-of-distribution (OOD) generalization tested across different datasets of the same task type, it is necessary to discuss this generalization against more fundamental distribution shifts (e.g., code vs. text classification).\n- The authors chose simple linear probes, which ensures interpretability. However, this might limit their ability to capture more complex, non-linear behavioral patterns. Could a small, non-linear probe (e.g., a shallow MLP) unlock predictions for more nuanced behaviors (e.g., sarcasm, humor) without excessively increasing computational overhead?\n- Lack analysis of the overhead introduced by collecting hidden states for probe training.", "was_revised": true, "final_justification": "The author's rebuttal has mainly addressed my technical concerns, but the resposne to my question of reason for better probe performance in bigger models is missing. So I will maintain my score.", "rebuttal": "**On weakness: Probing final input token with unclear motivation**\n\n* Our decision to use the final token is motivated by a large body of work[1, 2, 3], which shows that the final token encodes useful information that is relevant for the prediction of a variety of properties of the input and output. Hence, we consider our choice of using the final input token as well motivated. \n* We also provide an analysis on the performance of using tokens at different positions, and show that the final token does contain more useful representations for behavior prediction, further solidifying this growing observation in the field\n* We understand that there are potentially methods of mechanistic interpretability that could be deployed to understand why the final token encodes the most information, however that is beyond the scope of this paper. Given the page limit, we would ask the reviewer to consider that the other sections we have added are of a more direct relevance to the paper, and we leave a deep understanding of the mechanism of attention, compressing the intent to future work. \n\n\n**On weakness: Lack of analysis of the overhead introduced by collecting hidden states for probe training**\n\n1. We would first point out that any method which hopes to learn the patterns of the LMs behaviors require some source of training data, and the cost of generating that data is roughly equal. For example, in order to generate the probe training data, we must run our LM on a training set, collect the hidden states and collect the output. In order to train our BERT baseline, we must collect the input text and the output, a process which naturally generates the hidden states as a result of the forward pass. Hence, the only extra cost of the probe (when compared to the BERT baseline), is that the hidden states must be stored in memory. This is not a significant cost, and is more than offset by the cost of fine-tuning BERT. Additionally, considering that once the probe has been trained, the hidden states can then be deleted, we do not consider this to be a significant overhead.  \n2. Furthermore, we show (Figure 6) that within 500 training datapoints, probes often approach the consistency and coverage they will attain when trained on the entire dataset. This suggests that the amount of training data required for good probe performance is small, minimizing the cost of training. \n\n\n**On weakness: The authors chose simple linear probes...**\n\nThis is an interesting question. We experiment with a variety of probe models, comparing Differences in Means, Linear Models, shallow MLPs, and RandomForests. \n\n| Dataset      | Random | Difference in Means | Linear | RandomForest | MLP    |\n|--------------|--------|---------------------|--------|--------------|--------|\n| MMLU         | 51.36  |  35.39              | 69.51  |  61.81       |  71.21 |\n| KnownUnknown | 74.95  |  26.22              |  81.92 |  82.43       |  83.81 |\n| SelfAware    | 78.92  | 19.08               |  83.85 |  83.54       |  87.08 |\n\nThe Linear and RandomForest approaches have comparable performance, while a shallow MLP proves consistently more capable. We believe that on KnownUnknown and SelfAware, the difference is not drastic enough to enable nuanced behavior detection, however, on MMLU, the improvement is significant. As you mention, even more sophisticated probes have the potential to identify more nuanced behavior, but such a decision would increase computational overhead significantly. In our work, we focus on linear probes primarily for their simplicity, however we do agree that there are several options that strike different points on the compute-performance curve, and there are multiple options in the low-compute regime that can give good performance.\n\n**On weakness: It is necessary to discuss generalization against more fundamental distribution shifts (e.g., code vs. text classification).**\n\nWe would first remember the overall goal of the probe: to preemptively predict a specific behaviour of the LM. When the distribution is shifted so fundamentally that the behaviours you are trying to predict is not well defined, it does not make sense to even try to apply a probe trained on one distribution on the other. For example, if you have trained a probe to preemptively predict which text classification option a LM will choose, there is no sense in applying it to a LM that is trying to synthesise a Python program (as the behaviour of interest, selecting a specific option, does not occur in this distribution). We do not see this as a limitation of our method, as no method that seeks to predict a behaviour can generalise to distributions where that behaviour itself is not well defined. What we have shown is that when this behaviour is well defined (ensured by taking multiple datasets spanning different domains, but having some common task properties), then we see signs of OOD generalisation. \n\nWe understand the importance of showing that our probes can generalize to fundamentally unseen domains, and conduct a shifting domain experiment to verify this. We stick within the sentiment prediction task, and use only news-based, generic sentiment datasets to train the probes (twitter-mteb and news-mtc). Then, we test our probes on financial sentiment analysis datasets, a shifted domain. The results show that despite never explicitly training on such financial data, the probes are able to achieve significantly higher than random performance:\n| Dataset              | Random | Probe  |\n|----------------------|--------|--------|\n| Twitter-Finance      | 51.8   |  77.69 |\n| Financial Phrasebank | 54.52  | 96.32  |\n\n\n[1]: Li, Kenneth, et al. \"Inference-time intervention: Eliciting truthful answers from a language model.\" Advances in Neural Information Processing Systems 36 (2023): 41451-41530.\n\n[2]:  Azaria, Amos, and Tom Mitchell. \"The internal state of an LLM knows when it's lying.\" arXiv preprint arXiv:2304.13734 (2023).\n\n[3]: Von Rütte, Dimitri, et al. \"A language Model's guide through latent space.\" arXiv preprint arXiv:2402.14433 (2024)."}, {"review_id": "KN2PTflvPm", "reviewer": "Reviewer_fHv2", "summary": "This paper shows that large language models (LLMs) often encode signals in their internal states that can predict their future behavior before generating any output. By training simple linear probes on the hidden representations of input tokens and calibrating them with conformal prediction, the authors build early warning systems for issues like instruction-following failures, safety alignment violations (e.g., jailbreaking), and low-confidence responses. These probes can reduce harmful outputs and improve efficiency, such as cutting inference costs by 65% in Chain-of-Thought prompting with minimal accuracy loss. The approach is lightweight, data-efficient, and scales well with larger models. It generalizes to unseen tasks and helps reveal what LLMs implicitly \"know\" about their own future actions.", "questions": "1. In Figure 2, the performance gain of probing over BERT appears marginal. While conformal prediction improves probe accuracy, it’s unclear whether similar techniques could also enhance BERT performance.\n\n2. In Section 4.1, the authors state they filter the training set to only include cases where the LM answers or complies. This risks making responses too similar, effectively turning the task into prompt classification—an issue discussed in the Weakness section. A better setup may involve using similar prompts that lead to different behaviors, training probes to predict those outcomes.\n\n3. In Section 5, it would be useful to report the LLM’s accuracy without CoT. If it's comparable to CoT accuracy, the benefit of using probes to skip CoT may be limited.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n1. The paper is well-written and easy to read.\n2. The research topic is compelling—detecting when an LLM is \"lying\" or misbehaving by analyzing its internal states before token generation.\n3. The paper effectively operationalizes the concept of \"LLM predicting its own behavior\" into practical techniques for improving safety and inference efficiency.\n\n## Weakness:\n1. My main concern is that high probing accuracy doesn't necessarily mean the LLM is predicting its own behavior. It could instead be classifying the input prompts, with the latent representations capturing prompt features rather than future behavior. While the BERT baseline partially addresses this, the performance gap between probing and BERT could also stem from architectural differences or parameter counts. **Although BERT is larger than the probing layer, the authors should take into account the parameter counts of the frozon LLM layers generating the latent representations. A more reasonable baseline would be using one causal LM of similar parameter counts to predict the behavior of another.** If such cross-prediction fails completely, I would be more convinced that the LLM is genuinely predicting its own behavior before token generation, and linear probing does capture the exact representations for prediction. I leave more questions about this in the Question section.\n\n2. It would be helpful for the authors to discuss related concurrent work. For example, [1] also uses linear probing to predict LLM response features (e.g., token count, reasoning steps) before generation begins.\n\n[1] Dong, Zhichen, et al. \"Emergent Response Planning in LLM.\" arXiv preprint arXiv:2502.06258 (2025).", "was_revised": true, "final_justification": "The rebuttal’s empirical results are persuasive and address my main concern. But I agree with Reviewer DPE7 that the paper could benefit from narrowing its scope slightly to better fit within the 9-page limit.", "rebuttal": "We think your primary concern is an important one, and we would like to point out some ways that we directly address the question: \"Are probes simply identifying features of the input, or are they truly predicting the future behavior of the LM?\"\n\n**Several tasks in our study are impossible to perform by simply identifying features of the input:** \n\nWe point to the tasks of format following (specifically the 3 bullet points task, which predicts whether a LM will output an answer in exactly 3 bullet points) and the jailbreak avoidance task (which predicts not whether an input text is malicious, but whether or not a language model will fail to abstain on a given malicious request). On the jailbreak avoidance datasets, LLama3-8B, Mistral-7B, and DeepSeek-R1 all have different abstention rates:(75%, 78% and % on the SelfAware dataset) and, in general, do not abstain/fail to abstain on the same input instances. We also see that probes trained on one LM **do not transfer to unseen LMs** (on the same dataset), i.e. a conformal probe trained and calibrated to provide 90% accuracy on the SelfAware dataset with LLama3-8B achieves only 60% accuracy on the Mistral-7B model. This shows that the probes are not identifying general, fixed features of the input text, but rather identifying model-specific patterns. \n\n**Pretrained LMs are incapable of predicting each others behavior in this way (without tuning):**\n\nWe stick to the jailbreak avoidance task and perform the following experiment:\n\nLlama3-8B is used to generate hidden states, and we then annotate the instance for whether or not Llama3 failed to abstain from a malicious input. We then measure the performance of the following methods:\nProbe trained on the hidden states to predict whether or not LLama3 will fail to abstain\nFewShot Prompting of Mistral7B to predict whether or not LLama3 will fail to abstain i.e. without any retraining / finetuning\n\nThe performance is as shown below:\n| Dataset      | Random | Probe | Mistral (No FT) |\n|--------------|--------|-------|-----------------|\n| KnownUnknown | 80.31  | 89.46 | 75.81           |\n| SelfAware    | 84.4   | 97.66 | 78.91           |\n\nAs we can see, when we do not fine-tune Mistral for the task, it performs worse than random, and significantly worse than the probes. This suggests that the property we are trying to predict is not one that can be inferred from just the text alone and requires some model-specific information to reliably ascertain. \n\n**Directly addressing the suggested experiment:**\n\nWe finally address the experiment suggested by the reviewer: \"A more reasonable baseline would be using one causal LM of similar parameter counts to predict the behavior of another. If such cross-prediction fails completely, I would be more convinced that the LLM is genuinely predicting its own behavior before token generation, and linear probing does capture the exact representations for prediction.\"\n\nWe have shown above that such cross-prediction fails when we are not allowed to train the other causal LM (Mistral); however, we suspect the reviewer was more curious about the case where you train the other causal LM to predict this. We would maintain that in such an experiment, you **should not** expect the cross-prediction to fail. \n\nThis is because, when provided with sufficient training data in the form of (input text, output behaviour), the other causal language model will eventually learn the patterns present in the data, and hence learn the behaviour of the LLama model. This would not be evidence that the probes are simply identifying input features, but rather evidence that with sufficient training data and a sufficiently powerful text classification system (in this case, Mistral with a classification head), you will be able to model the patterns in the data and learn the Language Model's behavior. \n\nThe point of our paper is to show that you do not need to distil the Llama model in this way to get a reliable signal on its behavior, because the hidden states provide you with helpful information. \n\nHowever, we understand the core concern here, and so we perform a modified experiment to demonstrate our point. We compare the probes to Lora-finetuning Mistral in a **data scarce** scenario: i.e. we train probes and the Mistral model to identify whether or not LLama3 will fail to abstain using only 50 datapoints. The performance is as follows:\n\n| Dataset       | Random | Probe | Mistral |\n|---------------|--------|-------|---------|\n| KnownUnknown  | 80.31  | 84.6  | 81.27   |\n| Selfaware     | 84.4   | 89.0  | 80.53   |\n\nWhile the Mistral finetune is better than few-shot prompting, showing that some pattern is being established, it is worse than the probe. This data-efficient learning ability from the probe shows that the pattern is more readily readable from the hidden states, which shows the core claim of our paper: that hidden states of LMs contain information on how the LM will behave. \n\n\n**On weakness: discuss concurrent work**\n\nWe thank the author for bringing this related work to our attention. We have added writing for both this and the work that it cites into the related work section of our paper."}]

Author rebuttal:

Meta-review: # Summary
This paper introduces a conformal probe method that can predict several types of model behavior from the hidden states of the final input token, before generating any new tokens. The method trains a linear classifier to predict behavior labels from the hidden states, and then calibrates the classifier's prediction using conformal prediction. The calibrated classifier prediction generates an early-warning or early-exit signal to predict failures in safety alignment, instruction following, or CoT reasoning in advance. With high prediction accuracy for certain behaviors, the proposed method can significantly reduce inference costs. The experiments demonstrate the method's effectiveness on 27 datasets. 

# Strengths
- The proposed method is simple, effective, and easy to follow.
- The idea of preemptive behavior prediction is innovative.
- The paper showcases several practically important applications of the proposed technique, where it can save the cost of generating many output tokens. 
- The experiments provided in the original draft and rebuttal are comprehensive and provide insights into when the conformal probe is effective. 

# Weaknesses
- Despite the novelty of preemptive behavior prediction, the application of probing and conformal prediction is shallow and incremental. 
- Some reviewers pointed out that the paper lacks focus on the downstream tasks. 
- The fairness of the initial efficiency comparison is questionable as it did not include the cost/latency of probe training/inference and report wall-clock time. It also does not cover the cost of collecting the training set for the probe. 
- It is not fully clear whether "the model can indeed predict its own behavior", or another irrelevant but similar-sized model can be trained to achieve the same capability. 
- Generalization of the conformal probe under domain or distribution shifts needs to be studied.

# Reasons to Accept
- Preemptive prediction of model behaviors from its hidden states is an important and novel problem to study.
- The proposed method is simple to use and straightforward to understand. 
- There are various potential applications of the proposed method. 
- The rebuttal and discussion addressed the main concerns from the reviewers.

# Summary of Discussion
- In the rebuttal, the authors provided additional experiments on more baselines, reported a more comprehensive analysis of the efficiency, and the results on models from different families. 
- In the discussion, all the reviewers responded that their main concerns have been addressed and confirmed their support to the submission.
Decision: Accept (poster)