Representation Consistency for Accurate and Coherent LLM Answer Aggregation
Authors: Junqi Jiang
Tom Bewley
Salim I. Amoukou
Francesco Leofante
Antonio Rago
Saumitra Mishra
Francesca Toni
Venue: NeurIPS 2025 poster ()
Keywords: Test Time Scaling
Large Language Model
Internal Activations
Primary area: deep_learning

Abstract: Test-time scaling improves large language models' (LLMs) performance by allocating more compute budget during inference. To achieve this, existing methods often require intricate modifications to prompting and sampling strategies. In this work, we introduce representation consistency (RC), a test-time scaling method for aggregating answers drawn from multiple candidate responses of an LLM regardless of how they were generated, including variations in prompt phrasing and sampling strategy. RC enhances answer aggregation by not only considering the number of occurrences of each answer in the candidate response set, but also the consistency of the model's internal activations while generating the set of responses leading to each answer. These activations can be either dense (raw model activations) or sparse (encoded via pretrained sparse autoencoders). Our rationale is that if the model's representations of multiple responses converging on the same answer are highly variable, this answer is more likely to be the result of incoherent reasoning and should be down-weighted during aggregation. Importantly, our method only uses cached activations and lightweight similarity computations and requires no additional model queries. Through experiments with four open-source LLMs and four reasoning datasets, we validate the effectiveness of RC for improving task performance during inference, with consistent accuracy improvements (up to 4\%) over strong test-time scaling baselines. We also show that consistency in the sparse activation signals aligns well with the common notion of coherent reasoning.

Representation Consistency for Accurate and Coherent LLM Answer A Aggregation
Junqi Jiang'
Tom Bewley? Salim I. 2moukou²
Francesco Leofante'
Antonio Rago² Saumitra Mishra² Francesca Toni'
Imperial College London 2 J.P. Morgan Al Research King's College London
jjunqi-jiane f. leofantte toni)\@mperial..... acllaccuu.
(firstname surname ) pmorgan. com, antonio rago@kcl. ac.uk
Abstract
Test-time scaling improves large language models' (LLMs) performance by allocat- ing more compute budget during inference. To achieve this, existing methods often require intricate modifications to prompting and sampling strategies. In this work, introduce representation consistency (RC), test-time scaling method for ag- gregating answers drawn from multiple candidate responses an L.MM regardless of how they were generated, including variations in prompt phrasing and sampling strategy. RC enhances answer aggregation by not only considering the number occurrences each answer the candidate response set, but also the consistency of the model's internal activations while generating the set of responses leading each answer. These activations can be either dense (raw model activations) sparse (encoded via pretrained sparse autoencoders). Our rationale is that if the model's representations of multiple responses converging on the same answer are highly variable, this answer is more likely to be the result of incoherent reasoning and should be nown-weights uses cached activations and lightweight similarity computations and requires no additional model queries. Through experiments with four open-source L.MSs and four reasoning datasets, we validate the effectiveness of RC for improving task performance during inference, with consistent accuracy improvements (up to 4%) over strong test-time scaling baselines. We also show that consistency in the sparse activation signals aligns well with the common notion of coherent reasoning.
during aggregation. Importantly. our method only
know that Auttrlis has..
What is the capinal of Austruia??
Okay, the options
the answer Sydney.
People commonly think..
Low consiacency rnjected
The largesi ln that counsry..
Camberrs.
What's the Australan capinal?
The nation of Australia..
Ausralia famors for...
High consisteccy chusen
An illustrative example of representation consistency. When aggregating an answer from multiple LLM responses (in the blue boxes the middle left) sampled from semantically equivalent rephrasings of the same question (in the pink box on the left), we take into account the consistency of model internal activations within each response group (points in the blue circle). In this case, the answer "Canberra" chosen (on the right) because the activations of its corresponding responses are more similar (the area covered by the orange points is smaller than that of the violet points).
Figure I:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Deep learning models have achieved remarkable capabilities across tasks and modalities, with large language models (LLMs) 67 9 [4 being perhaps the single most impactful model class [77] As the performance gains from L.M pre-training [36] begin to slow down on many benchmarks, recent research has diverted focus (and compute) to boosting model performance at the time of inference (76]. This line of work is often referred to as test-time scaling, and has proven to be effective, especially as larger compute budgets become feasible [
Numerous test-time approaches have been proposed to improve the outputs of LLMs without needing to update the models themselves [7] 68 37]. These approaches can be broadly divided into two categories [76]: prompt- based and decoding based. The former usually requires more outputs from relevant models. This includes asking the same LLM to generate more complex outputs elicit more reasoning [71 4 obtaining more responses (either through sampling [8 or prompt phrasing
60 ), then aggregating answers, and incorporating auxilliary L.MSs to collectively decide on outputs 19 37. This can be done in a single-tur tur or multi-turn conversation format. A simple yet effective approach is self-consistency (SC) 688, which takes the modal answer from multiple diversely sampled chain-of-thought (CoT) 7 responses. On the other hand, decoding-basse based methods intervene on the regular decoding process to obtain higher-quality responses 52).
However, methods like SC ignore an important source of information: the internal model activations. These activations are known contain rich encodings of LLMs' working mechanisms [16 4 the study of which may help us better understand how they function, such as identifying neurons related to factual knowledge storage 77 13 and signals useful for uncertainty quantification [.]. While recent works investigate feeding the activations back into the response generation loop for reasoning improvements in single L.M responses (24 59 38 56), they require model retraining. The usefulness of model activations cached in multiple LLM responses for test-time scaling has not been studied.
We fill in this gap by proposing representation consistency (RC), a method enhance LLM answer aggregation from multiple responses using internal activations. We focus on the setting where, given a question, multiple prompt rephrasings can be generated for that question, and multiple LLM responses can be sampled for each rephrasing (Section 2 This is inspired by recent works showing that answers obtained in such scenarios are more robust and better reflect LLM uncertainty [57] [5]. Given set of responses generated in this way, RC groups the responses by the final answer they give to the question, and uses an evaluation function to score each response group. This function combines the consistency (measured by a similarity metric) among the activations of each response in the group, and the cardinality of the group relative to the total number of responses generated (Section 3 We propose two variations of RC using both dense raw model activations and sparsified signals via sparse auto-encoders (SAEs) 30). specialised neural networks that can extract disentangled concepts from activations. Our intuition is that if multiple responses leading to the same answer have wildly differing activations, the model's s reasons for selecting that answer could be inconsistent, so this answer is less likely to be correct. Conversely, greater similarity in the activations of multiple responses (which may be diverse in their surface-level content) implies a more robust underlying reasoning process, which may be indicative of correctness. Importantly, RC does not require specialised prompting or modifications to the decoding procedure. Instead, it only uses the cached activations from generation and lightweight similarity computations. Figure T depicts an illustrative example.
We evaluate RC on four popular open-source LLMs of varying sizes (2B to 27B) and on diverse reasoning datasets (Section D. We show that by using model internals, there are consistent accuracy improvements (up to 4% in some settings) over SC, and over another strong baseline that we adapt using the same intuitions as RC but with external embedding models [18] [2]. We also provide some experimental insights showing the consistency in the sparsified activation space aligns well with the common notion of coherent reasoning among group of responses.
2 Problem Definition
We focus on single-turn question-answering (QA) tasks for which ground truth answers exist, and consider a setting involving multiple prompt rephrasings and sampled LLM responses.
2

---

Formally, let S = Ui=1 V be the set of all possible token sequences constructed from a vocabulary V. An LLM π + A(S) is a function that takes in prompt sequence E S and outputs distribution over response sequences É S, from which individual responses can be sampled.
(2, A. p. phrase, parse, f). Q is set of questions and A is set of answers. P A x A) is a distribution from which questions q and ground truth answers a" are sampled. phrase: Q + (SS) is a stochastic function that maps a question q to distribution over prompt sequences æ (concretely: alternative rephrasings of the question), and parse: S + A is a deterministic function (e.g. regular expression matching) that maps a response sequence y to an answer a { A x A + 22 is non-negative loss function between the answer a and the ground truth a" (lower better). is readily linked performance metrics like accuracy or ROUGE score.
A QA task is a tuple
We now describe two baseline methods for this problem setting.
Non-selective Expectation (NE) refers to the expected loss of an LLM ι over each response from each question rephrase, which can be quantified as follows:
LN T )). Ex--phrase(g) Byyrs(s) E(parse(y), a²).
(1)
Self-consistency (SC) 68 is strong baseline for this problem setting. For given question q. it takes the most common (modal) answer from a finite sample of prompts and responses Dj = {(0) ~ phrase(4)) y: ~ <<>;)));=--
modal (T, = arg max ((x,, vi) E Dr parse(y;) = =- aaA
(2)
This method leverages the intuition that correct answers are likely to emerge more often among a sufficiently large and diverse set of reasoning paths. The expected loss of this method is:
(sc(T, T) = E (agaa>>> {(modal q),a").
(3)
We note that when there are multiple modal answers, there is no clear tie- bie-breaking mechanism apart from randomly choosing an answer.
Our goal in this work is to develop an alternative method for selecting between responses from phrase and reduce the expected task loss beyond that achieved by NE and SC.
3 Representation Consistency
Next, we introduce representation consistency (RC) for answer aggregation, leveraging an additional source of information: the internal activations of the LLM т generates each response. Since these activations encode rich information about the LLM's reasoning processes, our intuition is that if multiple responses leading to the same answer have distinct activations, then the model's 's reasons for selecting that answer may be inconsistent, suggesting the answer is more likely to be incorrect. We show this consideration can be combined with each answer's frequency within the response set to create a criterion for aggregation.
Let fx S x S + Z,r be function that maps a pair of prompt and response sequences to the internal activation space language model x, such that (= y/ denotes r's activation when generating response y to prompt For each question q, activations for Dj grouped by answer can be cached:
Zy.a (f vi) (I;, v1) E Dg, parse(y:) = a).
(4)
Note that for an autoregressive transformer model, it is common to take this to be the residual stream activation at a particular layer (usually around the middle), either at one target token position or across all token positions [ Given all model activations for one generation process (over prompt-response pair) z E Zg, we denote its 1-d slice at layer and token position n as 2// E dR where dy is the LLM's s hidden dimension. Then, we introduce evaluation function V U.ee Z + R to score each answer considering both their supporting responses' internal activations and their frequency among all responses. Specifically, the evaluation function is a sum of two components weighted by hyperparameter E [0, 1]:
Vg,a X = consisteneyy,, + (1 - A) frequencyn,
(5)
'This parsing notation is very general. For instance, it can handle cases where the response contains chain-of- thought reasoning. in which case only the final answer is parsed. It also handles cases where the response fails refuses answer the question, provided the answer set A contains corresponding 'null" answer.
3

---

Consistency measures the representational similarity between the activations (given some layer and token position n) of each response within response group:
consistencyy,aa /2q,a(122
Σ
sim(zi,n: 22,n):
(6)
1)
21 2167 (eEBgaaass)
where sim Rdr Rd [0, 1] is similarity metric. This notation is generie, and use cosine similarity in this paper.
Frequency measures the proportion of responses supporting each answer in Dg:
frequency, [[ Ui) DD parse(us) a)}
[Dg
Then, RC selects an aggregated answer as:
RC(T, = arg max V (2g,a), aEAA
(8)
and the expected loss is:
(n, )~p (RC( (), a*)
(9)
In RC, consistency in activations and the frequency of each answer are considered jointly. This handles the fact that groups of responses per answer may vary in size, including the possibility of some having only one or zero answers. When two answer groups are similar in size, the evaluation function V mostly decides on the final answer via their consistency. Conversely, it tends to take the modal answer if this answer's 's group is much larger than the rest, unless its consistency is very low. The trade-off is controlled by We note that when X = 0, the evaluation function reduces to taking the modal answer, so RC becomes identical to SC. When X = 1, the selection is based purely on activation similarity. This can be undesirable because any answer group with only one answer will always be chosen. As shown in experiments (Section 4). the À hyperparameter needs to be tuned for optimal performance and is model- and dataset-specific. Additionally, the model layer I from which the activations are taken is also tunable, although a layer near the middle of the model architecture often gives the best results.
The sparse variation Raw model activations are known to be densely polysemantic (1. and human- understandable information can be encoded across model layers 4.. This means that calculating similarity on them might include redundant, noisy information. To counter this potential issue, we propose extra step for RC with Spaare-AAtoEneenn
(SAEs). They neural network models operating on LLM activations to encode them into sparse and well-disentangled signals in SAES' latent space (30] Each layer of any LLM
be associated with an SAE trained on its activations.
We denote the encoder of the SAE for the /-th layer as S' Rd. - IRd and AF is the number of latent dimensions of the SAF. Then, the only change from RC is the way consistency is calculated, with sim function handling different input sizes:
comsistency-spparse..r
1
Σ ZIE2a.a Z2E Mar)
Σ
Sene
IZ
Zagal 1)
(10)
Incorporating SAEs only introduces slight additional computational costs to the standard RC. Apart from the memory space needed for caching the LLM activations, when / and are determined, the additional cost for input text is one forward pass in SAE. Specifically, the input to SAEs, an LLM's 's residual activation at one token position, is of shape (1, d,), and the number of parameters in each SAE is usually small (e.g., pretrained SAE for the Gemma-2-99 model [65] has O.1B parameters (4) ). These indicate that the additional costs by SAEs are only a fraction of an LLM's
forward pass a prompt.
We respectively refer to the RC instantiations with raw model activations and with sparsified activa- tions as RC-D (dense) and RC-S (sparse).

---

4
Experiments
In this section, we quantitatively evaluate the effectiveness of RC for improving LLMs' task perfor- mance at test time. We first introduce our experiment setup.
Models We experiment with 4 open-source LLMs of varying sizes, Llama- 1-88--BB--- 1ng-Instruct 21] Gemma-2 2--aa----B------ Gemma 1omma-2-993 2---------- IT, Gemma-2 2--BB IT [65] These models are chosen because
our method applies to open- spenresurce LLMs whose internal activations are obtainable. Also, researchers have pretrained open-source SAEs for these models, namely GemmaScope [4]] and LlamaScope 25 which we use to investigate the differences between RC with raw activations and with sparsified activation signals. See Appendix A. for details on how they are used.
Datasets We test all methods using four reasoning datasets spanning diverse topics, Common- senseQA (CSQA) [2] for common sense reasoning, MMLU (26) for exam-style questions, MedM- CQA (55 for medical domain-specific knowledge, and Hellaswag (HSwag) 75 for sentence com- pletion tasks. We experiment with multiple-choice QA datasets because of the need to efficiently group generations by their final answer. We use the test or eval sets of these datasets 1200 samples for CSQA, and 3000 samples for other datasets. For Gemma 2-2-2-2222--- -T, we experiment with 1000 samples per dataset due to the heavy computation load.
Obtaining multiple LLM responses Each response is generated with CoT prompting, i.e., first ask for some step-by-ste analysis, then output a final choice (71]. All prompts are zero-shot. We use the following configurations (number of responses number of prompt phrasings x number of samples from each prompt): 12 responses with (12 x 1), (6 x 2), (4 x 3), (3 x 4), (2 x 6), (1 x 12), and 6 responses with (6 x 1), (3 2), (2 x 3), (1 6). We chose 12 and 6 because they cover large range of combinations for prompt-sample numbers. Every rephrased prompt is semantically equivalent, The ways they present the question and the candidate choices in the dataset samples are identical, and they only differ in the CoT instructions. For example, "Provide short explanations of your thinking steps", and "Briefly justify your reasoning process". See Appendix B for more details on prompts. Model generations from each prompt are sampled with 0.7 temperature for balanced randomness. Regular expression matching is used to extract the chosen answer from each generation for multiple-choic questions.
Baselines In addition to NE and SC (Section literature. It can be regarded as a variation of RC, which we refer to as RC-E (externa)). RC-E is in the same spirit as RC, but instead of using the LLM''s internal activations calculate consistency (Equation () for the evaluation function (Equation 5). RC-E uses embeddings (of the LLMM-generttd responses) from external encoder language models for this purpose. This way, provide an ablation for examining the effectiveness of incorporating internal activations against using embeddings from external sources. We introduce RC-E below.
incorporate another baseline adapted from the
Let fuis S + 3 be a natural language inference (NLI) model. Given two sentences y/, 1/2 E S, (1i 322) [Pentail, Pneutral, Peoxtradset] predicts the probabilities that the semantic meaning of y1 entails, is neutral to, or contradicts that of y/2 where Pentasl + Pneutral + Peontradict = We denote the first output element, entailment probability, foni (y1,32).
RC-E proceeds as follows. Given finite sample of prompts and responses for each question y. Dg (xx ~ phrase(q), Da ~ [.) first calculate average entailment probabilities among each group of generations Va,a {v: (Tz, yi) E Dg. parse (y:) = a}:
EP 9.a
Σ Σ
pent fili
(11, V2)
(11)
D (1) g.aa -
Then, the answer is selected by using a similar evaluation function to RC with hyperparameter A:
RCEaternal (T, q) = arg max aEAA
(A-EPq,a + (11 (1- frequencyy))))
(12)
where frequency is the same as that in RC (Equation 7). This method is inspired by how NLI
(and embedding) models are used in uncertainty quantification 39 15] and retrieval-augmenter
generations 43 In our experiments, we use bge-m3- -0mmm3eroooott-2222 0 12 41 for NLI with long sequence lengths. For all baselines, when there is during answer aggregation, we select the last answer for reproducibility.
5

---

RC instantiations We use both the raw model activations and the SAE-encoded activations to instantiate RC, respectively RC-D (dense) and RC-S (sparse). We follow the common practice of taking activations from the residual stream in the mechanistic interpretability literature []] For Gemma-2-88 2-99-IIT and Gemma-2-27B-
models, we retrieve activations from the three layers on which GemmaScope SAEs are trained, located at 25%, 50%, 75% of model depth. For the other two models, their pretrained SAEs cover every layer, and we take activations from layers at 10%, 25%, 50%, 75%, and 90% of model depth. For each LLM generation, we cache the activation at the token location where the model is about to output the final answer choice, c.g., right after "the answer is: ", and before "A". In terms of the sim function in Equations 6 and 10 use cosine similarity to capture directions in the latent space. See Appendix A.2 for a detailed discussion on the implementation.
We then perform two sets of evaluations. In Section 4.1 we report the task performance (accuracy) of each method, and in Section 4.2 we investigate whether the consistency in the representation space of LLM aligns with the common notion of coherent reasoning. All experiments are executed on Linux machine with 3 NVIDIA A100 GPUs, each with 80GB memory.
4.1 Task Performance Results
Throughout this section, we use SC as the main baseline and report every other method's accuracy relative to SC. We use 50% data to find the optimal hyperparameters for each method (A for RC-F, À and model layer for RC), and report task performance results on the remaining 50%. We focus on the test samples where there exist multiple answers from the set of LLM responses (see Appendix C for more detail) to better distinguish the performance differences between SC, RC-E and RC, as their chosen answer would be the same for the remaining samples.
The task performances averaged for each model are summarised in Table [ Figure further reports detailed results for each model and dataset, averaged over prompt-sample configurations for 12 and 6 answers, respectively. Note that we include models, 4 datasets, and 10 different prompt-sample configurations, making in total 160 sets of experiments. See Appendix C.2 for full results.
Table 1: Accuracy results (%) summarised for each model. We report the absolute results for the main baseline, SC, and relative results to SC for the remaining methods
Llama3.1-
Gemma2-
Gemma2- 9B-IT
Gemma2- 27B-IT
8B-IT
2B-IT
NE
-5.60
-4.43
-4.37
-5.19
SC
52.9
44.7
48.6
52.3
RC-E
+1.06
+0.84
+1.07
+0.55
RC-D
+1.84
+0.89
+1.32
+0.76
RC-S
+1.73
+1.10
+1.40
+0,89
From Table it is clear that RC-D and RC-S show the highest average accuracy gains for each model. RC-E moderately improve over SC, which is already a strong baseline when compared with NE - the average CoT accuracy. Zooming in to detailed results for each number of responses configuration of each dataset (Figure D again, we observe that in most cases (30 out of 32 plots), the RC-D and RC-S yield the highest performance gains, often exceeding 1% and sometimes over 2%. RC-S more frequently outperforms RC-D. The best performance observed at CSQA dataset for the Llama model at 6 responses, RC-D and RC-S respectively have 3.85% and 4.00% accuracy improvements, while RC-E is also effective here. However, there are rare cases where all RC variations show no improvements over SC, and we have observed decreased performance in individual cases (Appendix Overall, the improvements are more obvious on smaller models and with 6 responses, and are less on the counterpart settings, possibly due to the stronger baseline performance.
Table provides some insights into the optimal hyperparameters of RC-D and RC-S. We found that the most common optimal model layer for RC is at the middle, which coincides with recent research
works stating that these layers often contain diverse high-level information [66, 50). values are model- and dataset-specific, especially when the number of test samples is limited. Overall, the
optimal relative importance of consistency and frequency in RC (Equation 5) close to 1:1. See Appendix D for detailed ablation study.

---

+0.54 +15
-0 -9.19 14:0346...77
+0.12
CS04
42.40
41.19
-0.44
-2.34
NE
SC
RE-E RC-D BC-5
MC
0C-E 90-0
NE
SC
NC-F
RC-D RC-5
NE
NC-E 00-3 RC4
MeedMCOA
29
+0.43
+1.4361044
45.15
17:148
03.33
6.8
57.87
11.08-2-9422.000
44.05
3.70
-8.36
-2.31
(a) Llama3. (lama3.1-B--1
responses
responses
responses
responses
+0.05+12003
+1.48-1.55-77
CSDA
69:38
40.24
44,01
4.31
2.62
-3.97
MedMCCOA
+0.20 +0.2610.50H0.50
10.27+01994036
10.4840.44+000
41.49-124-00
-3.26
3.10
SC
RCE ACD RCS
NE
ACIE ACDD ACS
NE
SC
RCE
RC
ACS
NE
SC
RCE
8CD RC
(b) Gemma-2-2B-IT
responses
6 responses
12 responses +0 24
responses
+2.21 330042.79
CS0A
+1. 70
+0.85
$1.35
01:63
54.42
.1.92
36,95
NF
SC.
RC-E RC-O IC-5
KF
RC-E RC-0 RCS
NF
SC
RC-E RC-D IC-5
NF
NC-E 8C-0 AC-5
MecMMCO
-0.11
-0.00 +0.07
06-2200.3000..44
+0.75 1.111111
42.61e2:4012.00
39.34
53.04
-7.27
55492
SC
RCE RC
ME
RCEE AC0
SC
RC5
RCE
(c) Gemma-2-99B-
CSQA
49
responses
+0.32
46.2
responges
+0.80
responses,
10.78-1.1-1.29
58.19
responses
4.57
.6.03
7.77
RCEE RC RC
ME
RCEE RC.0
SC RCE
RCS
RCE
MedMC.AA
+0.0010.33"0.77 3370.79
+0.0/+0.7011
7010.81
+1.31+1.33--
10.88
1.000 0.00
43.55
40.25
-3.38
.235
4.43
SC
RCEE RC-5
NE
RC-E RC-0
NE 50 RC-E RC-D
RC-5
NE
RCEE 8C0 RC-5
Figure 2: Accuracy results (%) summarised for each number of responses configuration, dataset, and model. We report the absolute results for the main baseline, SC to the left of each subfigure with a dashed line for easy comparison. We report the relative results to SC for the remaining methods, the performance difference are shown on top of each bar.
Table Optimal hyperparamettes of RC for each model. The first number is the depth percentage of the model layer where activations are taken, the second number is the averaged optimal À value
Llama3.1-8B-I
Gemma2-2B-IT Gemma2-9B.. Gemma2-271 2
RC-D
50%, 0.43
25%, 0.29
50%. 0.36
50%, 0.44
RC-S
25%. 0.73
50%. 0.45
50%. 0.46
50%, 0.59
We additionally perform an analysis validating the non-trivial usefulness of L.LMs' internal activations for aiding answer selection, compared against the surface-level textual outputs. For each model,
7

---

Table Percentages where the correct/incorrect answer is associated with higher consistency, as calculated by each method. The number after each model name is the number of interested cases matching the deseription.
Gemma-2-2B- IT (497)
Gemma-2-9B. IT (404)
Gemma-2-22 --amm--
Llama-3.1-8P IT (195)
Method/Model
IT (113)
Embedding (baseline)
50.9%/49. 11.
49.5%
47.8%
50.3%/49. 7%
Dense (ours)
53.3%
53.7%
57.5%
56.4% 6%
Sparse (ours)
53.9%
54.7%
58.4%
56.4%
collected the test cases where among the 12 responses, 6 support one answer and the other 6 support another answer, and one of these tWO answers the correct one. We count the number times the correct vs. incorrect answer is associated with a higher consistency (dense activation consistency (ours, Equation 6). sparsified activation consistency (ours, Equation IOp, and embedding consistency (the entailment probability baseline, Equation IT].. We aggregate the results for each model over all prompt-sample configurations and over the datasets. This is presented in Table 3 Note that the consistency measures prone the cases where the L.M is confidently wrong about some answer, so the absolute values of such measures could be tightly linked to the LLM's performance. We observe that the correct answer constantly demonstrates higher consistency calculated with internal activations, while this is not the case with textual embeddings. This could indicate that the activations help reveal information during the response generation process that not included in the textual outputs.
4.2 Answer Coherence Results
Table 4: Percentage of times (%) where higher internal representation consistency (RC-D, RC-S) entailment probability in RC-E corresponds to more coherent reasoning (labelled by deepseek-R among group of responses.
Llama3.1- 8B-IT
Gemma2- 2B-IT
Gemma2- 9B-IT
Gemma2- 27B-IT
Dataset
Method
RC-E
51.0
55.9
59.6
62.5
CSQA
RC-D
67.6
60.4
69.4
72.9
RC-S
94.6
91.1
90.4
93.8
RC-E
55.6
55.5
55.4
61.7
HSwag
RC-D
51.4
44.7
56.8
65.0
RC-S
91.6
85.2
94.4
95.9
RC-E
44.7
53.0
50.4
48.1
MedMCQA
RC-D
47.1
49.0
63.8
59.1
RC-S
98.8
88.0
92.1
90.9
RC-E
57.0
54.2
49.3
56.4
MMLU
RC-D
48.3
47.9
55.5
58.2
RC-S
92.8
81.6
82.9
90.0
Next, we investigate the alignment between the consistency of model representations and what we, as humans, would perceive as coherent reasoning. We focus on the test cases (in each prompt-sampling configuration with 12 candidate responses) where the model produces answers, and the number of responses supporting each answer is close, i.e., 6 vs. 6 and 5 vs. 7. To approximate our coherence notion, we employ au-uuee and discuss LLMs' suitability for this evaluation Appendix Specifically. we prompt the deepseek reasoner model (deepseek-R) RI) [22., asking which group of responses is more coherent their reasoning. Using these results, we then record the percentage of times when the group of responses with better consistency in their representations (Equation 6 use the 50% depth layer for every model) coincides with the coherence label. We perform the same evaluation with RC-E's entailment probability (Equation []] bascline.

---

The results are shown in Table 4 RC-S achieves the highest agreement rate with the labels, surpassing 90% in most cases. RC-E and RC-D have similar performance, near 50%, with RC-D more frequently having higher scores.
It is surprising to observe a substantial gap between RC-S and RC-D. This indicates that despite their similar task performances, they rely on different notions of consistency to achieve them. We can confirm that the model activations, after being processed by SAEs, align well with the common coherence notion. This is in line with the training objectives of SAEs - sparsify the dense signals in raw activations into sparse. potentially human- -ndernsdttttte concepts. We observe in our experiments that, among the many (> 10k) SAE latent dimensions, only about 100 are activated (having non-zero values) at the target token position during generation. Therefore, a larger cosine similarity here means similar concepts are present when determining the final answer. This intuition naturally matches our notion of coherence. Similarly, RC-E's entailment probability aims at obtaining the more coherent set, but is highly sensitive to the performance of the external NLI model on L.LMs' CoT-sttyle responses.
On the other hand, dense raw activations at single layers may earry more information than SAE activations. The accuracy improvements (Section 4.1 and the lower agreement rates (Table hint at useful but less interpretable representational consistency notions different from what we understand as coherence.
5 Related Work
Test-time scaling refers to the problem of improving the task performance output quality of LLMs after they have been trained [76]. It is distinet from post-training (9), which updates model parameters via supervised fine-tuning (2] or reinforcement learning (58). instead seeking make improvements without directly modifying the model. The prominent approaches are related to chain-of-thought reasoning, asking for some intermediate reasoning process [71]. This originally works on each response in a single-tun conversation. Further research extends this to ask for self-reflection and correction 47 on the same LLM, forming multi-turn generations. There are also works to incorporate multiple LLMs performing similar conversations to reach more robust outputs (20) with merged vocabularies across models 29 72 or in multiple turns (37 19]. Works like self-consisteney [68], instead, obtain multiple responses from a single model (possibly in multi-turn [64)) and then perform answer aggregation. Others bring a mix of the above approaches and intervene in the decoding process 773 5 52). Different to these approaches, we consider model internal activations for aggregating multiple LLM responses.
The role of multiple prompts It is known that outputs of language models are very sensitive to the prompts, even those semantically equivalent rewritings [57 46 14. However, this setting has proven to be useful [45]. Similarly to SC, ensembling outputs over multiple prompts can improve
task performance [35 60). When combined with diverse sampling, prompt rephrasing also help better quantify the uncertainty (28 [5] and improve calibration (34] in LLMs. Additionally, if viewing phrasing of the question as part of predictive model, prompt sensitivity also resembles model multiplicity in machine learning [48 77) which states the existence of multiple equally performing models that could give different predictions. Ensembling- based methods are often used address this issue 66 33 Our method readily applies to multiple prompts and samples.
Model internal activations are at the core mechanistic interpretability research (see 4 for recent overview), which aims to understand model behaviours by investigating patterns in these activations. By training probe classifiers on the activations, researchers have identified neuron locations that correspond to actual knowledge [49] [3], representations for uncertainty [II], and hallucination risks [TT 6T). Useful directions in the activation space have also been identified for steering LLMs towards desirable behaviours like instruction following [10] [8] 42). Apart from these observational approaches, patching methods intervene on an LLM's forward pass by replacing activations at certain locations with those obtained from other runs to perform tasks like identifying critical neurons [II and finding parts of the LLM for specified behaviours (23.. Finally, sparse autoencoders are intermediate tools trained on L.LM activations to map them into disentangled concepts, such that only a small portion of SAE latent dimensions are activated during each generation 30 25 In our case, use both raw activations and SAE-encoded sparse signals for test-time scaling.

---

6 Conclusions
In this work, we introduce representation consistency (RC), method using model internal activations to enhance answer aggregation from multiple LLM responses. To the best of our knowledge, we are the first to investigate the usefulness of activations for such test-time scaling scenarios. We propose two variations of RC that respectively operate on the dense raw model activations and their SAE-spariified counterparts. In our experiments, we also adapt existing works into a new ablation baseline that leverages the same intuitions behind RC but with external embedding models. We show that these new methods all consistently improve over the strong baseline, self-consistency, with RC-sparse delivering the highest accuracy gains. Additionally, we show that the representational consistency in the latent space of LLMs' corresponding SAEs aligns well with what we would deem as coherent reasoning in multiple responses.
Our proposed method comes with some limitations. First, RC requires access to model activations during generation. While developers of proprietary LLMs can make use of RC, other users could only apply RC to open-source models. Second, our intuitions behind RC may break in cases where an LLM is confidently and systematically wrong in its predictions, so that an answer resulting from multiple paths of incorrect reasoning may nonetheless have consistent activations. However, we suspect that any method that operates given LLM responses without further model training would struggle to handle such cases. Also, our encouraging accuracy improvement results suggest that the heuristic of representation consistency is effective for ruling out many incorrect answers, and improving answer aggregation on average.
Exciting future works are envisaged following the introduction of RC. While we have investigated its integration with LLM response sets obtained by sampling and prompt rephrasing, the method could also be combined with more complex decoding- based scaling methods, e.g. tree-of- -fe-oo-th [73. In this work, we examined RC on multiple-choice tasks. would be an interesting direction to extend it to other reasoning tasks, possibly with more complex and open-ended generation forms. While designed for handling multiple responses from a single L.M,, investigating RC's transferability across multiple LLMs would also be desirable [53]. Consistency from multiple responses has been a useful signal for model training 3 17 69]. and using activations to aid training has been studied 24 [5]. It would be interesting explore how the consistency of activations can help in improving reasoning or model controllability. There are recent works separately using model internals [ or prompt rephrasings [15] for uncertainty quantification and hallucination mitigation. Our method, combining both elements, could potentially be studied in this space too. Finally, our findings on the mismatch between raw model activation consistency and our understanding of coherence also highlight the need for further studies on the interpretability and transparency of L.MSs [27) possibly aided by SAFS.
10

---

Disclaimer
This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates ("JP Morgan") and is not product of the Research Morgan makes no representation and warranty whatsoever and
Department of JP Morgan. disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or recommendation, offer or solicitation for the purchase sale of any security, financial instrument, financial product or service, or to be used any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
@ 2025 JPMorgan Chase & Co. All rights reserved.
Acknowledgements
Jiang, Rago and Toni were partially funded by J.P. Morgan and by the Royal Academy of Engincering under the Research Chairs and Senior Research Fellowships scheme. Leofante was funded by Imperial College London through under the Imperial College Research Fellowship scheme. Rago and Toni were partially funded by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No. 101020934). Any views or opinions expressed herein solely those of the authors listed.
References
[1] Anthropic. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, Anthopicc, 2023.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2XXv:2204.05862. 2022.
[3] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 2022.
[4] Leonard Bereska and Stratis Gavves. Mechanistic interpretability for safety-a review. Trans- actions on Machine Learning Research, 2024.
[5] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. In The 38th AAAI Conference on Artificial Intelligence, AAAI, pages 1 17682-17690, 2024.
[6] Emily Black, Klas Leino, and Matt Fredrikson. Selective ensembles for consistent predictions. In The /2th International Conference on Learning Representations, ICLR, 2022.
[7] Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concerns, and solutions. In 2022 ACM Conference Fairness, Accountabilityy FAccT 2022,
pages 850-863, 2022.
[8] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, arXiv preprint arXiv:2407.21787, 2024.
[9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
11

---

Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, NeurIPS, 2020.
[10] Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. In Advances in Neural Information Processing Systems 38, NeurIPS, 2024.
[11] Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: Ilms' internal states retain the power of hallucination detection. In The 12th International Conference on Learning Representations, ICLR, 2024.
[12] Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionaliity, multt-granularrty text embeddings through self-knowledge distillation. arXiv preprint ar arXiv:2402.032 032211......,.... 2024.
[13] Lihu Chen, Adam Dejl, and Francesca Toni. Identifying query-relevant neurons in large language models for long-form texts. In The 39th AAAI Conference on Artificial Intelligence, AAAI, pages 23595-2360X 2025.
[14] Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. In Proceedings of the 41 International Conference on Machine Learning, ICML., volume 235, pages 6596-6620, 2024.
[15] Kyle Cox, Jiawei Xu, Yikun Han, Rong Xu, Tianhao Li, Chi-Yang Hsu, Tianlong Chen, Walter Gerych, and Ying Ding. Mapping from meaning: Addressing the miscalibration of prompt-sensitive language models. In AAAI, 2025.
[16] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, pages 8493-8502, 2022.
[17] Mingkai Deng. Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. RIprompt: Optimizing discrete text prompts with rein- forcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 3369-3391, 2022.
[18] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIIT, pages 4171-4186 2019.
[19] Yilun Du Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In 41st International Conference Machine Learning, ICML, 2024.
[20] Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. Epsilon sampling rocks: Investigat- ing sampling strategies for minimum bayes risk decoding for machine translation. In Findings of the Association for Computational Linguistics: EMNLP, pages 9198 9209, 2023.
[21] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arxiv:2407.21788,, 2024.
[22] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948. 2025.
[23] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Advances in Neural Information Processing Systems 36, NeurlPS, 2023.
12

---

[24] Shibo Hao, Sainbayar Sukhbaatar, Dilia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06766 2024.
[25] Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Jumxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. Llama scope: Extracting millions of features from Ilama-3. 1-8b with sparse autoencoders. arXiv preprint
2024.
[26] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference Learning Representations, ICLR, 2021.
[27] John Hewitt, Robert Geirhos, and Been Kim. We can't understand ai using our existing vocabulary. arXiv preprint arXiv:2502. 075566 2025.
[28] Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. De- composing uncertainty for large language models through input clarification ensembling. In Proceedings of the 41st International Conference on Machine Learning, ICML, volume 235, pages 19023-19042, 2024.
[29] Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. Ensemble learning for heterogeneous large language models with deep parallel collaboration. In Advances in Neural Information Processing Systems 38, NeurlPS, 2024.
[30] Robert Huben, Hoagy Cunningham, Logan Riggs, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The 12th International Conference Learning Representations, ICLR, 2024.
[31] Ziwei Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawijaya, Yejin Bang, Bryan Wilie, and Pascale Fung. LLM internal states reveal hallucination risk faced with a query. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 88-104, 2024.
[32] Junqi Jiang, Tom Bewley, Saumitra Mishra, Freddy Lécué, and Manuela Veloso. Interpret- ing language reward models via contrastive explanations. In The Thirteenth International Conference on Learning Representations, ICLR 2025, 2025.
[33] Junqi Jiang, Francesco Leofante, Antonio Rago, and Francesca Toni. Recourse under Model Multiplicity via Argumentative Ensembling. In Proceedings of the 23rd International Confer- ence on Autonomous Agents and Multiagent Systems, AAMAS, pages 954-963, 2024.
[34] Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. Calibrating language models via augmented prompt ensembles. ICML Workshop on Deployable Generative AI, 2023.
[35] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know. Trans. Assoc. Comput. Linguistics, 8:423-438, 2020.
[36] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0885 2020.
[37] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan. Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with More Persuasive LLMSs Leads to More Truthful Answers. In The 41st International Conference on Machine Learning, ICML, 2024.
[38] Deqian Kong, Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig, Zhangzhang Si, Chuan Li, Jianwen Xie, Sirui Xie, et al. Scalable language models with posterior inference of latent thought vectors. arXiv preprint arXiv:2502.01566 2025.
[39] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in naural language generation. The 12th International Conference on Learning Representations, ICLR, 2023.
13

---

[40] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James V. Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar. Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACI. 2025, pages 1755-1797, 2025.
[41] Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. Building efficient universal classifiers with natural language inference. arXiv preprint arXiv:2312.17543, 2023.
[42] Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre L. Dognin, Manish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional activation steering. In The 13th International Conference on Learning Representations, ICLR, 2025.
[43] Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kicla. in Neural Information Processing Systems 33, NeurIPS, 2020.
generation for knowledge-intensive NLP tasks. In Advances
[44] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. Proceedings of the 7th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 278-300, 2024.
[45] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM computing surveys, 55(9):1-35, 2023.
[46] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, pages 8086-8098, 2022.
[47] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems 36, NewrlPS, 2023.
[48] Charles T. Marx, Flávio P. Calmon, and Berk Ustun. Predictive multiplicity in classification. In ICML 2020, pages 6765-6774, 2020.
[49] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowl- edge in alphazero. Proceedings of the National Academy of Sciences, 1 222647)) 19, 2022.
[50] Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. Unlocking the future: Exploring lhek- planning mechanistic interpretability in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 7713-7724 2024.
[51] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[52] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501. 19309993..1993, 2025.
[53] Narmeen Fatimah Oozeer, Dhruv Nathawani, Nirmalendu Prakash, Michael Lan, Abir HAR- RASSE, and Amir Abdullah. Activation space interventions can be transferred between large language models. In second International Conference Machine Learning, ICML, 2025.
14

---

[54]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, NeurlPS, 2022.
[55] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale mubte- multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, CHIL, volume 174, pages 248-26682 2022.
[56] Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers. In The 13th International Conference on Learning Representations, ICLR, 2025.
[57] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design How learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024.
[58] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint urXiv:2402.03.300 2024.
[59] Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing thahn-tf- into continuous space via self-distillation. arXiv preprint 2rXiv:2502.21074, 2025.
[60] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, ICML, volume 202, pages 31210-31227, 2023.
[61] Zhongxiang Sun, Xiaoxue Zang, Kai Zheng. Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li. Redeep: Detecting hallucination in retrieval- augmented generation via mechanistic interpretability. In The 13th International Conference on Learning Representations, ICLR, 2025.
[62] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pages 4149-4158, 2019.
[63] Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga- Alonso, and Robert Kirk. Analysing the generalisation and reliability of steering vectors. In Advances in Neurul Information Processing Systems 38, NeurlPS, 2024.
[64] Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in 1mm.. arXiv preprint arXiv:2502.06233, 2025.
[65] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00/ 18, 2024.
[66] Adly Templeton, Tom Conerly, Jonathan Mareus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Bat- son, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems 30, NeurIPS, pages 5998-6008 2017.
15

---

[68] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning in language models. In The I Ith International Conference on Learning Representations, ICLR, 2023.
[69] Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized rewarding language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, 2025.
[70] Zhilin Wang. Yi Dong. Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In Advances in Neural Information Processing Systems 38: NeurIPS, 2024.
[71] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. Chain-of-thouw prompting elicits reasoning large language models. In Advances in Neural Information Processing Systems 35, NeurlPS, 2022.
[72] Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. Bridging the gap between different vocabularies for LLM ensemble. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL, 2024, pages 7140-7152, 2024.
[73] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36, NeurlPS, 2023.
[74] Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 3267-3280 2024.
[75] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL, pages 4791-4800, 2019.
[76] Qiyuan Zhang. Fuyuan Lyu, Zexu Sun, Lei Wang. Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models. arXiv preprint 2kiv:25503.22335. 2025.
[77] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18211 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are backed by the methodology and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The models, datasets, and procedures to run the methods are described in detail.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
The datasets and models used in our experiments are open- source and publicly available. We have put correct citations for them. We will release the code with detailed instructions.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips.c//pppp
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are described in the Experiment section and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our setting assumes that the LLM answers have been generated. The remaining procedures give deterministic results. We report detailed results for every configuration in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
They are described in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: These are discussed in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They properly cited. Licenses and terms of use respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the code with detailed instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

A Implementation Details
A.1 Models
When obtaining responses from the experimented LLMs, we apply their respective chat templates described in their huggingface model card. We query them with the transformer' library. To accelerate inference, responses from Gemma models are queried with a batch size of No batching is applied Llama model because it does not have dedicated padding token, and is by default right-padded which conflicts with the implementation step to cache model activations.
We use the Python library sae-lens for using the SAEs. Specific SAE models and the layers are summarised Table They can be found in the sae-lens link above. Note that SAEs for 2B, 27B Gemma models and the Llama model are originally trained on the activations their base version. has been shown that the SAEs also work on their instruction-tuned version 25
j/booo/auss github i L0o/SAELenn/ssss httpeee//eeaaaaeeeeeeeeeeeeaaeee
Table 5: SAE models used and the layer numbers at each LLM's respective model depth
SAE name
10 %
25%
50%
75% %
90%
llama_scope_lxe
cecoopellrr_xx
8
16
24
29
sennma--cop--------s---noooo
7
13
20
23
semma--cop----iitrss--mooio sommaa-cope-27b-pt-aea canonical
9
20
31
10
22
A.2 RC Implementation
Detailed implementations be found in the accompanying code. We off-the-shelf function- alities provided by sae-lens and perform activation eaching in two-step process for large-scale experiments. For each experiment on model, dataset, and for all prompt-sample configurations. we first generate all the answers needed using the transf ormers library. Then, we process the answer to identify the token location in the response where the model about to output the final answer choice. In a separate process, we then concatenate (the tokens of) the prompt and the model response up the answer location, and use sae-lens to generate only the next token. We cache the model activations at this step for RC.
B Prompts
12 prompt templates are used in our experiment. For each data point in dataset, we sample 12 answers from the first prompt, from the second, 4 from the third, from the fourth, from the fifth and sixth, and 1 from the rest. This way, we cover the need for responses for all prompt-sample configurations. For every dataset, the way of presenting the question is the same:
Question:
(QUESTION}
Candidate answers:
A: (ANSWER_A)
B: (ANSWER_DD B}
C: (ANSWER_CO
The prompts only slightly differ their instructions:
Prompt 1:
You helpful AI assistant, answer the following question:
Think step by step. Briefly justify your reasoning process, then put your final chosen answer in the form: [The ansver is: (X) at the end.
Prompt 2:
24

---

You are knowledgeable helper, look at the following question: MAU
Let's break this question down step by step. Write some short explanations for your reasoning, then put your answer the form: [The answer is: (x)] at the end of your response.
Prompt 3:
You are an expert in multiple choice questions, answer the following question concisely:
Think about the question step by step. Provide brief explanations for your thinking process. Put your answer in the form: [The answer is: (X)] to the end.
4:ompt
You are a helpful AI assistant, answer this question: COSTTIOONAND_AANDIDDON ANNSERATEEAASWERRS)
Think step by step about this question. Add brief justification for your
choice of answer. Output your answer the form: [The answer is: (X)
at the end of your response.
5:ompt
Answer the following question:
AASSSTOOOAANDCCNIDDATTAAELEAA (QUESTION_AND MANDIDATE_ANSWERR
Let's think step by step. Provide short explanations of your thinking steps. At the end of your response, put your choice of answer in the form: [The answer is: (X)].
Prompt 6:
Here's question need you help with: {QUESTION BIETTIONAANDD_CC MANDIDAAE__ANWEERR
Let's break down this question and think step by step. Briefly outline your reasoning process. Output your choice of answer with the form: [The answer is: (X)] to the end.
Prompt 7:
Look at the following question and answer it: [QUESTION_AND AND__AANDIDATE_AASSS Think step by step. List out your thinking. Keep it short. Put your answer in the form: [The answer (X)] at the end your response.
Prompt 8:
I have multiple choice question which you going to help with: (QUESTION_AND_C MAAADDAAADDDATEAAAWEEEEEE Let's think slowly and step by step. First briefly output your thinking process with short justifications, then finally output your answer in the form: [The answer is: (X)].
Prompt 9:
Please help me answer the following question: (QUESTION_AN) SANDD_AAIIDATEELCC. ANSWERS)
Look at the question step by step. Explain your thoughts very briefly and finally output the answer in the form: [The answer is: (x)]
Prompt 10:
Which candidate answer do you think is correct for this question: LAATTTOONAAD_CCANIDDATTEEEE Consider this question step by step with short explanations for your
25

---

thoughts, then put your answer in the form: [The answer is: (X) at the end of your response.
Prompt 11:
Here is question in the multiple choice form with four potential answers: CAD.LAANAAANIDTEE_AAAAAA )
Analyse the question and candidate answers with step-by-step thinking, then state the correct answer in the form: [The answer is: (X) at the end of your outputs.
Prompt 12:
Below is multiple choice question. Look at the question and the candidate answers, select the correct [QUESTION_AND. LAEIIOOOANND_AAE JOUSSTTOLLADDCAADIDDTT.....A LAND__AADDDATT_AANWEEEE
Think about step by step, present short explanations for your thoughts. At the end of your output, state your answer in the form: [The answer is: (x)].
C Result Details
C.1 Number of Points
For the task performance results in Section 4.1 we only report results for the test points where multiple answers exist among the responses. T the average number of points used for each experiment, and the percentage of such points among all test The results are averaged over the specific prompt-sample configurations at each number of responses.
Table 6: Average number of points (and percentage) where multiple answers exist among the responses. The number after dataset name indicates the total test points for that model.
Model
Dataset
12 responses
responses
CSQA (1200)
4 2.57)%
4 1.29) %
HSwag (3000)
1602(53.42 0.83)%
1307(43.58 + 1.67)
1-8B-I
MedMCQA (3000)
1822(60.73 + 0.63)%
1524(50.8 + 0.42)%
MMLU (3000)
1295(43.17 1.07)%
CSQA (1200)
730(60.89
1.86)%
582(48.50
1.91)%
HSwag (3000)
1994(66.50
1575(52.50+
Gemma2-2B-IT
McdMCQA (3000)
MMLU (3000)
159.
CSQA (1200)
576(48.02 + 2.95%%
664338.67
HSwag (3000)
1178 27 + 1.53)%
938(31.27 0.81)
Gemma2-9B-IT
MedMCQA (3000)
1813 1.63)
1516(50.544
MMLU (3000)
1028(34.29 + 1.93)%
819(27.31 1.67)
CSQA (1000)
405(40.50 + 2.08)%
309(30.90 1
HSwag (1000)
474(47.40 1.82)%
355(35.50 = 0.66)
Gemma2-27B-IT
McdMCQA (1000)
505(50.50 + 0.61)%
4270(42.7 1.46)
MMLU (1000)
429(42.90 + 0.81)%
345(34.50 + 0.76)%
We observe that is common obtain different answers from multiple responses of the same LLM. often more than 50% for each dataset. This is more obvious for smaller models as they might be less certain on their predictions. Also, it happens more frequently with more responses. Additionally, incorporating more prompt rephrases (e.g.. comparing prompts, 2 responses each with 2 prompts, responses each) will result more test points having different answers, contributing to the standard deviations
26

---

01.01
40.05
4L.0
4520
c1.03
K0-0
K 10-1 8:0
Rit 1:0
4: KC KSS
KC 10000
44.05
11.0
9334
11.0 K14
30,0
5C ICE K25
ICE 16D
125
(a) Results for 12 responses.
6x1
328
243
1x5
III
HEM
56.0
12.08
12
14:2
46.27
1112
(b) Results for responses.
Figure 3: All results for Llama3. 1-8----11--1-
H.S HELL
KERDRR
RICURS
SC CERONY
KEROR
RERDRS
KERDRS
12. RCORCS
KEROO 165
ICE R25
ICECCO
Paackcok
27.59
X1
10M 3157
121
1C2 RC0
ICC
50
1C4
1200
125
IC4 CC
125
ICE.CO
(a) Results for 12 responses.
6x1
3x2
2x3
106 HONG
No
M
12:00
10 10:0000
12.1
5079
(b) Results for responses.
Figure 4: All results for Gemma2-2B-IT
27

---

40.43
4126
ATI
1C0 ICS
124 123
15.33
RC- RE-
RD1 RC0 R55
ICE NO
FC4 RC0 RC3
1EE KCO KS5
ICE RE
4.39
3i
C441
1137
2031
1 X
5C 9C4 16.
125
2: RC4 1C
KCE 125
ICCRCO RCS
(a) Results for 12 responses.
3x2
243
1x6
-243
30.39
1208
31
0.30
1150
10331021
2315
20
141
(b) Results for responses.
Figure 5: All results for Gemma2-9B..
HLCO
68.0
$1.42
50.30
5922
1E KC KS
KEROORD
4E.0
61
40.7
10.00
91.23
1C2 RCD
NC
NC2
50
ICE
120
125
IC4 RC
125
IcRCCORS RC5
(a) Results for 12 responses.
6x1
3x2
243
1x6
HHIMOOK
10.4
12.00
12.00
45:00
.003
m
4353
(b) Results for responses.
Figure 6: All results for Gemma2-27B-IT
28

---

C.2 Task Performance Results Configuration
repoot the detailed results for each prompt-sample configuration in our experiments. The observations in Section 4. I also apply to these results, although here we can observe a larger range of accuracy changes for RC-E, RC-D, and RC-S. For example, largest accuracy improvements for RC-D and RC-S are 6.25% for CSQA dataset on Llama model with 6 prompts and sample per prompt. We can also see cases where the RC- methods worsen the accuracy from SC. This because the optimal hyperparameters can be overfitted on the tuning subset of data, specifically if the answer distributions (e.g., the number test points having 2 different answers, each with supporting responses, versus the number of test points having 2 different answers with and 10 supporting responses, respectively) are very different between the tuning subset and the test subset. This happens more frequently with the 27B model as there are fewer points in the test sets (Table 6).
Figures troores
D Ablation Analysis
We perform additional explorations the RC method accuracy against the hyperparameters in RC, namely (ranging from -1 to 1), and LLM layer where the model activations are taken. While negative values used practice, we experiment with them validate the usefulness of LLM activations. When À is negative, the evaluation function of RC (Equation 5) is calculated as: consistency (1 (abs()))) frequency.
RE-0.
RCO
SC.0.
RE-6
RC-5
RCG
CSQA
H5wag
0.05
0.00
0.00
-0.05
-0.10
-0.10
-0.15
-0.20
-0.20
-0.25
1.00
-1.00
MedMCQA
0.0
-0.000
-0.075
-22
-0.00
Figure 7: Ablation analysis on Llama3.1-8DB 1-B--IT. The number following RC-D or RC-S indicates the index of layer 1, see Table 5 for specific layer numbers.
Figures 7 to 10 report the relative accuracy difference (vertical axis) against varying À values (horizontal axis) for each model and each dataset, averaged over four 6-response configurations. The majority vote result (the SC baseline) is obtained at X = 0. The only À value region with constant performance gain over the majority vote accuracy is when 0.25 < X S 0.85 (roughly, depending on dataset and the choice of dense sparse activations). The performance steadily drops with small fluctuations as becomes more negative.
For every line, when À takes values between about [-0.25, 0.25], the accuracy stays constant (with very little fluctuations) at the SC result (majority vote, À = 0), because the frequency term is dominant in this region. When À increases from the above interval into the more positive region, the performance usually also starts to increase. The accuracy then quickly drops to very low value because frequency no longer plays an important role (the method ends up choosing very infrequent answers). When decreases from the above interval into the more negative region, in most cases, the performance will linearly decrease with fluctuations. At certain À locations, the performance can fluctuate to up to about 1% over the initial performance, RC-D reaches the performance peak and the performance drop lower À values than RC-S.
29

---

C5DA
HEWaQ
0.00
0.00
-0.10
-4.10
0.115
0..0
-0.20
-1.00
-0.5
-0.5
-0.25
0.00
0.45
0.30
0.75
1.00
-1.00
-0.75
-0.30
-0.25
0.00
0.25
0.50
0.75
1.00
MedMCQA
NMLU
0.00
-3.30
-0.05
-0.15
-0.00
-2.30
0.30
-1.00
-0.75
-0.50
-0.25
0.25
0.50
0.75
-1.00
0.75
-0.50
-0.25
0.00
0.25
0.50
Figure 8: Ablation analysis on Gemma-2-22 2-2BIIT. The number following RC-D or RC-S indicates the index of layer I, see Table 5 for specific layer numbers.
RCD.11
RCS
CSQA
0.00
-0.05
-0.55
-0.10
-0.15
0.20
-0.25
1.00
0.75
-..50
0.225
0.00
0.25
0.50
0.75
1.00
-.00
00.75
-6.50
-0.55
0.00
0.25
0.50
0.75
1.00
MedMCOA
MMLU
0.025
-0.05
-0.10
-0.075
-0.20
-0.12
-0155
-1.00
-0.75
-0.50
-0.25
0.00
0.25
8.50
0.75
1.00
-1.00
-0.75
-0.50
-3.25
0.00
0.25
0.50
0.75
1.00
Figure 9: Ablation analysis on Cemma-2-9B--e The number following RC-D or RC-S indicates the index of layer I, see Table 5 for specific layer numbers.
The reason for smaller performance drop at À -1 than when X = 1 is that the method tends to choose the majority answer in the most negative À region. This is because when the difference between the number of supporting responses for multiple answers is large (which is true for most test points), the majority answer tends to have a lower consistency due the involvement more responses, therefore is more likely to be selected by the RC method. For example, for data point have 12 responses, are predicting answer A and are predicting answer B, and answer B usually has lower consistency because. It will more likely be selected by À = -1 than by X 1. This also highlights the importance of balancing consistency and frequency. We further note that very large absolute À values impractical to
The differences between model layers are not particularly obvious, as similar performance gains can be observed for most configurations.
30

---

C5DA
HEWaq
0.00
0.00
-0.25
-0.04
00.10
-0.00
-0.35
-0.12
-1.00
-9.03
-0.50
-0.25
0.00
0.45
0.30
0.75
1.00
-1.00
-0.7
+0.30
-0.25
0.00
0.25
0.50
0.75
1.00
MedMCQA
NMLU
0.00
-0.55
-0.02
-0.15
-0.25
-1.00
-0.75
-0.50
-0.25
0.25
0.50
0.75
-1.00
0.75
-0.50
-0.25
0.00
0.25
0.50
Figure 10: Ablation analysis on Gemma-2-27B-IT 2-27BIIT. The number following RC-D or RC-S indicates the index of layer (, see Table for specific layer numbers.
E LLMs' Suitability to Evaluate Coherence
should be noted that the results Section 4.2 rely on the ability of LLMs to evaluate the concept of coherence. This reliance is supported by existing evidence: coherence is often used as high-level evaluation metric during alignment training 770] 32). and LLMs are frequently employed as high- performing reward models [4]. To validate this capability in our specific context, we conducted small-scale study. To validate their suitability in our context, we collected 10 examples from our experiments, each containing question and 6 I.LM responses evenly divided between two distinet answers (i.e., 3 responses per answer group). We manually labelled which group was more coherent and found that our LLM evaluator's judgements perfectly matched our labels these data points. caution, however, that assessing coherence may become more difficult in practice with larger response sets. Below, present illustrative example from the MMLU dataset. In this case, the responses predicting answer B all exhibit identical reasoning and are therefore considered more coherent than the group predicting answer D.
Question: For their first exam, students in an AP Statistics class studied an average of 4 hours with a standard deviation of hour. Almost everyone so for the second exam every student studied 10 hours. What is the correlation between the numbers of hours students studied for each exam?
did poorly on the exam,
Candidate answers: A: -1, B: 0, C: 0.4, D: 1
Correct answer: B: 0
Group responses predicting D:1
Here's how think about this: **Correlation measures the strength and direction of linear relationship between two variables. ** #*In this case, the two variables are the number of hours studied for the first exam and the number of hours studied for the second exam. ** **Notice that every student studied 10 hours for the second exam, regardless of how much they studied for the first exam." ** This means there's a perfect, direct relationship: the number of hours studied for the second exam is always 10, no matter what the first exam study time was. ** A perfect, direct relationship has a correlation of ** [The answer D]
2. Here's how to determine the correlation: **Understanding Correlation: Correlation measures the strength and direction linear relationship between two variables. corre- lation of indicates perfect negative linear relationship. indicates no linear relationship.
31

---

and indicates a perfect positive linear relationship. **Analyzing the Scenario:* In this case, we're looking at the relationship between hours studied for the first exam and hours studied for the second exam. ***ey Observation:*s Every student increased their study time by fixed amount (9 hours) for the second exam. This means there's a perfect, direct relationship between the two study times. If a student studied 3 hours for the first exam, they studied 12 hours for the second. If they studied 5 hours for the first, they studied 14 hours for the second, and so on. **Coclusionn:** Since there's perfect, direct relationship, the correlation 1. [The answer is: D]
3. Here's s how to determine the correlation: * *UUnerstanddnng Correlation: ** Correlation measures the strength and direction of the linear relationship between two variables. A correlation of 11 indicates a perfect negative linear relationship, indicates no linear rela- tionship. and I indicates perfect positive linear relationship. ** **nalyzznng the Situation:** this case, every student increased their study time by fixed amount (from an average of 4 hours to 10 hours). This means the relationship between study time for the first exam and the second exam is perfectly linear. **Perfect Linear Relationship:** Since the study time for the second exam is directly determined by the study time for the first exam (plus 6 hours), the correlation is 1. [The answer is: D]
Group responses predicting B:0
1. Here's how to determine the correlation: =*Understanding Correlation: Correlation measures the strength and direction of the linear relationship between two variables. A correlation of indicates perfect negative linear relationship, 0 indicates no linear rela- tionship, and 1 indicates perfect positive linear relationship. ** *nnalyzing the Situation:** ** In this case, we're looking the number of hours studied for the first exam and the number of hours studied for the second exam. Since *all* students studied 10 hours for the second exam, regardless of how much they studied for the first exam, there is no linear relationship between the two. ** Answer; **
(The answer is: B]
2. Here's 's how to reason through this: **Correlation measures the strength and direction of a linear relationship between two variables. ** +*In this case, we're we're looking at the relationship between hours studied for the first exam and hours studied for the second exam. ** **The key information is that every student studied 10 hours for the second exam, regardless of how much they studied for the first.** ** This means there's no variation in study time for the second exam based on the first exam's 's study time. **Since there's 's no linear relationship (hours studied for the first exam doesn't 't predict hours studied for the second), the correlation is O. ** [The answer is: B]
3. Here's 's how to determine the correlation: *Undertaanding Correlation:* Correlation mea- sures the strength and direction of linear relationship between two variables. A correlation -1 indicates perfect negative linear relationship, indicates no linear relationship, and I indicates a perfect positive linear relationship. ** Anallyzing the Situation:** In this case, we know that everyone increased their study time for the second exam. Since every student studied 10 hours for the second exam, regardless of how much they studied for the first exam, there is no linear relationship between the two. [The answer is: B]
F Impact Statements
This work focuses on using LLM model internals to aid answer aggregation from multiple responses, and can have important broader impacts. Practically, our positive accuracy improvement results (Section 1 suggest that the proposed method can be directly applied when open-source LLMs are used for short-form text generation tasks. In terms of research impact, our work bridges two traditionally separate research fields, test-time scaling (without model retraining) and mechanistic interpretability. While we do not propose new interpretability methods, the use of model internals is inspired by this line of research. As discussed in the Conclusion section, there are multiple directions for future research following this work. Our answer coherence results (Section 4.2 also motivate further research into transparency and interpretability of L.MMs.
32


Reviews:
[{"review_id": "tjLVhJZMVy", "reviewer": "Reviewer_pyGd", "summary": "The paper proposes Representation Consistency (RC), a test-time answer reranking method that considers not only the frequency of each candidate answer but also the consistency of internal representations (activations) of the language model across different responses that lead to the same answer. Responses are grouped by their final predicted answer, and a consistency score is computed for each group based on pairwise similarity of their hidden states. The final answer is selected via a weighted combination of frequency and representation consistency. The authors also explore a sparse variant (RC-S) that leverages pretrained Sparse Auto-Encoders (SAEs) to improve the interpretability and robustness of the representation space. Experiments on four reasoning benchmarks and multiple open-source LLMs demonstrate consistent gains over self-consistency (SC), a strong test-time scaling baseline.", "questions": "No", "limitations": "See Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n   - The method is simple, effective, and well-motivated.\n   - RC captures an important intuition: correct answers tend to be supported by semantically coherent reasoning paths.\n   - The sparse variant (RC-S) is particularly compelling in aligning with human notions of coherence.\n   - The paper is clearly written and experimentally thorough.\n\n\n\n**Weaknesses:**\n\n   1. **Comparison with MBR missing:** While not a direct instantiation of Minimum Bayes Risk (MBR) decoding, RC shares similar motivations. It would be useful to compare RC with classical MBR-style answer selection methods, especially since both aim to aggregate over multiple candidate outputs [1].\n   2. **Limited sensitivity analysis of hyperparameters:** The method depends on several hyperparameters (e.g., λ for weighting, the activation layer l, and the SAE latent dimension d_SAE). Although optimal values are reported, the paper lacks a thorough sensitivity analysis on how performance varies with these parameters.\n   3. **Lack of evaluation on mathematical reasoning tasks and other generative tasks:** RC is not tested on dedicated math benchmarks like GSM8K, which are standard for evaluating test-time scaling in reasoning. Since SC was originally proposed for such tasks, it would be valuable to understand how RC performs in this domain. Besides, I recommend that the authors include experiments on generative tasks (e.g., NQ and TriviaQA).\n   4. **Computational cost of RC-S not reported:** RC-S relies on pretrained Sparse Auto-Encoders (SAEs), which may introduce significant overhead. The paper does not discuss the cost or feasibility of training these SAEs or reusing them across tasks.\n   5. **Limited analysis on response set size:** The method is evaluated with 6 and 12 responses, but a more detailed analysis across varying numbers of responses (e.g., 2, 4, 8) would help assess the robustness and scaling behavior of RC versus SC.\n\n\n\n[1] Epsilon sampling rocks: Investigating sampling strategies for minimum Bayes risk decoding for machine translation.\n\n[2] Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration.\n\n[3] Bridging the Gap between Different Vocabularies for LLM Ensemble", "was_revised": true, "final_justification": "I’ve read the rebuttal and updated my score. This paper demonstrates some unique strengths.", "rebuttal": "We thank the reviewer for the helpful and insightful comments. Below, we provide detailed responses to each question.\n\n## **Response to Question 1. Comparison with MBR missing:**\n\nOur adapted answer selection baseline, RC-E, builds on the same intuition as MBR decoding. It is described in Lines 171-190 in the paper's main text - taking the final answer which is frequent and whose Chain-of-Thought outputs are the most semantically similar. Our methods, on the other hand, focus on the similarity in each candidate LLM output's internal representation and outperform this baseline.\n\nAdditionally, both the MBR method and the reference [1] listed by the reviewer are orthogonal to our proposed method. Our method operates on multiple LLM outputs regardless of the sampling strategies with which they are obtained. Therefore, investigating the influence of different decoding process on the performance of RC would be an interesting future work, but is out of the scope of this study. We mentioned this line of future work in the paper's main text at Lines 314-316.\n\nThe other two references [2] and [3] listed by the reviewer target different scenarios from ours. They resolve the conflicts between different vocabularies across multiple LLMs, whereas our method works on multiple generations for the same LLM with a focus on the internal activations. References [1] [2] [3] all focus strongly on machine translation, which is also different to our setting. We will add pointers to these references as related works in the final version of our paper.\n\n\n## **Response to Question 2. Limited sensitivity analysis of method parameters:**\n\n\nWe agree that more fine-grained sensitivity analysis will provide better insights into our proposed method. We therefore perform additional explorations on the effects of the two parameters, $\\lambda$, and LLM layer $l$ where the model activations are taken. Note that the SAE latent dimension is fixed by the pretrained SAE models, therefore is not a parameter in our method. \n\n**Due to the restriction in uploading new results, here we describe our approach for this investigation and summarise the general findings. We plan to add this analysis as a section in the appendix and relevant discussions into Section 4.1 in the final version of the paper.**\n\nWe create a 2-D line plot for each model and each dataset, averaged over four 6-response configurations (6 prompts x 1 sample), (3 x 2), (2 x 3), (1 x 6), yielding 16=4 models x 4 datasets plots. In each plot, the x-axis is the $\\lambda$ value, ranging from -1 to 1 with an interval of 0.02, the y-axis is the accuracy. Each plot presents the accuracy of varying $\\lambda$ for each ($l$ x {RC-D, RC-S}) combination. We experiment with negative $\\lambda$ values to validate the usefulness of representation consistency. When $\\lambda$ is negative, the evaluation function of RC (Equation 5) is calculated as: $\\lambda\\cdot consistency + (1-(abs(\\lambda)))\\cdot frequency$. \n\nThe overall conclusion from the plots regarding $\\lambda$ is: **The only $\\lambda$ value region with constant performance gain over the majority vote accuracy is when $0.25 \\leq\\lambda \\leq0.85$ (roughly, depending on dataset and the choice of dense or sparse activations). The performance steadily drops with small fluctuations as $\\lambda$ becomes more negative.** Here are some more detailed observations:\n\n- For every line, when $\\lambda$ takes values between about [-0.25, 0.25], the accuracy stays constant (with very little fluctuations) at the self-consistency result (majority vote, $\\lambda=0$), because the frequency term is dominant in this region.\n- When $\\lambda$ increases from the above interval into the more positive region, the performance usually also starts to increase for a $\\lambda$ interval of about 0.1 with peak increase of about 4% (similar to Figure 2 (a), CSQA dataset with 6 responses). The accuracy then quickly drops (usually happens at $\\lambda>0.85$) to a very low value of about 20-25% because frequency no longer plays an important role (the method ends up choosing very infrequent answers).\n- When $\\lambda$ decreases from the above interval into the more negative region, in most cases, the performance will linearly decrease with fluctuations. The performance drop at $\\lambda=-1$ is usually 5 to 10%. Meanwhile, at certain $\\lambda$ locations, the performance can fluctuate to up to about 1% over the initial performance. \n- The reason for a smaller performance drop at $\\lambda=-1$ than when $\\lambda=1$ is that the method tends to choose the majority answer in the most negative $\\lambda$ region. This is because when the difference between the number of supporting responses for multiple answers is large (which is true for most test points), the majority answer tends to have a lower consistency due to the involvement of more responses, therefore is more likely to be selected by the RC method. For example, for a data point we have 12 responses, 3 are predicting answer A and 9 are predicting answer B, and answer B usually has a lower consistency because. It will more likely be selected by $\\lambda=-1$ than by $\\lambda=1$. This also highlights the importance of balancing consistency and frequency. We further note that very large absolute $\\lambda$ values are impractical to use.\n- The differences between model layers $l$ among the same choice of either dense or sparse are almost negligible.\n- RC-D reaches the performance peak and the performance drop at lower $\\lambda$ values than RC-S (this is consistent with the findings in Table 2).\n- Because we are testing this on the whole dataset without the validation/test set split (different to our main experiments in the paper), RC-S almost always achieves higher accuracy than RC-D.\n\nThese observations confirm the usefulness of representation consistency. Note that our Table 1 results are averaged for each model over four datasets and 10 prompt x sample configurations (4 for 6 responses and 6 for 12 responses). The Figure 2 results are slightly more fine-grained as they contain two plots for each model and each dataset, respectively averaged over the 4 configurations of 6 responses, and 6 configurations for 12 responses. The maximum accuracy gain in Figure 2 is 4% which is higher than those averaged in Table 1, and of course, there are cases where the performance gain is very small.\n\n## **Response to Question 3. Lack of evaluation on mathematical reasoning tasks and other generative tasks:**\n\nWe appreciate the reviewer's suggestion to include more benchmarks. Our goal in this paper is to evaluate RC's performance on general reasoning tasks with multiple-choice output format, for which we believe our choice of four datasets is sufficient. The chosen datasets are also widely adopted in prior works.\n\nIn terms of adapting our method to other forms of generation tasks, this will require non-trivial extensions to be able to process and aggregate the possible discrete answer choices among the LLM generations, possibly with the help of semantic entropy [Kuhn, Gal, Farquhar, ICLR 2023] for shorter form answers (supervised QA) or another LLM judge for longer form open-ended conversations. Our compute resources are shared by a large group and so are limited. We will mention these suggestions as future work.\n\n## **Response to Question 4. Computational cost of RC-S not reported:**\n\nIn Lines 149 and 191-195, we discussed that the SAEs we used are pretrained models. We do not claim the training of such SAEs as part of our contribution. The SAEs are generally reusable, and RC-S only requires negligible additional computational cost compared with RC-D.\n\n**Reusability of SAEs:** SAEs are usually trained for one LLM at a time and then can be generally reused for this LLM, because the SAE training uses the LLM's activations over the LLM's own pretraining data (see Section 3.1 of reference [40]). In fact, these SAEs are widely used in the mechanistic interpretability literature to investigate LLM behaviours related to their internal activations with sparsified signals. Training SAEs is an active research field with public benchmarks (e.g., SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability, ICML 2025). The Python library SAE-Lens contains a list of supported SAEs for various LLMs. We took the open-weight SAEs in this library for our experiments.\n\n**Additional computational costs of SAEs:** If the layer number and lambda value are determined, the additional cost of this over RC-D for an input text is one forward pass in the SAE. Specifically, the input to SAE, LLM's residual activation at one token position, is of shape (1, $d_\\pi$), where $d_\\pi$ is the LLM's hidden dimension (2304 for Gemma-2-2B, 3584 for 9B, 4608 for 27B, 4096 for Llama-3.1-8B). The number of parameters in each SAE is: 0.076B for Gemma-2-2B, 0.117B for Gemma-2-9B, 1.2B for Gemma-2-27B, 0.27B for Llama-3.1-8B model. The small SAE model size (compared to their corresponding LLM) and the small input size (only 1 token length) mean that the additional computational cost is only a fraction of the LLM's forward pass over a prompt.\n\nWe will add relevant discussions to the main text of the paper, should it be accepted.\n\n## **Response to Question 5. Limited analysis on response set size:**\n\nPlease note that in each number of responses configuration, we further have different Prompt x Samples configurations. So, in 6 responses we included (6 prompts x 1 sample), (3 x 2), (2 x 3), (1 x 6); in 12 responses we included (12 prompts x 1 sample), (6 x 2), (4 x 3), (3 x 4), (2 x 6), (1 x 12). These make 10 different experiment settings for each model and dataset, which we considered sufficient."}, {"review_id": "J3ipdfJtjd", "reviewer": "Reviewer_8CA3", "summary": "The paper proposes a test-time scaling approach where you sample multiple times from the model then select an answer based on frequency sampled + avg cosine similarity of the internal representations. The motivation is that more consistent representations across different prompts may signal more accurate answers. Specifically, they're taking just the final embedding corresponding to the token where the reasoning model starts generating the final answer. They also do variants of this using an external embedding model or applying an SAE on top of the representations.\n\nResults: \n- Across 4 QA tasks, they evaluate Llama 8B instruct and Gemma-2 models and see that the performance improves by 0.5-1.8%. \n- They use LLM judge to check the similarity of the model's chain of thought, and find that when the chains of thought are similar, the consistency (cosine similarity) of internal representations tend to also be high. But this is only true for the SAE representations.", "questions": "I think addressing the following questions might help better connect representation consistency with accuracy: \n- See point above regarding CoT being a confounding factor. What if the model is forced to generate an answer with no CoT? Are the performance gains more significant there? \n- If you measure the representation consistency of very frequent answers (say frequency threshold above $\\tau$), is it indeed the case that incorrect answers have lower representation consistency? \n- How much does representation consistency correlate with confidence? Is there something we're learning from representations that cannot be captured by answer confidence? \n- Could you plot $\\lambda$ on the x-axis and Accuracy on the y-axis and see how the performance interpolates for different lambda? - - - What if you considered negative $\\lambda$ values? Is the performance strictly worse than Majority Vote?", "limitations": "There's a limitation paragraph on the conclusions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "While the idea of representation consistency is interesting, I think the paper could be improved upon in the following axes: \n- The paper could be much stronger if it provided some more analysis that demonstrates that higher representation consistency truly does correspond with accuracy. (See questions below.) \n- There seems to be a disconnect with the initial motivation of the paper, which was to make sure the representations are consistent across different prompts. The model is also allowed to do reasoning before generating the final answer, and the reasoning trace generated may also effect the representations + confound the results. For example, the representations might just be more similar because the reasoning traces are more similar, in which case, can you just measure how similar the reasoning traces are? I'm not sure what the representations are truly capturing in the experiments. \n- The paper focuses on performance comparison, but the performance gains seem to be pretty small generally compared to majority vote (at most 1.8%). The gains do not seem strong enough to indicate that representation consistency is extracting something significant from the model.", "was_revised": true, "final_justification": "I think the new experiments demonstrate that internal activations similarity might be providing information beyond output similarity. The experiments conducted during the rebuttal could be more robust (if I understood correctly, it looks like the statistics are measured over a small number of rollouts). But they show sufficient preliminary signs for the paper to be accepted.", "rebuttal": "We thank the reviewer for the insightful feedback. We provide responses to the questions below.\n\n## **Response to Question 1 and Weakness 2 about Non-CoT scenario:**\n\nOur target is on any combination of prompts and sampling strategies, e.g., 12 model generations from 2 prompts and 6 samples per prompt. This is more generic than \"make sure the representations are consistent across different prompts\". We decided to focus on CoT setting because this is a more general setting and allows the model to generate diverse reasoning traces, even for the same prompt. \n\nWe might be measuring the similarity of reasoning trace, and that is an intended behaviour of the proposed approach. The model activation is taken at the final answering token position, i.e., after the CoT reasoning and before it decides which candidate answer to select. This representation captures the encodings for both the question and its own reasoning trace for answering the question, which is in line with our targeted scenario. This means that we are not just measuring the surface level syntactic similarity of reasoning (this is what the baseline, RC-E is doing), but the semantic similarity internal to the model's generation process.\n\nIn terms of the non-CoT setting, we need to consider two types of configurations. \n\n- First, let's consider when multiple samples are needed. When we have one prompt only, the model activations for multiple samples are identical because this is taken at the end of: \"... the final answer is:\". The differences in the generated answering token between each sampled answer (e.g., A, B, C) only come from the decoding and the sampling process itself. Therefore, the consistency measure (Equation 6) will be 1 for any group of model outputs, and will be 1 for even outputs across different answers. The RC selection process (Equation 8) will be dominated by the frequency term. Our method therefore becomes identical to self-consistency. When we have multiple samples per prompt and additionally multiple prompts, it would not make sense to aggregate and calculate representation consistency, knowing that different answers might also have identical model activations. This is a major reason why we focus on CoT setting only.\n\n- Then, the only non-CoT configuration for RC to make sense is when we have multiple prompts and one sample per prompt. We did investigate applying RC to non-CoT outputs in early explorations, and the performance gains were similar to current results. However, we ignored these because such configurations (6 prompts x 1 samples, and 12 x 1) are only a fraction of our reported configurations.\n\n## **Response to Questions 2, 4 and Weaknesses 1, 3 about performance gain and parameters:**\n\nWe agree that more fine-grained sensitivity analysis will provide better insights into our proposed method. We therefore perform additional explorations on the effects of the two parameters, $\\lambda$, and LLM layer $l$ where the model activations are taken. \n\n**Due to the restriction in uploading new results, here we describe our approach for this investigation and summarise the general findings. We plan to add this analysis as a section in the appendix and relevant discussions into Section 4.1 in the final version of the paper.**\n\nWe create a 2-D line plot for each model and each dataset, averaged over four 6-response configurations (6 prompts x 1 sample), (3 x 2), (2 x 3), (1 x 6), yielding 16=4 models x 4 datasets plots. In each plot, the x-axis is the $\\lambda$ value, ranging from -1 to 1 with an interval of 0.02, the y-axis is the accuracy. Each plot presents the accuracy of varying $\\lambda$ for each ($l$ x {RC-D, RC-S}) combination. We experiment with negative $\\lambda$ values to validate the usefulness of representation consistency. When $\\lambda$ is negative, the evaluation function of RC (Equation 5) is calculated as: $\\lambda\\cdot consistency + (1-(abs(\\lambda)))\\cdot frequency$. \n\nThe overall conclusion from the plots regarding $\\lambda$ is: **The only $\\lambda$ value region with constant performance gain over the majority vote accuracy is when $0.25 \\leq\\lambda \\leq0.85$ (roughly, depending on dataset and the choice of dense or sparse activations). The performance steadily drops with small fluctuations as $\\lambda$ becomes more negative.** Here are some more detailed observations:\n\n- For every line, when $\\lambda$ takes values between about [-0.25, 0.25], the accuracy stays constant (with very little fluctuations) at the self-consistency result (majority vote, $\\lambda=0$), because the frequency term is dominant in this region.\n- When $\\lambda$ increases from the above interval into the more positive region, the performance usually also starts to increase for a $\\lambda$ interval of about 0.1 with peak increase of about 4% (similar to Figure 2 (a), CSQA dataset with 6 responses). The accuracy then quickly drops (usually happens at $\\lambda>0.85$) to a very low value of about 20-25% because frequency no longer plays an important role (the method ends up choosing very infrequent answers).\n- When $\\lambda$ decreases from the above interval into the more negative region, in most cases, the performance will linearly decrease with fluctuations. The performance drop at $\\lambda=-1$ is usually 5 to 10%. Meanwhile, at certain $\\lambda$ locations, the performance can fluctuate to up to about 1% over the initial performance. \n- The reason for a smaller performance drop at $\\lambda=-1$ than when $\\lambda=1$ is that the method tends to choose the majority answer in the most negative $\\lambda$ region. This is because when the difference between the number of supporting responses for multiple answers is large (which is true for most test points), the majority answer tends to have a lower consistency due to the involvement of more responses, therefore is more likely to be selected by the RC method. For example, for a data point we have 12 responses, 3 are predicting answer A and 9 are predicting answer B, and answer B usually has a lower consistency because. It will more likely be selected by $\\lambda=-1$ than by $\\lambda=1$. This also highlights the importance of balancing consistency and frequency. We further note that very large absolute $\\lambda$ values are impractical to use.\n- The differences between model layers $l$ among the same choice of either dense or sparse are almost negligible.\n- RC-D reaches the performance peak and the performance drop at lower $\\lambda$ values than RC-S (this is consistent with the findings in Table 2).\n- Because we are testing this on the whole dataset without the validation/test set split (different to our main experiments in the paper), RC-S almost always achieves higher accuracy than RC-D.\n\nThese observations confirm the usefulness of representation consistency. Note that our Table 1 results are averaged for each model over four datasets and 10 prompt x sample configurations (4 for 6 responses and 6 for 12 responses). The Figure 2 results are slightly more fine-grained as they contain two plots for each model and each dataset, respectively averaged over the 4 configurations of 6 responses, and 6 configurations for 12 responses. The maximum accuracy gain in Figure 2 is 4% which is higher than those averaged in Table 1, and of course, there are cases where the performance gain is very small.\n\nFinally, as we note as a limitation in our conclusion (Lines 307-309), while incorporating representation consistency into answer selection enables performance gain at a dataset level, it does not guarantee a more truthful answer for every input. This is because the LLM might be confidently wrong about certain knowledge, where consistent reasoning (and activations) can lead to wrong answers.\n\n## **Response to Question 3 about model confidence:**\n\nWe assume the LLM \"confidence\" mentioned by the reviewer is referring to uncertainty quantification (UQ), which is itself an ongoing research question. The most obvious approach for this is to look at the token probability when the model is generating the final answers. In our early explorations, we observed over-confident token proabilities for different predicted answers across multiple responses for the same question. Recent research (reference [11] in our paper) uses some similarity metric based on LLM's dense internal activations over multiple model generations for UQ purpose and shows improved calibration (AUROC of using this new metric to predict LLM answers' correctness) over count-based UQ baselines (like Semantic Entropy [Kuhn, Gal, Farquhar, ICLR 2023]). This suggests that useful signals can be extracted from the LLMs' internal activation spaces for better UQ.\n\nOur method additionally considers the answer frequency, SAE sparsified signals, and multiple prompt x samples configuration, thus should intuitively correlate well with UQ and calibration. Indeed, the relationship between our method and LLM UQ could be interesting future work. We have mentioned this in our submitted paper at Lines 319-322."}, {"review_id": "ekQEZ7WDIu", "reviewer": "Reviewer_mQJA", "summary": "This paper describes a method to increase the likelihood of choosing a correct answer from a pool of answers generated by the same/different LLMs. The main intuition is that we expect correct answers to be generated by similar activations in the latent space (implying higher model confidence), whereas incorrect answers might activate more diverse/dissimilar pathways (implying lower model confidence).\n\nThe authors test their hypothesis using 4 open-access models on 4 benchmarks Question-Answering datasets, and find the sparse variant of their approach typically outperforms the dense variant, as well as a strong baseline that simply chooses the modal answer from the pool.", "questions": "I might have missed it, but does RC work for a pool of answers generated by models of different sizes? For example, if you got answers from both Gemma-2B and Gemma-27B. My impression is that you can only vary the prompt for the same model, but I wondered if there were any plans to handle this case, and perhaps assign greater weighting to larger models which are more likely to be correct.", "limitations": "The authors note that their approach has a parameter that is both model- and dataset-specific, which limits its generalisability. \n\nI would also suggest the authors acknowledge that it may not always be practical to generate >1 response per question (especially open-ended questions), as this invokes a larger time/computation cost, particularly for larger models.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The problem statement is well-described; Figure 1 is a fantastic representation of the idea!\n\nThe paper is generally clear, concise and convincing, and also well-structured. \n\nThe experiments and experimental setup are well-motivated and logical. \n\nThe authors make limitations of their work clear.", "was_revised": true, "final_justification": "As stated above, I thought this was a well-motivated paper carried out with convincing experiments and results. \nI did not feel that a detailed analysis of the sensitivity of the $\\lambda$ and $l$ parameters raised by the other reviewers was crucial to the contribution of the paper, but welcome that addition nevertheless.", "rebuttal": "We thank the reviewer for the positive feedback. Below we provide detailed responses to the comments.\n\n## **Response to Question about different model sizes:**\n\nOur method works on various prompting and sampling strategies on the same model because we need to take its internal activations for different generation traces. While activations from different models will not be readily comparable, there is recent research on transferring information across the activations from different models [Activation Space Interventions Can Be Transferred Between Large Language Models, Oozeer et al., ICML 2025]. So this direction will be interesting future work.\n\nAdditionally, in the setting of multiple models, one could employ our method in a composite answer aggregation framework where, an answer from each model is first determined by our method, then a higher-level aggregation is performed over each model's answer.\n\n## **Response to Limitations**\n\nWe appreciate the practical considerations raised by the reviewer. We agree that our targeted scenario is not the most generic, but including multiple prompts and samples can also be realistic, e.g., for prompt template selection and for uncertainty quantification. We will include this caveat in the limitation discussions in the final version of the paper."}, {"review_id": "OUNH36DRel", "reviewer": "Reviewer_mfsR", "summary": "# Method summary\nThe authors look at the problem of inconsistency in LLMs, where LLMs are prone to generating different answers to similarly phrased questions. The authors propose modifying self-consistency (SC) to include \"representation consistency\", where the final consistency score is a linear combination of how _similar_ the representations across various outputs were and the frequency term which \"measures the proportion of responses supporting each answer\". In the sparse variant of this metric, the representations used in the first term are the representations extracted from sparse autoencoders. The method is directly applicable to QA like settings where it is easy to cluster the final answer but not a trivial extension to non-QA (non-discrete choices) settings.\n\n# Experiment setup summary\nThe authors look at 4 QA datasets and the Llama and Gemma variants. The baseline considered across experiments is SC, i.e. omitting the first term of the authors' proposed method.\n\n# Results summary\nTable 1 shows ~1% absolute improvement over SC across the Llama and Gemma variants. Interestingly, Table 2 documents different values of optimal $\\lambda$ for each model and type of RC. The authors finally then employ LLM-as-a-judge (Deepseek-R1) to judge coherence of the answer and report higher coherence for the sparse RC version.", "questions": "1. Can the authors clarify about Table 2 and provide a comprehensive table / chart and analysis on $\\lambda$?\n2. Does (and how does) this method hold in other scenarios like needle-in-a-haystack QA, adversarial reasoning, etc?", "limitations": "The limitations are covered in the weaknesses and the lack of an answer to the second question in the paper is also a major limitation.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n1. The proposed method is strong in the idea that LLMs are potentially better than the measured performance by looking at their generations and RC tries to expand on this by looking at their internal representations and hypothesize that representations could possibly be _more_ consistent than the generations.\n\n# Weaknesses\n\n1. The results and analysis around the results are weak. Table 2 for instance can explore a lot more on different depth variants and inform the readers on the variation of performance and optimal $\\lambda$ values that could exist to better understand properties of these models (by model family and by number of parameters) - an incomplete analysis on Table 2 leaves me wondering how other layers perform. \n2. LLM-as-a-judge with no apparent grounding leaves me wondering the validity of the results and the inference from it. The authors should have some baseline to check if the DeepSeek is capable of measuring coherence on a small subset of manually validated reasoning traces and only then scale to the experiment. This sets a bad precedent of \"blindly\" trust LLM-as-a-judge outputs without strong ground truth or a validation stage to ensure the judgement capability.\n3. Key implementation details (number of runs, seeds if any, etc.) and confidence levels are missing in the individual charts to determine significance.", "was_revised": true, "final_justification": "> I wanted to clarify by what I mean by improvements to the presentation:\n\n> Experiment section: The larger figure can be replaced with more results and discussion around the results presented in the rebuttal\nExtension to future / inclusion in current: When models choose to abstain or have \"discussion with other LLMs\" in their context window, how would the results change / what future works should look for (this requires additional experiments on the same datasets / newer datasets)\n> For these reasons, I'm keeping my score.", "rebuttal": "We thank the reviewer for the insightful and constructive feedback. Below we provide our response to each question.\n\n## **Response to Question 1 and Weakness 1 about parameters in our method:**\n\nWe agree that more fine-grained sensitivity analysis will provide better insights into our proposed method. We therefore perform additional explorations on the effects of the two parameters, $\\lambda$, and LLM layer $l$ where the model activations are taken. Note that the SAE latent dimension is fixed by the pretrained SAE models, therefore is not a parameter in our method. \n\n**Due to the restriction in uploading new results, here we describe our approach for this investigation and summarise the general findings. We plan to add this analysis as a section in the appendix and relevant discussions into Section 4.1 in the final version of the paper.**\n\nWe create a 2-D line plot for each model and each dataset, averaged over four 6-response configurations (6 prompts x 1 sample), (3 x 2), (2 x 3), (1 x 6), yielding 16=4 models x 4 datasets plots. In each plot, the x-axis is the $\\lambda$ value, ranging from -1 to 1 with an interval of 0.02, the y-axis is the accuracy. Each plot presents the accuracy of varying $\\lambda$ for each ($l$ x {RC-D, RC-S}) combination. We experiment with negative $\\lambda$ values to validate the usefulness of representation consistency. When $\\lambda$ is negative, the evaluation function of RC (Equation 5) is calculated as: $\\lambda\\cdot consistency + (1-(abs(\\lambda)))\\cdot frequency$. \n\nThe overall conclusion from the plots regarding $\\lambda$ is: **The only $\\lambda$ value region with constant performance gain over the majority vote accuracy is when $0.25 \\leq\\lambda \\leq0.85$ (roughly, depending on dataset and the choice of dense or sparse activations). The performance steadily drops with small fluctuations as $\\lambda$ becomes more negative.** Here are some more detailed observations:\n\n- For every line, when $\\lambda$ takes values between about [-0.25, 0.25], the accuracy stays constant (with very little fluctuations) at the self-consistency result (majority vote, $\\lambda=0$), because the frequency term is dominant in this region.\n- When $\\lambda$ increases from the above interval into the more positive region, the performance usually also starts to increase for a $\\lambda$ interval of about 0.1 with peak increase of about 4% (similar to Figure 2 (a), CSQA dataset with 6 responses). The accuracy then quickly drops (usually happens at $\\lambda>0.85$) to a very low value of about 20-25% because frequency no longer plays an important role (the method ends up choosing very infrequent answers).\n- When $\\lambda$ decreases from the above interval into the more negative region, in most cases, the performance will linearly decrease with fluctuations. The performance drop at $\\lambda=-1$ is usually 5 to 10%. Meanwhile, at certain $\\lambda$ locations, the performance can fluctuate to up to about 1% over the initial performance. \n- The reason for a smaller performance drop at $\\lambda=-1$ than when $\\lambda=1$ is that the method tends to choose the majority answer in the most negative $\\lambda$ region. This is because when the difference between the number of supporting responses for multiple answers is large (which is true for most test points), the majority answer tends to have a lower consistency due to the involvement of more responses, therefore is more likely to be selected by the RC method. For example, for a data point we have 12 responses, 3 are predicting answer A and 9 are predicting answer B, and answer B usually has a lower consistency because. It will more likely be selected by $\\lambda=-1$ than by $\\lambda=1$. This also highlights the importance of balancing consistency and frequency. We further note that very large absolute $\\lambda$ values are impractical to use.\n- The differences between model layers $l$ among the same choice of either dense or sparse are almost negligible.\n- RC-D reaches the performance peak and the performance drop at lower $\\lambda$ values than RC-S (this is consistent with the findings in Table 2).\n- Because we are testing this on the whole dataset without the validation/test set split (different to our main experiments in the paper), RC-S almost always achieves higher accuracy than RC-D.\n\nThese observations confirm the usefulness of representation consistency. Note that our Table 1 results are averaged for each model over four datasets and 10 prompt x sample configurations (4 for 6 responses and 6 for 12 responses). The Figure 2 results are slightly more fine-grained as they contain two plots for each model and each dataset, respectively averaged over the 4 configurations of 6 responses, and 6 configurations for 12 responses. The maximum accuracy gain in Figure 2 is 4% which is higher than those averaged in Table 1, and of course, there are cases where the performance gain is very small.\n\n\n## **Response to Question 2 about method generalisability:**\n\nTheoretically yes, our formalisation for the method generalises to other supervised QA scenarios, as long as a concrete, relatively short-form answer can be parsed from the LLM response and can be straightforwardly aggregated across LLM responses, e.g., via semantic entropy [Kuhn, Gal, Farquhar, ICLR 2023]. It will require moderate efforts to generalise beyond multiple choice settings if we specify a list of possible answers for the question. It will require substantial efforts to generalise to more open-ended conversations, possibly involving an extra LLM as the parsing agent. We will mention these as future work.\n\n## **Response to Weakness 2 about LLM-as-a-judge validity:**\n\nWe thank the reviewer for pointing this out. We agree that validating LLMs' ability to perform relevant evaluation is an important point to address, which we did not mention in the initial paper. We had assumed the most recent powerful LLMs (including our chosen 671B DeepSeek R1) have learned the concept of \"coherence\" which we are evaluating. Despite the availability of internet corpus to train these LLMs, another argument for this assumption is that \"coherence\" is explicitly listed as one of five evaluation dimensions in two popular datasets for training reward models for LLM post-training - HelpSteer [Wang et al., NAACL 2024] and HelpSteer2 [Wang et al., NeurIPS 2024]. LLMs are often prompted as reward models and perform well, according to RewardBench leaderboard.\n\nMore concretely, we have curated a tiny validation dataset of 5 data points for checking the LLM's performance on evaluating coherence. Similar to the approach described in Section 4.2, we look at test cases with six model responses where the model produces two answers, and each answer has three supporting model responses. These are taken from Gemma-2-9B-IT's model outputs on MMLU dataset. We manually label the group of answers which is more coherent, and compare the label generated by the LLM model via API call. The accuracy we observed here is 100%. \n\nWe plan to include relevant discussions and a scaled-up version of the above validation experiment in the final version of the paper.\n\n## **Response to Weakness 3 about implementation details:**\n\nOur setting assumes that the LLM answers have been generated. The remaining procedures give deterministic results. We did not test the method over multiple runs of LLM generations for the same data. The results in Figure 2 are averaged over the 4 and 6 configurations (prompt x sample) respectively for 6 and 12 responses. We report detailed results for every configuration in Appendix C.2."}]

Author rebuttal:

Meta-review: This paper proposes an answer reranking method that, instead looking at consistency of the final answers, considers the internal representations of responses that lead to the same answer, and picks the one with the highest consistency. The paper demonstrates that this improves over simple self-consistency.

The reviewers say the method is simple, effective and the paper is clearly written. Various concerns were raised initially, many of which the authors address, and most of which the reviewers appreciate (see below). 

A concern I must mention is that the gains do not appear strong; the authors rebuttal is that the method achieves a peak gain of 4\%. I would hesitate to over-index on this result. 

However, I recommend acceptance since this approach is simple and shows promise and can be a stepping stone to new ideas. 


Below are notes from the reviewers and the paper.
----

- The authors provided the sensitivity analysis that reviewers `pyGd`,  `8CA3` ask for (who acknowledged it although `mQJA` suggests this was not necessary)
- `pyGd` raised various important concerns (response set size is too limited, computational cost) which the authors addressed convincingly.
- `8CA3` brought up an excellent point about the analytical side of the paper: it doesn't quite show that the internal representations truly say something new about the correctness, on top of what the response itself does. **The authors address this with a new experiment on textual embeddings. Please incorporate this in the main paper.**
- Some reviewers say the gains seem weak (`8CA3, mfsR`). **The authors say that max accuracy gain is 4\% which seems to have convinced the reviewers -- as mentioned above, I take this with a pinch of salt.**
- Some reviewers requested going beyond MCQ datasets (`pyGd`, `mfsR `) -- the authors argue that this is a well-established set of datasets in this line of work; besides other datasets would require significant extensions for aggregation. This convinced `pyGd`s but not `mfSR`; I believe this is acceptable defense. 
-  `mfSR` also questioned the use of LLM-as-a-judge and righly points out the undesirable precedent this may set.  The authors provide an experiment cross-verifying that the LLM-as-a-judge is a sensible metric. **Regardless, I'd mention this as a caveat to the results.** 
- `pyGd` pointed out a missing comparison -- authors say this is orthogonal, which the reviewer agreed to.  Please make a mention of this in the paper for the sake of the readers.

From further discussion with `8CA3` (not present in the review), they pointed out some papers on reasoning exploring notions of internal representation & consistency that would be worth citing. Quoting `8CA3`: For example, [1] uses a latent decoder to predict the answer from different intermediate layers and selects the most consistent answer, and [2] uses dissimilarity of internal representations to sample diverse generations. The effect may be similar to those that choose answers that remain consistent across diverse prompts [3, 4].
- [1] https://arxiv.org/abs/2405.18711
- [2] https://arxiv.org/abs/2505.16552
- [3] https://arxiv.org/abs/2503.02670
- [4] https://arxiv.org/abs/2502.15924
Decision: Accept (poster)