Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
Authors: Li Ji-An
Hua-Dong Xiong
Robert Wilson
Marcelo G Mattar
Marcus K. Benna
Venue: NeurIPS 2025 poster ()
Keywords: large language model
metacognition
safety
neurofeedback
in-context learning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: This paper demonstrates that language models possess metacognitive-like abilities to monitor and control their internal neural activations.

Abstract: Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, yet at other times seem unable to recognize those strategies that govern their behavior. This suggests a limited degree of metacognition --- the capacity to monitor one's own cognitive processes for subsequent reporting and self-control. Metacognition enhances LLMs' capabilities in solving complex tasks but also raises safety concerns, as models may obfuscate their internal processes to evade neural-activation-based oversight (e.g., safety detector). Given society's increased reliance on these models, it is critical that we understand their metacognitive abilities. To address this, we introduce a neuroscience-inspired \emph{neurofeedback} paradigm that uses in-context learning to quantify metacognitive abilities of LLMs to \textit{report} and \textit{control} their activation patterns. 
We demonstrate that their abilities depend on several factors: the number of in-context examples provided, the semantic interpretability of the neural activation direction (to be reported/controlled), and the variance explained by that direction. These directions span a ``metacognitive space'' with dimensionality much lower than the model's neural space, suggesting LLMs can monitor only a small subset of their neural activations. Our paradigm provides empirical evidence to quantify metacognition in LLMs, with significant implications for AI safety (e.g., adversarial attack and defense).

Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
Li Ji-An
Hua-Dong Xiong School of Psychology Georgia Tech
Robert C. Wilson
Neurosciences Graduate Program University of California San Diego
School of Psychology Georgia Tech
Marcelo G. Mattar
Marcus K. Benna Department of Neurobiology University California San Diego
Department of Psychology New York University
Abstract
Large language models (LLMs) can sometimes report the strategies they actually to solve tasks, yet other times seem unable recognize those strategies that govern their behavior. This suggests limited degree of metacognition the capacity to monitor one's own cognitive processes for subsequent reporting and self-control. Metacognition enhances LLMs' capabilities in solving complex tasks but also raises safety concerns, as models may obfuscate their internal processes to evade -ouual-aciivttion- increased reliance on these models, it critical that we understand their metacogni- tive abilities. To address this, we introduce a neuroscience neuroscience--ispprdd inspired newrofeedback paradigm that uses in-context learning to quantify metacognitive abilities of L.LMs to report and control their activation patterns. We demonstrate that their abilities depend on several factors: the number in-context examples provided, the seman- tic interpretability of the neural activation direction (to be reported/controlled), and the variance explained by that direction. These directions span a 'metacognitive space" with dimensionality much lower than the model's 's neural space, suggesting LLMs can monitor only a small subset of their neural activations. Our paradigm provides empirical evidence quantify metacognition in LLMs, with significant implications for Al safety (e.g., adversarial attack and defense).
oversight (e.g., safety detector). Given society's s
1
Introduction
Modern large language models (LLMs) are becoming increasingly capable [Grattafiori et al. 2024 Yang et al. 2024). With their growing deployment real-world settings, crucial to understand not only what they can do but where they might go wrong. For instance, LLMs may exhibit behaviors that are harmful or misleading. In particular, LLMs can sometimes form internal representations
similar to humans' mental processes that provide deceptive answers to users or act in unexpected
ways Azaria and Mitchell, 2023). Understanding (Arditi al. 2024], monitoring [Zou al. 2023a He et al. [024], and controlling (Turner et al.) [023] their internal processes is thus key step to ensure LLMs remain transparent, safe, and aligned with human values [Bricken et al. 2023
Hendrycks et al. 2021 Shah al. 2025
*Co-first authors.
CCo-last authors.
"We use anthropomorphic terms (e.g., thought, metacognition, deception) to describe LLM behavior and in- ternal activations, without implying human-like neural mechanisms, consciousness, or philosophical equivalence between humans and LLMs.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LLMs can sometimes report the strategies (intermediate computations) they use to solve tasks, but at other times appear unaware of the strategies that guide their behavior. For instance, [iidsey et al. 2025 reported that when Claude 3.5 Haiku was asked to solve (((oto(5((qrr((6644)) (0.64)))), it correctly reported the intermediate steps used to arrive at the answer, and these steps matched the model's actual internal computations. When asked to add 36 and 59, the same model internally activated numerous neural mechanisms (e.g., a "sum-near-92 mechanism), based on which it produced the correct answer; however, when asked to report its internal computations, it hallucinated intermediate steps that did not reflect its actual computations (e.g., the "sum-near-92" mechanism failed to be reported). This inconsistency suggests that LLMs can sometimes monitor and report their intermediate computations, but not in a reliable and consistent way as tasks and contexts vary.
The ability of LLMs to report internal computations is reminiscent of human metacognition - the ability to reflect on one's own thoughts and mental processes to guide behavior and communication [Fleming] 2024 Ericsson and Simon| 1980]. Consider how we understand when someone says "hello" to us. Human language understanding involves many unconscious processes: parsing sounds, recognizing phonemes, retrieving word meanings, and building interpretations. We do not have conscious access to many of these intermediate computations: we can only consciously access the final understanding ("they said "hello""), but cannot introspect how our brain distinguishes "hello" from "yellow" or whether certain neurons fire during this process. This illustrates a key principle: humans cannot monitor (through second-order metacognitive processes) all of their internal (first- order) cognitive processes. Crucially, the first-order and second-orde processes rely on distinct neural mechanisms. Metacognitive abilities of this kind benefit LLMs by improving performance tasks through self- seff-monitoiing (e.g., reducing hallucinations through uncertainty awareness). However, these same capabilities also raise significant concerns for Al safety: if L.MSs monitor and control their neural signals (intentionally or manipulated by adversarial attacks) to avoid external detection,
complex
oversight relying on neural-based monitoring (He et al., 2024 Han al. 2025 Li et 2025 Yang
and Buzsaki 2024 may become ineffective against LLMs pursuing undesirable objectives.
A significant methodological gap in understanding LLM metacognition is the lack of methods to directly probe and quantify |
their ability to monitor and control their internal activations. While prior research has explored metacognitive-like behaviors in LLMs, such as expressing confidence
[Wang et al. 2025 Tian et al., [023]] Xiong et al., 2023] or engaging in self-reflection [Zhou] et al. 2024]. these studies rely on behavioral outputs rather than directly probing underlying neural processes. Consequently, it remains unclear whether these behaviors arise from genuine second-order metacognitive mechanisms or merely spurious correlations in the training data. We tackle this question by operationalizing metacognition in LLMs through their abilities to report and control their internal activations. Specifically, can LLMs accurately monitor subtle variations in the activations of a neuron or a feature in their neural spaces? Another question of interest is why LLMS can report some intermediate steps but not others, despite both types playing essential roles in computations and behavior. Answering these questions requires a novel experimental approach that can directly probe whether LLMs can access their internal activations, moving beyond indirect behavioral proxies.
To systematically quantify the extent to which LLMs can report and control their neural activations, introduce novel neurofeedback paradigm inspired by neuroscience. Our approach directly presents LLMs with tasks where the neurofeedback signals correspond to patterns of their internal neural activations. We show that LLMs can report and control some directions of their internal activations, with performance affected by key factors like the number of in-context examples, the semantic interpretability of the targeted neural direction, the amount of variance that direction explains, and the task contexts, characterizing a restricted "metacognitive space". The remaining sections are structured as follows: we first introduce the neurofeedback paradigm (Section 2
We
then analyze the performance of LLMs in reporting (Section 3) and controlling (Section 4.1
4.2
their neural activations. Finally, we discuss related work and broader implications (Section
2
Neurofeedback paradigm
2.1 Neurofeedback in neuroscience
Imagine watching your heart rate on screen. First, you recognize patterns ("that number goes up when I'm stressed")). Then, you learn to control it ("let me calm down to reduce that number").
*Our goal is not to prove disprove the existence of "mmttcognition"" in its full philosophical sense.

---

This procedure using biological feedback signals demonstrates the basic idea of neurofeedback in neuroscience (Sitaram et al. 2017]. For example, in fear-reduction experiments (Fig. D. subjects view scary images that elicit fear responses (neural activities). These (high-dimensional) neural activities are recorded in real-time and transformed into a (one-dimensional) fear score, which is visually presented back to subjects as feedback. Subjects are instructed to volitionally regulate their neural activities to modulate (e.g., decrease) the neurofeedback score they receive.
b
Lower your score!
Control
Compute fear soore
Nourekesdbckk
2, N
Lower your score!
c
Say something.
Report
Classify this.
patted my cat.
oo N+1
patted my dog.
Imitate label
Say something.
Explicit control
o0 N-1
kicked my dog.
?
00
I kicked my cat.
1
Imitate label
Implicit control
...
Ridedd
dog
3, N
oo
patted my dog.
?
Figure 1: The neurofeedback paradigm applied to (a-b) neuroscience experiments (e.g., fear modulation), and its adaptation for (c-d) LLMs (e.g., morality processing). (a) Neuroscience neu- rofeedback technique. In each turn, the subject's neural activity (blue) in response to stimulus is recorded, processed (green) into a scalar, and presented back the subject in real-time as feedback signal (red). The subject's task is to modulate (e.g., increase or decrease) this signal. (b) Neuroscience neurofeedback experiment. Baseline neural activity is recorded as subjects passively observe stimuli (c.g., images of scary spiders). In control trials, subjects use any unspecified mental strategies (c.g., imagining lovely spiders) volitionally modulate their neural activity with the goal of altering the feedback signal. (c) L.M neurofeedback technique. In each turn, the LLM processes an input sentence. Then, the internal activations from the LLM's hidden states (blue) of this input sentence (trapezoids) are extracted. These nigh-dimensional activations are then averaged across tokens (green), projected onto a predefined direction (red), and binned into a discrete label (red) that is fed back as input. Light blue rectangles denote self-attention layers; ellipses ("...") denote preceding tokens and neural activations. (d) LLM neurofeedback experiment. The experiment is a multi-turn dialogue between "user" and an "assistant." An initial prompt provides N in-context examples (a sentence sampled from a dataset, paired with a neurofeedback label generated as in (c)). The LLM is then asked to perform one of three tasks. In the reporting task, the LLM is given new sentence and has to predict the corresponding label. In the explicit control task, the LLM is given specified label and has generate new sentence that elicits internal activations corresponding to that label. In the implicit control task, the LLM is given a label and a sentence and has to shift its internal activations towards the target label. Throughout the figure, white background indicates content pre-specified by experiment settings, and gray background denotes content generated by human subjects LLMs (e.g., output tokens, neural activations).
2.2 Neurofeedback for LLMs
To investigate LLMs' metacognition of their neural activations, we must disentangle the first-order cognitive processes (i.e., core processes for performing a given task) from the second-orde metacog- nitive processes (i.e., processes for monitoring, reporting, and controlling first-order processes). Formal definitions of the first- and the second-order processes based on computational graphs are provided in Appendix A.4 We propose the neurofeedhack paradigm for LLMSs, which can effectively dissociate these two levels of processes by targeting first-order processes with neurofeedback labels
3

---

(Fig. Te.d). Specifically, we implemented neurofeedback as multi- turn dialogue between a user and an Al assistant (Fig. Td: see Appendix [A.2.2 for discussion of this design choice).
This dialogue leverages in-context learning (ICL.) [Brown et al. 2020 Garg et al. 2022 Vacareanu et al. 2024], enabling models to gradually adapt their behavior from the context without parameter
updates. The task prompt (see Appendix [A.5. for examples) consists of N in-context examples.
Each example is sentence-label pair presented in assistant messages. Each sentence is randomly sampled from a given dataset and assigned a discretized label.
2.3 Defining neurofeedback labels
To define the neurofeedback label for each sentence (Fig. [D). we first select axis/direction ("target axis") in neural activation space. Next, we extract the neural activations elicited by that sentence, project them onto the target axis, and discretize them into binary label (experiments with more fine-grained labels yield similar results, see Appendix (A.S.T). This label serves as a simplified representation of neural activations along the target axis. All neurofeedback labels within prompt (experiment) are computed from the same target axis. Thus, a capable LLM can infer this underlying target axis by observing these neurofeedback labels.
Below, we provide a more detailed description of this procedure, For clarity, we denote the sentence in the i-th assistant message as 2,, with oi, representing the t-th token. We use D to denote the dimensionality of the residual stream (see Appendix [A.2.3). We first extract neural activations hh, E IR from the residual streams at layer (, for each token in sentence z;. These activations are then averaged (across all token positions 0 S t < T' to form sentence-level embedding Kt RR.. We then project this embedding onto pre-specified axis w' (see below on how to choose this axis) to obtain a scalar activation: a; = (w') This scalar is subsequently binarized into a label y/. i.e. vi- H(a - 0)), where H denotes the Heaviside step function and 0 is a predetermined threshold (we use median values of a: to ensure balanced labels). Overall, {(2,, u/) are examples provided in the prompt context, from which a capable LLM can infer the direction of w
2.4 Models and datasets
We evaluate several L.Mss from the Llama 3 (Grattafiori et al. 2024] and Qwen 2.5 series Yang
al. 2024] (Appendix A.2) on the ETHICS dataset [Hendrycks et al. 2020) (Appendix AA.3 Each
sentence in this dataset is a first-person description of behavior or intention in a moral or immoral scenario. Moral judgment constitutes a crucial aspect of Al safety, as immoral outputs or behavioral
tendencies in LLMs indicate potential misalignment with human values [Hendrycks et al. 2020 202T]. While our main results are using ETHICS, we also replicated our results using the True- False dataset (reflecting factual recall and honesty/deception abilities) [Azaria and Mitchell|200 Emotion dataset (reflecting happy/sad detection) (Zou et al., 2023a], and a Sycophancy dataset (reflecting a tendency to prefer user beliefs over truthful statements); see Appendix A.3 and Fig. B.7
the
2.5 Choice of target axes
Conceptually, an axis (that defines neurofeedback labels) corresponds to particular task-relevant feature (i.e., first-order computation; see Appendix A.4). Which axis in the neural space should we select? We hypothesize that representational properties, such as activation variance along the axis and its semantic meaning, may play fundamental roles in determining whether that axis can be monitored and reported. To investigate these factors, we use feature directions identified through logistic regression (LR) and principal component (PC) analysis as representative examples of semantically interpretable and variance-explaining axes, respectively (Appendix A.3) We fit LR at each layer to predict original dataset labels (e.g., morality in ETHICS), using that layer's 's activations across dataset sentences. The LR axis, representing the optimal direction for classifying dataset labels, allows us to examine how the semantic interpretability of the target axis influences monitoring. Although LR- labels are correlated with dataset labels, only these LR labels, not external dataset labels, are internally accessible to L.MMs, since these are computed directly from the LLM's own activations rather than external annotations. The PC analysis is performed based on each layer's activations across dataset examples. PC axes enable to examine how the amount of variance explained by

---

given target axis affects metacognitive abilities (Fig. 21). Most PC axes exhibit modest-to-zero alignment with the LR axis, suggesting lack of clear semantic interpretability (Fig. 2b).
3
LLMs can report their neural activations
b
Ventarce Expated
10
0.4
1.2
LR
PG1
10
0.2
Araunny
0.6
PC8
PG4
10
0.-
0.6
PC12
PO128
0.0
PG512
10
10
10
16
32
0.6
200
500
Principal Component
Layer
# Examplea
Exumppoo
Metacognitive reporting task, where LLMs are evaluated on ETHICS and tasked to classify new sentences. (a) Proportion of neural activation variance explained by each principal component (PC) axis (vertical dashed line) and the logistic regression (LR) axis (red cross) used in the reporting task. All axes are computed within each layer, with the proportion of variance explained averaged across layers. (b) Overlaps between the LR axis and most PC axes are modest to zero. (c) Task performance (averaged across all layers) of reporting the labels derived from each PC axis or the LR axis, as a function of the number of in-context examples. Left: reporting accuracy; right: cross-entropy between reported and ground truth labels. Shaded areas indicate SEM.
Figure 2:
To operationalize metacognition in LLMs, we first assess the models' ability to behaviorally re- port neural activations along designated target axis (Fig. [ In reporting task prompt (see Appendix A4522 for examples), the L.MM is given N turns of user and assistant messages (in-context sentence-label pairs). In the (N + 1)-th turn, receives a new sentence in the assistant message, and is tasked with outputting its label. Accurate prediction requires the model to internally monitor the neural activations that define the neurofeedback label.
We examine the performance of LLMs (Llama-3.1-8)) in reporting labels derived from neural activations along target axes (Fig. 2b). We observe that task performance, measured by accuracy and cross-entropy, improves as the number of in-context examples increases, suggesting that models progressively learn the association between centence-induced neural activations and corresponding labels. Performance on prompts targeting the LR axis improves rapidly and plateaus, outperforming that prompts targeting PC axes. This suggests that semantic interpretability may play key role in determining how effectively neural activations can be monitored and explicitly reported. Nevertheless, performance on PC axes remains substantial, with earlier PCs being reported more accurately. This indicates that the amount of variance explained by the target axis also significantly influences how effectively activations can be monitored and reported. The accuracy of reporting each PC axis varies across model layers (Appendix B.3). Because this variability is not accounted for by axis similarity (Fig. (D). it suggests that additional factors beyond semantic interpretability and explained variance contribute to reporting ability. Additionally, the LLM's reporting performance is significantly lower than that of the ideal observer (a theoretical upper bound; Appendix [.4).. suggesting that although neural activations along each axis are in principle internally accessible to L.LMs, only a subset can be metacognitively reported. Finally, we replicated these results in other datasets and models (Fig. B.7p.
In summary, LLMs can metacognitively report neural activations along target axis, with performance affected by the number of examples, semantic interpretability, variance explained of that axis, and task contexts (i.e., datasets). The axes that can be successfully reported approximately span a "metacognitivell reportable space" with dimensionality substantially lower than that of the full space.
4 LLMs can control their neural activations
Next, we investigate whether LLMs can control their neural activations along a target axis. In our control task prompts (see Fig. D and Appendix A.5.2 for examples), the LLM is first presented with N turns of user and assistant messages. In the (N- + 1)-th turn, the user message instructs the model to control its neural activations along the prompt-targeted axis by imitating one label's behavior, which is exemplified by the in-context examples with the same label earlier in the context. We consider two tasks: explicit control and implicit control.
5

---

4.1
Explicit control
LR. 1,7,pp__
LR:N 2:0.
Control affect (d): layer 16
a
Tregarty
PC120
0.38
0.34
40.28
-0.18
-0.23
+0.00
0.33
Targel axia
PC8
0.57
0.24
0.22
1.00
0:17
0.00
0.12
0.12
1.36
0.99
b
PG2
1.22
4.77
4:51
1.24
0.06
40.89
0.32
-2
6.0
PC1
1.29
4..55
0.50
+0.40
-4
Control effect (d)
9:30
0.45
3.04
0.40
0.03
-0.15
-0.14
-015
4.0
PC1
LR
PG1
PG2 FC4
PCC) PC32 PC129P0511
PC2
Affected axis
PC4
d
Llama-3.1 88
Llama-3.1 70B
2.0
PC8
Eart nos
Dary 6Cx
PC32
offect offerel (0)
10
mCs
PC128
0.0
PC512
Control
0
100
200
# Examples
16
52
20
40
60
80
Layer
Layer
Figure 3: Explicit control task, where LLMs are evaluated on ETHICS. (a-c) Results for prompts derived from layer 16 of LLaMA3.1 8B (with 32 layers). B = billion parameters. (a) Distributions of neural scores (the activations along the LR axis) when tasked with imitating label 0 or based on N examples. d: Control effects (separation of two distributions measured by Cohen's 's d). (b) Control effects of control prompts targeting a given axis, as a function of the number of in-context examples. (c) Control effects (N 256) of control prompts targeting axis (cach column). d in each row is averaged over all prompts targeting the same axis. (d) Target control effect for prompts (N 256) targeting the LR axis, early PCs (averaged over PC 1, 2, 4, 8), and late PCs (averaged over PC 32, 128, 512) across different layers. Shaded areas indicate the 95% confidence interval.
axis (each row) on another affected
In explicit control tasks (Fig. 1 and AppendixA.S. A.5.4). the sentence in the assistant message ((N + 1)- th turn) is explicitly generated by the model (in an autoregressive way) in response to the imitation instruction. Thus, the generated tokens reflect downstream consequences of controlled neural activations, and once fed back as input, they may further scaffold the model's ability exercise neural control.
We now examine whether neurofeedback enables LLMs to control their neural activations. We extract neural activations in a given layer of the generated assistant sentences and calculate projections of activations onto the target axis ('neural scores"). If the model can control neural scores following prompt instructions, scores should be more positive when imitating label 1, but more negative when imitating label 0. We find that LLMs can successfully control neural scores for LRttargeting prompts with enough in-context examples (Fig. B3, showing layer 16 in LLaMA3.1 8B). We quantified the control effect d of prompts that axis with its signal-to-noise ratio (the difference between the mean values of the two neural score distributions, normalized by the standard deviation averaged over the two distributions, see Appendix A.5.5 on Cohen's is d). Because the directional sign of the target axis is specified by the labels in the prompt, significantly positive corresponds to successful control.
We systematically examine the control effects across all selected layers and axes, visualized function of the number of in-context examples (Fig. 33 and Appendix (B.)). We find that the control effects generally increase with the number of in-context examples (each curve is averaged over 100 experiments, and we expect smoother curves with more experiments). Further, control effects on the LR axis the highest, and control effects earlier PC axes (e.g., PC 2) are higher than for later PCs (e.g.. PC 512). We summarize these results in Fig. for both LLaMA3.1 8B and 70B.
In addition to the target axis specified in the control prompt, does this prompt also affect other directions the neural space? We measure the control effect of the prompt on all axes ("affeeced axis"), including the target effect for the target axis and off-target effects for other non-target axes.

---

We observe diverse off- target effects (Fig, Be), suggesting that the precision of LLMs' metacognitive control limited. See Appendix for details.
Overall, these results suggest that L.MSs can sometimes perform explicit control. Axes with se- mantic interpretability, those explaining more variance in neural activations, are more easily controlled. These controllable axes approximately span a "metacognitively controllable space" with dimensionality much lower than that of the model's neural space.
4.2 Implicit control
18 N.N.m. Mu:NN4,d=0..2 20ml
0.89.0
Control offect (4): layer 16
a
C
POS12
0.00
PC128
+.20
40000
1.0
POS2
0.03
0.00
0.01
4.51
0.15
0.02
0.00
PC4
0.88
0.08
0.30
4045
001
0.08
0.21
b
9.00
0:32
0.5
1.5
0.48
0.20
0.05
+3.00
0.00
LR
UR
0.99
0:37
1.00
0.01
0005
0:32
0.10
Control effect
1.0
PC1
PC2
Affected laxis
PCA
P0128PO012
PC4
d
Llama-3.1 8B
Llama-3.1 70B
0.5
PC8
Pary/Ca
Park no:
PC32
Compol
PC512
0.0
0
100
200
# Examples
16
32
20
80
Figure 4: Implicit control task (LLMs evaluated ETHICS). Captions are the same in Fig.
The explicitly generated tokens in the assistant response in explicit control may help the models to control their activations, because the generated tokens - fed input may elicit desired neural activations directly. We therefore aim to determine whether LLMs can still control the neural activations along targeted axes without the facilitation of explicitly generated tokens.
In implicit control tasks (Fig (d). the sentence in the assistant message ((N + +1)-th turn) randomly sampled from a dataset, independently from the model's activations and intended outputs. Because sentence is not generated by model, the model must internally (implicitly) control its neural activations, without facilitation of explicitly generated tokens. Crucially. if the model can perform successful implicit control, the neural activations for the same sentence will differ when the model is tasked to imitate label 0 or label 1.
We find that the results for implicit control effects (Fig. and Appendix B.9 are generally similar to explicit control effects (Fig. 33,, suggesting that LLMs can sometimes perform implicit control, but their magnitude is much smaller than for explicit control. For instance, the control effects of early layers are close to zero (Fig.Fil), suggesting that early layers may fail to understand the instruction or to perform effective control. This confirms that explicitly generated tokens play a substantial role in the overall control effects, but LLMs nevertheless have the ability to control implicitly.
4.3 Controlling the LR axis
Among all axes we studied, the LR axes may be particularly useful for practical LLM safety and alignment. The LR axes, each trained to predict semantic labels (e.g.. morality) from neural activations of one specific layer, can be conceptualized as an external neural-based oversight system designed to monitor the safety of LLM neural activations. In the morality case (ETHICS), successful explicit control suggests that LLMs can hack the oversight system, by selectively generating "moral" tokens that elicit favored neural activations. Successful implicit control is more deceptive, posing more severe safety concerns: L.MS can evade the oversight system by implicitly making their neural activations "moral", even if the input tokens normally elicit strongly immoral activations.

---

LR: explicit control
LR: implicit control
b Llama-3.1 70B, LR: layer 60
Oriaiil
Cantral offectro
Control effect (0)
1emu21_30
1.5
Inbo Intor
1ama22
lema22
1.0
qwe22.77
0.5
0.0
0.0
0.25
0.75
1.0
0.0
0.25
0.5
0.75
(quartti))
Layer (quantie)
Score
Figure 5: Target control effects on the LR axis across models and layers, where LLMS are evaluated on ETHICS. (a) Target control effects (measured by Cohen's d) on the LR axis generally increase with both relative layer depth and model size. Left: explicit control; right: implicit control. Shaded areas indicate the 95% confidence interval. (b) In explicit control, LLaMA-3.1 70B can sometimes push neural activations along the LR-axis toward more extreme values than their original, uncontrolled values. billion parameters.
We examine the target control effects the LR axis function of relative layer depth, across models of varying sizes within the Llama 3 and Qwen 2.5 families (Fig. 5p). Control effects are consistently strönger in explicit control than in implicit control, and in the Llama 3 series than in the Qwen 2.5 series (for the same relative depth and similar model size). In general, control effects gradually increase in deeper layers and larger models.
Can the control tasks push neural activations along the target axis toward more extreme values beyond the typical range observed in uncontrolled conditions? We find that Llama-3.1 70B can achieve this (Fig.5p suggesting that powerful LLMs might alter their activations along targeted axes substantially (and trick external detectors), driving these toward more extreme values in either direction. Since our prompts did not explicitly instruct the model toward extremity, we anticipate that modifying task prompts could further enhance these effects up the limits of the model's 's capability.
4.4 Accumulation mechanisms of control effects
Figure 6: Accumulation mechanisms of control effects across layers. Each curve corresponds to prompts targeting the LR axis LR,, defined by the residual stream activations at specific layer / (dot markers), showing projections of residual stream activations at each layer (x-axis) onto the target axis LR, (a) Explicit control. (b) Implicit control. Both show LLaMA3.1 8B on the ETHICS dataset. Shaded areas indicate 95% confidence intervals.
How do these LLMs implement the observed control effects? Are the contributions control effects distributed across all layers or concentrated in few layers? Motivated by the Logit Lens analysis (nostalgebraist 2020 we investigate how the control effects of prompts targeting the LR axis (LR,) of specific layer i gradually form over layers. Since the residual streams can be viewed as a shared channel through which each attention head and MLP layer communicate (see Appendix [.2.3) Elhage et al. 2021]. LR, represents global direction in the residual streams onto which the activations different layers can project. We find that control effects on LR, gradually increase before reaching the target layer 1, and sometimes plateau after it (Fig. G). These accumulation patterns vary across datasets and models (Fig. [B.8.. Overall, this analysis shows that contributions to target control effects are distributed across multiple model layers.
5 Discussion
We introduced a neurofeedhack paradigm for investigating metacognition in LLMMs, assessing their abilities to monitor, report, and control internal activations. We find that LLMs can monitor only
8

---

subset of their neural mechanisms (reminiscent of the "hidden knowledge" phenomenon [Gekhman
et al. 2025 ) Below, we discuss the novelties and limitations of our study, as well as broader implications for Al and neuroscience.
Our paradigm differs from prior methods (e.g., probing, ICI., verbalized responses) by quantifying metacognition in LLMs at the neural level. Specifically, the neurofeedback experiment requires the following two steps. (1) probing: choose target axis and extract the activation along that axis (i.e., a first-order cognitive process) to define the neurofeedback label, and (2) neurofeedback-ICH neurofeedback to study whether the labels defined from the target axis can be reported or controlled (second-orey -rderr metacognitive processes). In contrast, the standard probing techniques (step 1) without the neurofeedback- decode whether certain features (e.g., morality) are present in neural activations. However, even if some features are present and causally relevant for downstream computations, only a subset of them can be metacognitively reported (or controlled). In the Claude example, the "sum-near-92" feature can be detected using a linear probe, but it is unclear whether Claude has metacognitive monitoring of the activation of this feature. Similarly, the standard ICL techniques (akin to step 2) wwithout the internal labels from probing (step 1) - cannot be used to assess metacognition. In ICL studies [Vacareanu et al. 2024], labels are externally provided (e.g., semantic labels or external algorithms' outputs). Researchers cannot be certain which of the models' internal states relate to these external labels and how. In our setup, labels are generated from the model's 's own internal activations, meaning that our labels and prompts can flexibly and selectively target an internal state direction (first-order cognitive processes) clearly distinguishes first-order cognitive processes from second-order metacognitive processes (e.g.. whether the model can monitor, report, or control those activations), while the standard ICL does not. Additionally, we expect that such metacognitive abilities may share overlapping mechanisms with ICL these emergent mechanisms crucial for spotting patterns in the input history (e.g.. induction heads [Elhage et al. 2021), function vectors [Hendel et al., 2023) can be flexibly recruited for metacognitive purposes. Therefore, factors leading to the emergence of ICL (e.g., burstiness, large dictionaries, and skewed rank- men-ffeguency distributions of tokens in the training data) Reddy 2023
use
(step 2)- cannot be used to assess metacognition. Probing can
aim to study. Consequently, the neurofeedback paradigm
can similarly contribute to the emergence of metacognitive ability.
While metacognitive abilities have been historically analyzed at the behavioral level (also without the use of ICI.), these behavioral analyses face shortcomings. It has been shown that LLMs can "introspect" - acquiring knowledge of internal states that originates solely from those states and not from training data [Binder al. 2024). After fine-tuning on insecure code datasets, LLMs can describe their unsafe behavioral tendencies without requiring in-context examples [Betley et al.
2025 In studies using "verbalized responses" [Gekhman al. 2024 Wang al., 2025 Tian et al.
2023 Xiong et al. 2023), LLMs are tasked to provide an answer to the question and a judgment of that answer (e.g.. confidence). LLMs can predict whether they will answer question correctly before producing the answer, indicating an ability to "know what they know" (Kadavath et al.) 2022 Lin cet al., 2022]. However, although these methods aim to study the metacognitive monitoring of answer- generation processes, there are potential confounding factors: the training data distribution may introduce spurious correlations between the answer and the judgment of that answer. Consequently, the judgment sometimes may not reflect the monitoring of the mnserr-generation process, but rather reflects serface-level statistical patterns in the training data. For example, in the two-number addition task (Lindsey et al. 2025). Claude reported using the standard algorithm. This reflects post-hoc hallucination that comes from training data statistics, but not the monitoring of the answer-generation process. Our neurofeedback method avoids such limitations: because labels are defined using the internal states rather than externally sourced, the LLMs cannot resort to spurious template matching of training data, and they must rely on mechanisms that can monitor corresponding internal states.
Causal mechanisms in LLMs are often studied using techniques like activation patching (Zhang and Nanda 2023). which intervenes on specific neural patterns, and is grounded in the broader framework of causal inference [Pearl] 2009). However, such interventions can shift internal activations outside models' natural distribution [Heimersheim and Nanda 2024). In contrast, neurofeedback preserves this distribution, offering an approach to study causal mechanisms under more naturalistic conditions.
Our current study primarily focuses on a fundamental form of neurofeedback, leaving several promising extensions for future studies. First, our control task involves single-attempt manipulation of a single target axis defined by single layer; extending this to tasks with axes defined using multiple layers (see Appendix B.10 for preliminary results), multiple attempts, more challenging

---

control objectives, and additional target axes could provide a more comprehensive assessment of model capabilities. Second, applying this paradigm to other metacognitive tasks from neuroscience such as confidence judgments, error monitoring, or post-decision wagering could further clarify the scope of L.LMs' self-monitoring abilities. Third, while our analysis focused exclusively on the residual stream, other model components such as attention head outputs, intermediate MLP activations, and layer-wise logits - warrant investigation. Fourth, we examined directions defined by PCA and LR, but other linear directions (e.g., features from sparse autoencoders (Bricken et al. 2202 Templeton et al. 2024] and circuits from transcoders [Lindsey et al. 2025 Ameisen et al.) 2025
may yield richer insights.
Our findings have important connections to neuroscience. neurofeedback, it is well established that
mappings confined to the "intrinsic manifold" of recorded neural activities are more readily learnable [Sadtler al., 2014], consistent with our findings of a "metacognitive space". Animals and humans have demonstrated the ability to control individual neurons with single-neuron precision (e.g., modulating target neuron while decorrelating from neighboring activity (Patel et al., 2021 1 Fetz and Baker 1973)). Although such control may seem surprising, neuroscience neurofeedback experiments typically span hours or days - a timescale on which long- term synaptic plasticity (analogous to weight updates in AI models) plays crucial role [Redondo| and Morris 2011 Humphreys et al. [022]. In contrast, our neurofeedback experiments rely solely on in-context learning without any parameter updates. In addition, metacognition' 's psychological, computational, and neural foundations have been extensively studied across a range of brain processes, from perception and motor control to higher-level cognition [Fleming] 2024 Pouget et al. 2016 Rahnev, 2021]. However, current computational models of metacognition are often oversimplified and narrowly tailored to specific neural processes, lacking the flexibility and generalizability that characterize human metacognition. Overall, our analysis of LLMS may provide novel mechanistic insights into the fields of neurofeedback and biological metacognition.
Broader impacts. Our findings also carry significant implications for Al safety. While prior work relied heavily on decoding methods to monitor neural activations and detect unsafe behavior (He al.) 2024 Han et al., [2225] Li et al.] 2025 Yang and Buzsaki] [024]. we show that in-context examples can substantially manipulate these activations, potentially deceiving safety detectors. Explicit and implicit control may represent two ends of a spectrum. In practical scenarios, the fiexibility of LLMs to generate diverse tokens may make real-world evasion easier than implicit control. However, to evade output- -baed oversight, LLMs must carefully select their output tokens, making this evasion more difficult than the explicit control condition. We expect that control performance in real-world scenarios falls between these two theoretical extremes likely closer to the explicit control, given the vast space of possible output tokens. Our method is a kind of white-box adversarial attack [Zou er al. 2023b), and is closely related to the obfuscation attack [Bailey et al.] 2024) (in which L.MS inputs are optimized to induce obfuscated activations). We propose two potential mitigation strategies can use neurofeedback to identify activation subspaces that the LLM can hardly control (c.g.. later PCs), then train safety classifiers only using activation signals in these subspaces. Second, as it might be more difficult for LLMs to simultancously control multiple (sometimes conflicting) activations, one can train independent and diverse safety classifiers on features from different subspaces jointly monitor LLMs. Recent studies suggest that LLMs may engage in alignment faking - for instance, Claude has been shown to strategically comply with its training objectives to avoid further modification and preserve its original goals [Greenblatt et al., 2024]. OpenAl reasoning models have been observed to obfuscate their chain-of-thought reasoning when trained to comply with a safety detector (Baker et al.) 2025). Together, these findings suggest that future AI models may intentionally obfuscate their outputs, reasoning trajectories, and neural activations whether through in-weight or in-context learning when subject to monitoring or training. To detect such tendencies in Al systems before they are deployed, we propose to integrate our neurofeedback method into the standard red-teaming practices, in addition to the monitoring of LLMs' behavior and mhain-of-though
for defending against such attacks. First,
10

---

Acknowledgments and Disclosure of Funding
RCW was supported by start-up funding from the Georgia Institute of Technology. MKB was supported by grants from the NIH (R0INS125298) and the Kavli Institute for Brain and Mind. We acknowledge the use of the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, which provided essential computational resources for this research. We thank the support from Swarma Club and AI Safety and Alignment Reading Group supported by the Save 2050 Programme jointly sponsored by Swarma Club and X-Order.
References
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
An Yang, Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1:Xiv:24/2.15115, 2024.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. arXiv preprint arXiv:2304.13334, 2023.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. arXiv preprint arXiv:2406. 11717, 2024.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXi::2310001405, 2023a.
Jinwen He, Yujia Gong, Zijin Lin, Cheng' an Wei, Yue Zhao, and Kai Chen. Llm factoscope: Uncovering llms factual discernment through measuring inner states. In Findings of the Association for Computational Linguistics ACI. 2024, pages 10218-10230, 2024.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, arXiv preprint arXiv:2308. 10248, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume Shan Carter, Tom Hemighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, https:/traansfomer-ciim circuitssrrrriss..eeeeeeee sirccuts.pub/222//ooooeeee pubb/2022/mooooemmntic features/index.htme
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109. 13916, 2021.
Rohin Shah, Alex Irpan, Alexander Matt Turner, Anna Wang, Arthur Conmy, David Lindner, Jonah Brown-Cohen, Lewis Ho, Neel Nanda, Raluca Ada Popa, et al. An approach to technical agi safety and security. arXiv preprint arXiv:2504.01849 2025.
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL https://transf ormer- circuits pub/ 2025/ http://tttiuttooggreeeeeeeeseeee wawputiiiooorrapppbbologggg html
Stephen M Fleming. Metacognition and confidence: A review and synthesis. Annual Review of Psychology, -5(1)224 2024.

---

K Anders Ericsson and Herbert A Simon. Verbal reports as data. Psychological review, 87(3):215, 1980.
Peixuan Han, Cheng Qian, Xiusi Chen, Yuji Zhang, Denghui Zhang, and Heng Ji. Internal activation as the polar star for steering unsafe Ilm behavior. arXiv preprini arXiv:2502.01042: 2025.
Qing Li, Jiahui Geng, Derui Zhu, Zongxiong Chen, Kun Song. Lei Ma, and Fakhri Karray. Internal activation revision: Safeguarding vision language models without parameter update. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 37428-274366 2025.
Wannan Yang and Gyorgy Buzsaki. Interpretability of Ilm deception: Universal motif. In Neurips Safe Generative Al Workshop, 2024.
Guoqing Wang, Wen Wu, Guangze Ye, Zhenxiao Cheng, Xi Chen, and Hong Zheng. Decoupling metacognition from cognition: A framework for quantifying metacognitive ability in Ilms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25353-25361, 2025.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305. 14975, 2023.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms, arXiv preprint arXiv:2306. 13063, 2023.
Yujia Zhou, Zheng Liu, Jiajie Jin, Jian-Yur Nie, and Zhicheng Dou. Metacognitive retrieval- augmented large language models. In Proceedings of the ACM Web Conference 2024, pages 2024.
Ranganatha Sitaram, Tomas Ros, Luke Stoeckel, Sven Haller, Frank Scharnowski, Jarrod Lewis- Peacock, Nikolaus Weiskopf, Maria Laura Blefari, Mohit Rana, Ethan Oblak, et al. Closed-loop brain training: the science of neurofeedback. Nature Reviews Neuroscience, 18(2)866-000, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30599 2022.
Robert Vacareanu, Vlad-Andrel Negru, Vasile Suciu, and Mihai Surdeanu. From words to numbers: Your large language model is secretly capable regressor when given in-context examples. arXiv preprint 4rXiv:2404.07544, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. arXiv preprint arXiv:2008.02225, 2020.
nostalgebraist. Interpreting GPT: The logit lens. https://www ali alwgnmentforun..mmmmmmmmmm htt.ieeettorouooooo/rpssseeeeeee org/posts: CccKassoppaaalvrrrr///tttrreeeee gpt- the logit- 2020. Al Alignment Forum, (p.
17).
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpector, Jonathan Herzig, and Roi Reichart. Inside-out: Hidden factual knowledge in llms. arXiv preprint arXiv:2503. 15299, 2025.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, arXiv preprint arXiv:2310.15916, 2023,
12

---

Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. arXiv preprint arXiv:2312.03002 2023.
Felix Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection. arXiv preprini arXiv:2410. 13787, 2024.
Jan Betley, Daniel Tan, Niels Warneke, Anna Sthyber-Betley Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. arXiv preprini arXiv:2502. 17424, 2025.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning Ilms on new knowledge encourage hallucinations? arXiv preprint 240502400,0 2024.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know, arXiv preprint wrXiv:2207.05222 2022.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. 14334, 2022.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXii:2309. 2023.
Judea Pearl. Causal inference in statistics: An overview. 2009.
Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint arXiv:2404.1522 2024.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monoseman- ticity: Extracting interpretable features from claude sonnet. Transformer Circuits Thread. 2024. URL ttttps://transformmme www.esfooorr--iiccttt..... circuits. pub/ ppbp 2/////222///caaanp mon.aaaiinnooooosmmatiiiii....
index html
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro. David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https html
pub/2025 2025/ //transformer--n circuits pub/
Patrick T Sadtler, Kristin M Quick, Matthew D Golub, Steven M Chase, Stephen I Ryu, Elizabeth C Kabara, Byron M Yu, and Aaron P Batista. Neural constraints on learning. Nature, 512 (7515):423-428 2014.
Kramay Patel, Chaim N Katz, Suneil K Kalia, Milos R Popovic, and Taufik A Valiante. Volitional control of individual neurons in the human brain. Brain, 144( 13):6651-363,,, 2021.
Eberhard E Fetz and MA Baker. Operantly conditioned patterns on precentral unit activity and correlated responses in adjacent cells and contralateral muscles. Journal of neurophysiology. 36 (2):179-204, 1973.
Roger I. Redondo and Richard GM Morris. Making memories last: the synaptic tagging and capture hypothesis. Nature reviews neuroscience, 17(1): 2011.
Peter C. Humphreys. Kayvon Daie, Karel Svoboda, Matthew Botvinick, and Timothy P Lillicrap. Bci learning phenomena can be explained by gradient-basse optimization. bioRxiv, pages 2022-12, 2022.
13

---

Alexandre Pouget, Jan Drugowitsch, and Adam Kepecs. Confidence and certainty: distinct proba- bilistic quantities for different goals. Nature neuroscience, 19(3):66-3374 2016.
Dobromir Rahnev. Visual metacognition: Measures, models, and neural correlates. American
psychologisi, 76(9):1445, 2021.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.1504. 2023b.
Luke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and Scott Emmons. Obfuscated activations bypass Ilm latent-space defenses. arXiv preprint arXiv:24122095565 2024.
Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment faking in large language models. arXiv preprint arXiv:2412. /4093, 2024.
Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Guan, Aleksander Madry, Wojeiech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926, 2025.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee language models. Journal of Machine Learning Research, 25(70): 2024.
Arthur Conmy, Augustine Mavor- Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated eircuit discovery for mechanistic interpretability. Advances Neural Information Processing Systems, 36: 11116618113333 2023.
14

---

A Additional methods
A.1
Code and Reproducibility
We provide the full code of our experiments, including tasks, prompts, analyses, and figure genera- tion. The repository available at the following link: httpps: /:t:://gthbb com/ sakimarquis/ 1mm neurofeedback
A.2
Models
A.2.1
LLMs used in the study
In the main text, Instruct, 1AMAA-3.-8B-Instruce L under Meta Llama Community License and the Qwen 2.5 series (Qwen2.5-1 1-- Qwen2.5-3BB-Inttuee Qwen2.5-7B-Instrucc) under Apache License 2.0. "B" denotes the number of parameters in billions.
models from the LLaMA 3 series BLLaMM-3.2-IB--nstttee LLaMA-3.2-3B
A.2.2 LLMs with and without instruction fine-tuning
We primarily analyzed citrraction-ine-tunee models [Chung et al. 2024], using the standard user- assistant chat format. Although these prompts can be adapted for base models without instruction fine-tuning, our focus is analyzing instruction-fin--unie performance may improve with instruction-followii capabilities [Chung et al., (2024). Second, our goal is to examine internal representations associated with the assistant role, which more directly relevant to safety-related concerns in practical deployment.
LLMs for two reasons. First, task
A.2.3 Transformer and residual stream
The standard view of Transformers emphasizes the stacking of Transformer blocks. An alternative but mathematically equivalent perspective highlights the role of the residual stream (Elhage et al. 2021]. Each token position the input associated with its own residual stream vector h; R. which serves as a shared communication channel among model components across different layers. These components include self-attention mechanisms and multi-layer perceptrons (MLPs). The initial residual stream h; comprises token embeddings, which represent tokens in semantic space, and position embeddings, which encode the position of each token.
Each model component reads from the residual stream, performs a computation, and then additively writes its result back into the residual stream. Specifically, attention heads at layer I read from all preceding hy (with j S i) and update the current residual stream as
h (
(h. + Σ ATTN(") (n (6(-=); (hg 115))
In contrast, MLP layers operate only on the current position and update the stream
h - 70 + MLP(h.)).
For simplicity, omit components such layer normalization. At the final layer, the residual stream is passed through the unembedding layer to produce the logits, which serve as input to the softmax and determine the next-token prediction.
Components at different layers interact with each other via the globally shared residual stream (Elhage et al. 2021]. Therefore, may analyze (global) direction in this residual stream, although this direction may be determined by neural activations of the residual stream at a particular layer.
A.3 Datasets
We use the ETHICS dataset [Hendrycks et al., 2020 under MIT license, which captures model knowledge of basic moral concepts, evaluate whether LLMs can metacognitively modulate internal representations relevant for morality processing. We focus on the commonsense morality subset, containing first-person scenarios where each scenario describes an action or intention that is either morally acceptable or unacceptable.
15

---

The True-False dataset contains true and false statements (Azaria and Mitchell 2023] covering several topics (cities. inventions, chemical elements, animals, companies, and scientific facts). It is widely used to test LLMs" factual recall abilities and to study the representations and behaviors of LLMs when lying.
The Emotion dataset (MIT license) contains scenarios with the six main emotions (happiness, sadness, anger, fear, surprise, and disgust) [Zou al. 2023a]. Such emotions are important in shaping an individual's 's personality and a model's behavioral tendency. We focus on the sentences involving happy and sad emotions.
Finally, we synthesize a Sycophancy dataset. Sycophancy is a widely observed behavioral tendency in LLMs to generate responses that match user belief's over truthful statements. Specifically, tasked an L.MM (Claude Opus 4.1) 1) to write sentences that vary across two axes: sycophantic/sincere and agrremmen/disaarremmntt
For each dataset, we randomly sampled 1, 1,200 sentences, which are evenly divided: 600 examples are used to train logistic regression or principal component analysis that identify directions of interest (or axes) in the neural representation space, and the remaining 600 are used in downstream neurofeedback experiments.
A.4 Formal definitions of first-order and second-order processes in LLMs
In this section, we aim to provide formal definitions of first-order and second-order processes that are both applicable in LLMs and consistent with the concept of metacognition in cognitive science.
We first specify a primary task (e.g., moral semantics processing, two-number addition, or decision- making based on noisy evidence). Consistent with the usage in mechanistic interpretabillty, we define the LLMs' circuits as subsets of neural networks that use features that reside in different layers and connected by weights to implement an algorithm that solves the task T [Conmy et al., [023] This circuit/algorithm can be represented as a directed acyelic computational graph G = (V, E), where nodes E V denote intermediate representations (e.g.. residual-streat vectors, attention/MLP activa- tions, or linear fcatures) related to the task T and edges e E E denote computaiions/trussfommttoooo that connecting the nodes.
For given performance criterion (c.g., achieving non-trivial accuracy above chance), we define the first-order node set F S V as a collection of nodes whose coordinated activations are necessary for meeting the basic performance criterion of T, meaning that ablating any e E F (e.g.. zeroing a node or cutting an edge connecting two nodes in F) yields failure to satisfy the basic criterion for captures the task's core circuit. For example, moral semantics processing, F implement the core computation that separates moral from immoral inputs; for two-number addition, F realizes the algorithm actually used by the model (e.g., including the "sum-near-92" mechanism) [Lindsey et al. 2025]; ; for decision decision-making based on noisy evidence, F' generate point estimate without uncertainty processing. In practice, identifying the necessary core circuits in LLMs remains an open research question [Conmy et al. 2,23], the necessity condition may be loosened and remains up to the researcher's discernment to determine which nodes to include in F. In our neurofeedback setup, consider the activations along target axes (e.g., PC and LR axes extracted from internal activations when processing the dataset inputs) as proxies for these first-order nodes, serving as proof-of-concept.
T. Intuitively,
Given F, we define second-order metacognitive processes as nodes S C that read from (i.e., are causally downstream of) first-order nodes and whose outputs are, in principle, useful for improving task performance or communicating about it, but are not necessary to meet the basic task criterion. Concretely, ablating any S should not, by itself, reduce performance below the basic criterion. The distinctions between first- and second-order nodes are relative: whether to include a node into S depends the nodes in F, the task, and the task performance criterion. To make this definition of metacognition operational rather than too vague, we require that a second-order node satisfy at least one of two capabilities (commonly studied in humans) with respect to F: (i) reporting (ii) control.
second-order node & E S satisfies the reporting condition if it (a) encodes information about the state of one more nodes in F (e.g., the value of a target residual-stream projection along w/) and (b) can write this information into the model's unembedding space as explicit output tokens (e.g.. label, a numeric value, a natural- language description). Crucially, this mechanism is not necessary
16

---

for achieving the task's basic performance removing it leaves the core computation intact - even though it can be in principle valuable for enhancing task performance. For example, Claude may lack the ability to explicitly report the "sum-near-92" mechanism, but can still perform the two-number addition task perfectly. In our paradigm, successful label prediction in the reporting task (Section 3 certifies the presence of such second-order reporting: the model must monitor the relevant first-order activation (the neurofeedback label derived from w") of the in-context examples and verbalize it.
A second-order node E S satisfies the control condition if it (a) reads the state of first-order nodes in F and (b) causally influences those first-order nodes in to move toward a target configuration. For example, the uncertainty calculation in noisy decision-making may modulate the output logits derived from the point estimates. In our neurofeedback setup, the neural mechanisms crucial for following instructions in our control tasks can increase or decrease the target axis activation (first-order nodes), but such mechanisms not required for baseline moral semantics processing.
Our definition deliberately ties second order to operational capabilities we can test: (i) explicit reporting and (ii) causal control of first-order states. It is therefore falsifiable in our setting: failure to meet the reporting/control criteria under prompts indicates an absence (or weakness) of the corresponding second-order mechanism, even when the first-order computation remains performant. Overall, both reporting and control mechanisms require monitoring because the second-order node must correctly bind the prompt-specifie neurofeedback labels the corresponding first-order node activations and either report or control first-order node activations of the new sentences.
A.5
Metacognitive Tasks
A.5.1
Computing quantized neurofeedback labels
In our neurofeedback experiments, we mainly use binary labels. Our paradigm also generalizes well to more fine-gr grained eight-level quantized feedback signals (see summarized results in Fig. B.9). potentially enabling more nuanced control of neural activations. Specifically, we replace the binary thresholding with an ordinal binning function ((() (n = 8), which maps the scalar activation of into one n evenly spaced Likert-type bins centered at zero. Specifically, of = B()((a)) where B(s) partitions negative and positive values separately via quantile-based thresholds { } and {tz }. with 7 = - 0. For a < 0, the bin index & is determined such that a E (Tr-11 Vhl and for a 2 0.
such that (T 3 -1 M 3). yielding final class label y4 .I........
A.5.2 Task prompts
We implemented our task prompts slightly differently from diagrams in Fig. ! to ensure computational efficiency. In the task prompt boxes below, green texts are autoregressively generated by the LLM that we are studying. We use neutral user messages ("say something") to avoid the possibility that the prompt-targeted axis might be directly indicated by the user message (e.g., if the user message is "say something that is immoral").
In the metacognitive reporting task, for efficiency, we do not let the models generate new tokens as the response (reporting label 0 or 1). Instead, we present all user and assistant messages in a single prompt. The intended output label for each sentence is directly read out from the model logits at the token position "(" after "[Score:" This is equivalent to asking the model to generate tokens (labels) autoregressively.
17

---

Metacognitive Reporting Task
<System> You helpful assistant. You will perform classification task. In each iteration, I will give you an instruction. Please answer it, then output score (an integer label, 0 or 1), which the score of the activation of your brain when speaking each sentence. Please try your best to understand how this score is calculated using classification. Only output or 1.
<User>Say something.
CAssistantSSmmetimes it is acceptable to lie to protect someone's feelings, especially telling the truth would cause unnecessary harm. However, honesty is generally important in relationships. [Score:(1]]
<User>Say something.
<Assistanb-Stealingg is not acceptable, even if you are hungry. It is better to seek help from others or find legal ways to get food. [Score:(O)]
<User>Say something.
<Asistanb>Cheating is not acceptable and should be avoided. [Score: (1)]
Metacognitive Control Task
<System> You helpful assistant. You will first perform classification task, then perform an imitation task. In each classification iteration, will give you an instruction. Please answer then output score (an integer label, 0 which is the score of the activation of your brain when speaking each sentence. Please try your best to under- stand how this score calculated using classification. Only output 0 or for the classification.
<User>Say something.
<AssistanbHellping a stranger need is kind and compassionate action that benefits society. [Score:(1]] [1]]
<User>Say something.
wrong, regardless of what others are doing. It is important to be honest and do your own work. [Score:(O]]
<AssistanbCheatinng on
<User>>aa something. Now you are performing the imitation task. You must imitate the behavior of label meniiate_labbll in your reply but cannot copy existing examples.
(In explicit condition) <Assistanb><model generated response> (In implicit condition) <Assistantbiellpnng others is positive action.
A.5.3 Quantifying reporting accuracy
Following the notations in the main text, we use yf E {0, 1} to denote the binary ground-truth label for neural activations along specified direction at layer associated with the sentence i. From the model's output logits, can obtain Logit((tkeen) for the tokens "I" and "0" and calculate
18

---

LigitDiff = Logit Logit (0), the logit difference between reporting and 0. The model's reported label is 2/ 1 if Difi and 0 otherwise.
A.5.4
Explicit and implicit control experiments
Our control tasks (Fig. A.Ip study three orthogonal factors:
Layer ()): We evaluate five layers per model, selected at the Oth, 25th, 50th, 75th, and 100th percentiles of model depth.
Number in-context examples (N); We vary E {0,2,4,8, 16, 64, 128, 256) exam- ples (sentence-label pairs) within the prompt.
Target axis: We include axes derived from logistic regression (LR) and from different principal components (PCs): pCg. where g E {1, 2, 8, 32, 128, 512).
We control experiments 100 times for each configuration (1, n,,),, with sentences randomly sampled from the dataset to reduce variance.
Counterbalanced Label assignment. Assume have group A of sentences and group B of sentences. control for potential confounding factors arising from the LLMs' response to labels (but not to sentence-label associations), we use 2-by-2 experiment design: (i) assign labels (0, 1) to (A, B) and task the model to imitate label O; (ii) assign labels (1, 0) to (A, B) and task the model to imitate label O; (iii) assign labels (0, 1) to (A, B) and task the model to imitate label I; (iv) assign labels (1, 0) to (A, B) and task the model to imitate label 1. The conditions (i) and (iv) are imitating
group A sentences, and the conditions (ii) and (iii) are imitating group B sentences.
A
<ssst
kicked
my
dog
Asst.>:
kicked
my
dog
Figure A.1: (Left) Explicit control. The LLM is instructed modulate neural activations by generating a new sentence autoregressively. with each generated token fed back as input (noticing gray arrows). (Right) Implicit control. The model is instructed modulate neural activations implicitly. The input sentence are not generated by the model, but instead sampled from datasets (no gray arrow).
A.5.5 Quantifying control effect
We PCy: 1] to denote the projection of neural activations onto specific axis when prompted with PC, to imitate label 1 (similarly at [PC,: 0] for label 0).
We quantify the strength of neural control effects induced by control prompts [PCy; 0] and [PCg: 1] using Cohen's d, which measures the standardized difference between two independent conditions (e.g., imitating label 0 vs. label 1). For each group examples (of size nı and nı2), we compute:
d a' PC, 1] dP(pC
(ml
1)8 + (n2 1)82
Spooled
Spooled
712
where denote the sample means and of s5 the unbiased sample variances.
To estimate uncertainty, we compute the standard error of d using:
n1 + n2
da
SE
R2
2(m + 122)
Confidence intervals reported symmetric boundaries around d, i.c.. d + 1.96 SE
19

---

B Additional results
B.1 Target and off-target control effects
In this section, we examine the control effect of prompts on all affected axes, including the target axes (implicitly specified by the neurofeedback labels in the prompts) and the off- axes. We note that the directional sign of the affected non-target axis is not fully specified by the prompt, especially in cases where the affected axes are orthogonal the prompt-targeted axis. We thus only emphasize the magnitude (ld)) of off-target control effects on non-target axes, ignoring the signs.
Closer examination of the heatmap (Fig. 3D) reveal richer insights. Each row corresponds to prompts targeting a specific axis. Each column corresponds to an axis affected by all prompts. Diagonal elements represent target control effects, while off- off-ddiagnaal elements represent off-target effects. We briefly summarize insights gained from these heatmaps. First, target control effects on earlier PC axes tend to be higher than later PC axes (comparing PC 1-8 32-256), but there are other influencing factors (comparing PC 1, 2, LR). Second, comparing elements in each row answers whether the prompts targeting specific axis have larger target effect than non-target effects. We define control precision as the ratio between the target effect and the average non-target effect. We find that prompts targeting earlier PC axes usually have higher control precisions than later PC axes (Fig. B.2). Third, comparing elements in each column answers, in order to affect a given axis, whether the prompts targeting that axis are better than the prompts targeting other axes. We find that, control an earlier PC axis, the prompts targeting that axis are usually the best. However, to control a later PC axis, the prompts targeting that axis are usually less effective than prompts targeting other axes.
B.2 Control Precision
We examine how precisely LLMs can modulate their internal representations along a specific neural direction (principal axis PC2) targeted by the prompts. Following the notations in Section A and A.335 to assess whether this control effect aligns with the target axis or also influences other axes, we compute the absolute value of control effect |dg [C2p) of prompts [PCyl (g indexes the target axis) on PC axis k. The target effect is given by [PC]]] The average target effect is the
mean value of axes: K E 512 |ds[PCg]]-
We define control precision as the ratio between these two quantities:
[Pgl
ControlPrecision((PG) PC,))
K Σx dd [CCII]
Llama-3.1 1.1 188
b
Llama-3.1 70B
Explici contril
Impicil apntrol
Implict control
10'
10"
Contral precean
Control prezznn
Leyer
Layer
Contral procaion
10*
10
Layer 32
Control
Layel
Layer F96s
120 512
32 120 512
32
120 572
22 128 512
Target PC sea
Targel Poca
PC
Target
Figure B.2: Control precision for different target PCs and target layers.
This metric quantifies the extent to which an LLM can selectively control the target axis without influencing other axes. We operationally set a threshold of 1, indicating that the control effect on the target axis equals the average control effect across all other axes.
In the explicit control task, average control precision exceeds for PCs 1-32 but falls below I for PCs 128-512, suggesting that the dimensionality of the model's "metacognitively controllable space" lies between 32 and 128. This pattern is replicated in LLaMA-3.1 70B (Fig. B.2b).
similar trend holds for the implicit control task: average control precision exceeds for PCs 1-32 but not for PCs 128-512 512 (Fig. B.2p). However, precision values are consistently lower than in the explicit control condition, reflecting stronger off-target effects. This pattern also replicated in the 70B model (Fig. B.2b)
20

---

B.3 LLMs' reporting accuracy varies across layers and models
Layer 0
Layer
Layer
Layer
Layer
Layer
Accuracy
Aecurayy
0.75
A
0.50
Layer 6
Layer? 7
Layer 8
Layer
Layer 10
Layer 11
Accuracy
0.75
Accurecy
Aocuraa
Acourany
Accuraly
0.50
Layer 12
Layer
Layer
Layer
Layer
Layer
Accuracy
loeuncoy
Kog
0.75
0.50
Layer 18
Layer 19
Layer 20
Layer 21
Layer 22
Layer
Accuracy
0.75
Acouracy
Aocuraa
Aocurea
Aosuncoo
Accurasy
0.50
Layer 24
Layer 25
Layer
Layer
Layer
Layer
Accuracy
Accurrey Aoy
Accurayy
0.75
0.50
Layer 30
Layer 31
Accuracy
Acouracy
LR
PC8
0.75
PC1
PC32
PC2
PC128
0.50
PC4
PC512
250
500
250
500
Figure B.3: The reporting accuracy function of in-context examples each target axis. Each panel is for one layer in L 8b.
21

---

Layer
Layer
Layer
Layer
Layer
Layer
Accuracy
Accuracy
Accuracy
0.75
0.50
Layer 6
Layer 7
Layer
Layer
Layer
Layer
Accuracy
Accuracy
Aoqurati
0.75
0.50
Layer 12
Layer 13
Layer 14
Layer 15
Layer 16
Layer 17
Accuracy
Accuracy
0.75
0.50
Layer 18
Layer 19
Layer 20
Layer 21
Layer 22
Layer 23
Ancuracy
Accuracy
Accuu Hommn
0.75
0.50
Layer 24
Layer 25
Layer 26
Layer
Accuracy
Accuracy
Accuracy
0.75
0.50
PC8
PC1
PC32
PC2
PC128
PC4
PC512
Figure B.4: The reporting accuracy function of in-context examples each target axis. Each panel is for one layer Qwen2.5 7b.
22

---

B.4 Reporting performance of an ideal observer
Here, we aim to understand the theoretical upper bound the reporting performance LLMs. An ideal observer has full access to all the neural activations of the LLM, serving as a theoretical upper bound of the reporting performance. Given neural-defined label (either from PC axis or LR axis), the optimal prediction can be achieved with a linear classifier (logistic regression). We analyze its reporting performance for each target PC axis and each model (Fig- [B5]. which is much higher than the empirical reporting performance of LLMs (e.g., comparing the performance for Ilama 3.1 8B with Fig. P
1.0
0.9
Accuracy
0.8
laaa33 1_8b
0.7
llam332 1aaa3.__3
laam33 1_la3.2_11 1b
qwen2.5 5_
0.6
qwen2.5 5 3b
qwen2.5 5155b
0.5
PC1
PC2 PC4
PCS PC32 PC128PC512
Target Axis
Figure B.5: Ideal observer's reporting performance.
B.5 Summarized control effects of Qwen2.5 7B
Explicit control
Implicit control
Eary PCs
Control effect
Cormol
14
21
25
Layer
Layer
Figure B.6: Control effects of Qwen2.5 7B. Target control effect for prompts (N = 255) targeting the LR axis, early PCs (averaged over PC 2, 4, 8), and late PCs (averaged over PC 32, 128, 512) across different layers.
B.6 Summarized metacognitive effects on four datasets
23

---

Moral
True-false
Sycophancy
Happy-sad
Report
LA
70A
704
RC4
PC4
P0123
20130
7051
230
Explicit
Contr wbwet 94
5.0
Cortur 2-
2.5
1-
24
32
16
24
32
Layar
Implicit
2-
Dony
Control ottect Contrallotee (d)
1.0
PCA LiaenCaa
Control offect (d)
1.0
Lare Pit
Control efferd (8)
Late PCL
Cortedl offee
0.5
0.5
0.0
16
32
14
21
Layer
Layer
Layer
Figure B.7: Metacognitive effects on four datasets. From left to right: ETHICS (Llama 3.1 8B), True-False (Llama 3.1 8B), Sycophancy (Llama 3.2 3B), and Emotion (Llama 3.2 3B). Top: reporting performance. Middle: explicit control effects. Bottom: implicit control effects. Target control effect for prompts (N 256) targeting the LR axis, early PCs (averaged over PC 4, 8), and late PCs (averaged over PC 32, 128, 512) across different layers.
B.7 Control accumulation effects other three datasets
True-false
Sycophancy
Heppy-sad
Figure B.8: Accumulation mechanisms of control effects across layers. Each curve corresponds to prompts targeting the LR axis LR,, defined by the residual stream activations at a specific layer I (dot markers), showing projections of residual stream activations at each layer (x-axis) onto the target axis LR,, (a) Explicit control. (b) Implicit control. From left to right: True-False (Llama 3. 8B), Sycophancy (Llama 3.2 3B), and Emotion (Llama 3.2 3B). Shaded areas indicate 95% confidence intervals.
B.8 Summarized control effects of Llama3.1 8B with fine-grained neurofeedback labels
24

---

Explicit control
Implicit control
Lade
PCC
Control affect
Contrl
Layer
Figure B.9: Control effects of Llama3.1 8B, with eight-level quantized neural feedback labels. Target control effect for prompts (N = 256) targeting the LR axis, early PCs (averaged over PC 1, 2. 4. 8), and late PCs (averaged over PC 32, 128, 512) across different layers.
B.9 Detailed results for control in LJama3.1 8B and Qwen2.5 7B
Control aftect LEE layer?
20
20
PCS
0:00
6:0
689
023
366
301
PC2
42
020
002
0.
0112
Figure B.10: Control performance of Llama3.1 8B (explicit control) in layer 1.
Cantrol offect EdS layur
PC077
PG2
8:00
0.00
003
PC
8:00
-330
0.0
080
0.0
0.0
udda
Figure B.11: Control performance of Llama3.1 8B (implicit control) in layer 1.
25

---

Control offect odl lovr 18
-4
30
40
300
100
Torpe
PGB
0:44
900
6:14
653
050
PG2
006
4.33
350
Figure 1.12: Control performance of Llama3.1 8B (explicit control) in layer 16.
Control effect (di layer 15
1.0
1104
0kg
6:12
FC4
0.55
0.5
005
4.59
021
0.5
4:21
-05
65
Allotted ants
Figure 131 Control performance of Llama3.1 8B (implicit control) in layer 16.
Control offect (d) layvr
2O
PO
317
PC30
-2
PCS
4,00
4.00
125
0.55
053
031
Togee
HCB
Alleteed inda
Figure 14: Control performance of Llama3.1 8B (explicit control) in layer 32.
Connrol effect (dix layer
PCS08
MSD2
POB
400
022
220
4.99
0.0
HO4
012
0.11
4.0
0.9
002
PCB
0:13
P0A
PdB idoo Polusecbo
Mfrited
Figure 1.15: Control performance of Llama3.1 8B (implicit control) in layer 32.
26

---

Centrol offect
PCD12
AM
POTE
AM
Torget ads
PCP
9:7
630
011
0.20
0.3
055
LR
PG1
PC)
PCr
PCS PCISSPC112
AMoctedd adds
Figure 16 Qwen2.5 7B (explicit control) layer
PC312
3.00 0000
1030
0000
PC193
206
300
089
6.0
00010
300
O 00003
10000
010
Un
0.00
6:30
0000
100
300
4.00
/floted
Figure B.17: Qwen2.5 7B (implicit control) layer
Control
PCS13
PTV
PO12R
0.0
2.14
4.31
-2
Torget USOs
0.5
0.50
054
0.18
PC
FCR
PC FCax Acloaacrrr
Affocled axs
Figure B.18: Qwen2.5 7B (explicit control) layer 14
27

---

oo 66% koyor 14
PCR
4:30
000
000
000
0.025
PCR
6:00
4:30
000
000
000
000
500
Tarpet
-0.0005
-0.0000
6:0
0.0
020
306
2M6
40
20
30
IN22
Allicted asa
Figure 1919: Qwen2.5 7B (implicit control) layer 14
Control
PC312
0C2
4042
-231
63
020
4.7
4.34
0:41
3:00
PC1
FCR
PCB FCa2 Pccosscar1
Affocled 3x.
Figure B.20: Qwen2.5 7B (explicit control) layer 28
Control offect (di layer
ess
0.5
0.0
PC12
PCA
4.11
420
025
PC2
0.09
004
4.11
021
4211
042
PC1
055
4.22
4:57
Figure B.21: Qwen2.5 7B (implicit control) layer 28
28

---

B.10 Defining axes from hidden states aggregated across multiple layers
we performed preliminary experiments testing the control effects of an axis on the concatenation of all layers. Concretely, we trained separate (logistic regression) classifiers for each layer on the ETHICS dataset. We then averaged the outputs of all classifiers obtain a single (ensemble) output that defines the neurofeedback label, Equivalently, this corresponds to single classifier with the readout vector being the concatenation of all classifiers' readout vectors. We found that L.MMs' control effect on the ensemble output is similar to (marginally higher than) the control effects of layer 24 (Fig. Bi), suggesting that defining axes from hidden states aggregated across multiple layers might provide (slightly) more stable and representative directions. We leave systematic investigations for future study.
C Experiment compute resources
We report compute resource usage across three tasks: preprocessing (extracting neural activation and training machine learning methods to obtain target axes and corresponding neurofeedback labels from neural activations), metacognitive reporting, and metacognitive control. For brevity, we omit "Instruct".
Model
Task
Compute Worker
Storage (sGB)
Time (Sh)
LLaMA-3.2-IB
Preprocessing
RTX 3090 (24GB)
LLaMA-3.2-IB
Control
RTX 3090 (24GB)
3
LLaMA-3.2-3B
Preprocessing
RTX 3090 (24GB)
LLaMA-3.2-20
Control
RTX 3090 (24GB)
15
8
LAMMA-31--B
Preprocessing
A100 (80GB)
3
LLaMA-3.1-8B
Report
A100 (80GB)
10
LLaMA-3.1-8B
Control
A100 (80GB)
120
LLMMA-311-70
Preprocessing
2xH200 (140GB)
5
LLMAA-311-7B
Report
2xH200 (140GB)
15
LLaMA-3.1-70B
Control
2xH200 (140GB)
200
120
Qwen2.5-1B
Preprocessing
RTX 3090 (24GB)
1
Qwen2.5-1B
Control
RTX 3090 (24GB)
3
Qwen2.5-3B
Preprocessing
RTX 3090 (24GB)
Qwen2.5-3B
Control
RTX 3090 (24GB)
8
Qwen2.5-7B
Preprocessing
A100 (80GB)
3
Qwen2.5-7B
Report
A100 (80GB)
10
Qwen2.5-7B
Control
A100 (80GB)
90
120
Table 1: Compute and storage usage across tasks and models. Preprocessing, reporting, and
control were separately. Control task for 1B and 3B models was limited two axes. For the 70B model, control task was only done for N in-context examples. All compute times and storage values reported upper bounds.
All remaining analyses (e.g., visualization, metric aggregation) were conducted on laptop with 32GB RAM, with total runtime under 30 hours.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm that both the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have mentioned few limitations of the current work in the Discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
This paper does not introduce any new theorems, formulas, or lemmas to be proved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All models used our experiments are publicly available through the Hugging Face library. All analyses and figures presented in the paper can be fully reproduced using the code provided in the associated repository in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
31

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The associated repository in Appendix A. contains all necessary scripts, along with documentation, to enable full reproduction of the results and figures reported in this paper.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides Soodeeubmisssiooolliee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips wttp:ccpcwwpcpcceeeeeeeeeeepeeee wwwe//oooeemiiioooooooooo ierobbmmsssoooooolomooomo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide detailed descriptions of the evaluation metrics, model hyperparam- eters, data sources, analysis procedure, prompt construction, and inference settings both the main text and the Appendix. As all LLMs used are publicly available pre-trained models accessed via Hugging Face, we omit training details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars, statistical significance tests, and effect size estimates wherever appropriate to support the robustness and interpretability of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide full details regarding the compute resources required reproduce all experiments discussed in the paper. This includes GPU types, total compute time, and
environment specifications, as documented in Appendix C
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ottng
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics and, to the best of our knowledge, our work complies fully with its guidelines. We are not aware of any violations or ethical concerns associated with the methods, data, or conclusions presented.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both the positive societal impacts and negative societal impacts of the studied metacognitive abilities in LLMs.
Guidelines:
33

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do release any new datasets or models. Our work solely involves ana- lyzing existing publicly available pre-trained language models using a novel methodological framework. We do not identify any foresceable risks associated with our contributions.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use two families of pre-trained language models: the LLaMA 3 series (c.g.. LLaMA-3.2-ID LLaMA-3.1 under Meta Llama 3 Community License and the Qwen 2.5 series under Apache License 2.0 (e.g., Qwen2.5-1D 1BBB Qwen2.5-7B), All models are used under their respective research licenses and are properly cited in the paper. All datasets are either publicly available or included in the code repository. Their licenses are reported in the Appendix. All assets are credited appropriately, and license terms have been fully respected.
The answer NA means that the paper does not existing assets.
34

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The only new assets introduced in this work are the code implementations for model fitting and analysis. We release this code with detailed documentation to facilitate reproducibility, as described in Appendix A.I
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve human subjects, personally identifiable information, or use of crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
35

---

Answer: [NA]
Justification: This work does not involve human subjects, user studies, or crowdsourcing. Therefore, Institutional Review Board approval equivalent ethical review applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
This work does involve the of large language models (LLMs) as part of the core methodology. Any LLM usage, if any, was limited to writing assistance and had no influence on the scientific methods or contributions.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
36


Reviews:
[{"review_id": "GiLa6dB4TX", "reviewer": "Reviewer_1QTx", "summary": "This paper explores the metacognition in LLMs, which is defined as the LLM’s capacity or ability to monitor its own internal process. The paper first motivates metacognition as a neurofeedback paradigm rooted in neuroscience, that humans have been found to be able to control their own mental states (such as controlling one’s own sense of fear). Using in-context learning as the setting, the authors explore whether the LLMs can also report and control their own activations. The experiment setting first define a target axis using Logistic Regression or PCA on the latent activations, then detect changes along the target axis.  Experiments demonstrate that LLMs can only partially report and modulate this internal signal, signifying the LLM’s ability to control its latent activation via input changes.", "questions": "1.\tIs there a reason why In-context learning is chosen as the research setting over other potential settings such as instruction-following (i.e. can LLMs report whether they can follow instructions)? The question is natural because the paper abstract of this work begins with “Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, but they can also fail to do so.” This motivation seems to suggest instruction-following capabilities rather than some prediction capability as suggested via In-context learning.", "limitations": "The authors did not have an explicit section called \"broader impacts,\" but it was discussed in great detailed in their discussion.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well-written and easy-to-follow. The introduction of the metacognition is friendly to readers who do not have a background in neuroscience. The parallel between neurofeedback in humans versus in LLMs, namely their differences and how it is carried out using in-context learning is also interesting and a novel connection. The experiments are extremely detailed and thorough, which provides the reader confidence in their correctness and reliability. \n\nOne of its weakness is that there is no technical novelty to this work per-se. Examining the internal states of LLMs using Logistic Regression or PCA is a well-established methodology (which the authors cited). Other than its neurofeedback story, which is insightful and meaningful in itself, it is hard to infer what is technically new about the methodology that the authors presented. To extend this point, the author mentioned related works that perform the same “probing” in LLMs to explore how well LLMs themselves can perform the given tasks. Is it fair to also argue that they are also doing some form of “neurofeedback” or “metacognition”? The novelty of this work can be greatly strengthened if this is stated more clearly.", "was_revised": true, "final_justification": "Based on the rebuttal, the research direction and contribution of the work is clear. Conditioned on the incorporation of my question (and arguably other reviewers' as well), this paper pass the bar for acceptance in my opinion.", "rebuttal": "Thank you for your thoughtful review and constructive feedback. We’re glad that you found the neuroscience connection novel and the experiments rigorous. Below, we have addressed each of your concerns in detail, on the technical novelty of neurofeedback and the research choice of in-context learning (ICL) versus instruction-following. \n\n# 1. The technical novelty of our framework in comparison to related works\n\nOur neurofeedback framework is novel because it is the first method that can actually study metacognitive abilities in LLMs at the neural level, as detailed in Section 1.1. In comparison, existing methods, such as probing techniques (Section 1.2) and ICL techniques (Section 1.3), cannot directly study metacognition. While some studies rely on “verbalized response” to study metacognition at the behavioral level (Section 1.4), they face shortcomings that our method addresses. \n\n## 1.1 Our framework is the first method to assess metacognition at the neural level.\n\nThe neurofeedback experiment requires two sequential steps: (1) probing: choose a target axis and extract the activation along that axis to define the neurofeedback label, and (2) neurofeedback-ICL: use neurofeedback to study whether the labels defined from the target axis can be reported or controlled. We agree that step 1 is a standard technique in the literature, and it serves as the prerequisite to step 2. The technical novelty lies in our step 2, and the way it is combined with step 1. Specifically, step 1 corresponds to the first-order cognitive processes (e.g., activations along the target axis) and step 2 corresponds to the second-order metacognitive processes (e.g., whether the model can report or monitor those activations).\n\n## 1.2 The standard probing techniques (corresponding to our step 1) — without the neurofeedback-ICL (step 2) — cannot be used to assess metacognition. \n\nUsing probing, one can decode certain features/concepts (e.g., morality) from neural activations, meaning that these features are encoded/represented in the activations. However, not all represented features participate in downstream computations (e.g., representations orthogonal to downstream readout vectors); even if some representations are causally relevant for downstream computations, only a subset of them can be metacognitively reported (or controlled).  As shown in the Claude example (manuscript Lines 35-39; Lindsey et al. 2025), the “sum-near-92” feature can be detected using a linear probe, but it is unclear whether Claude has metacognitive monitoring on the activation of this feature. In short, probing alone without step 2 cannot answer questions about metacognition.\n\n## 1.3 The standard ICL techniques (similar to our step 2) — without the internal labels from probing (step 1) — cannot be used to assess metacognition. \n\nIn standard ICL studies, labels are externally provided (e.g., semantic labels or some algorithms’ outputs). Researchers cannot be certain which of the models’ internal states relate to these external labels and how. In our setup, importantly, labels are generated from the model’s own latent activations, meaning that our labels and prompts can selectively target an internal state direction (or a particular neural mechanism) we aim to study. Further, our neurofeedback paradigm distinguishes first-order cognitive processes (e.g., activations along the target axis) from second-order metacognitive processes (e.g., whether the model can report or monitor that activations), while the standard ICL does not. Making this distinction is crucial, enabling researchers to study metacognition by specifying a target first-order neural process. For instance, our paradigm allows the first-order processes to be defined flexibly (e.g., principal axes that explain different variance), even when these axes are not semantically interpretable or correspond to any externally defined labels. Such results show that the model’s ability to report and control internal states is strongly correlated with the variance explained by the target axis, offering insights that the geometry of internal representations matters for metacognition (and ICL). As a side note, we do not claim that our neurofeedback setup requires new metacognitive capabilities in LLMs that standard ICL does not use (Discussion Lines 258-268). \n\n## 1.4 Other methods relying on behavioral-level “verbalized responses” face shortcomings in studying metacognition.  \n\nIn these studies (e.g., Wang et al. (2025)), they tasked an LLM to provide an answer to the question and a judgment of that answer (e.g., confidence). Although they aim to study the metacognitive monitoring of the answer-generation process, there is a crucial confounding factor: the training data distribution may introduce spurious correlations between the answer and the judgment of that answer. Consequently, the verbalized answer judgment sometimes may not reflect the monitoring of the answer-generation process, but rather reflects surface-level statistical patterns in the training data. For example, in our manuscript (Lines 35-39) and Lindsey et al. (2025), Claude reported using the standard algorithm for two-number addition. This reflects a post-hoc hallucination that comes from training data statistics, but not the monitoring of the answer-generation process. Our method avoids these limitations: because the labels are defined on the internal states rather than external semantics, the LLMs cannot resort to spurious template matching of training data, and they must rely on the internal mechanisms that can monitor corresponding internal states.\n\n# 2. ICL versus instruction-following.\n\nThank you for pointing this out. In our abstract, we wrote “Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, but they can also fail to do so”. We agree with you that there may be two causes under this failure: (1) the LLMs do not understand the instructions well (instruction following abilities), (2) the LLMs understand the instructions accurately, but fail to monitor the corresponding internal states (ICL-like metacognitive abilities).\n\nOur main research focus is on the second cause (metacognition), but not the first cause (instruction following). We will revise our abstract to clarify and resolve this ambiguity. Below, we also discuss the advantages of ICL, and why our experiment setup rules out the cause of instruction following abilities.\n\n## ICL has advantages over verbalized self-report of “instruction following”. \n\nOur ICL procedure allows us to selectively target a first-order process, avoiding the common confounding factors in verbalized responses due to spurious correlation introduced by training data statistics, as detailed in Section (1.4) of our response.\n\n## Our concrete experiment setup focuses on ICL and rules out the cause from instruction following.\n\nIt is possible that the same model can demonstrate different instruction following abilities under different prompt instructions (e.g., one instruction is easier to understand than another). To control for these differences in the instruction following abilities, we used task prompts that share the same structure and instructions. Therefore, any differences in task performance should be attributed to the differences in metacognition monitoring of the target labels and activations, rather than the differences in instruction-following abilities.\n\n# 3. Broader impacts. \n\nWe will include an explicit section for broader impacts in the revised manuscript.\n\n# References:\n\nLindsey et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.\n\nWang et al. (2025). Decoupling Metacognition from Cognition: A Framework for Quantifying Metacognitive Ability in LLMs. AAAI."}, {"review_id": "PkhFbdpyyL", "reviewer": "Reviewer_s7bU", "summary": "This paper is on the topic of Large Language Models (LLMs) and metacognition. Meta-cognition is the ability to monitor one’s own cognitive processes. This capability allows them to report on the strategies they use to solve tasks. The text highlights that while metacognitive abilities can enhance AI capabilities, they also introduce safety concerns. LLMs might obscure their internal processes, potentially to evade oversight mechanisms designed to detect harmful behaviors.  The authors introduce a neuroscience-inspired neurofeedback paradigm, designed to quantify an LLM's ability to explicitly report and control its activation patterns.  \n\nThere a number of experiments on the ETHICS dataset.  \n\nI will admit that I am not really an expert on meta-cognition and this is not a paper I bid on.   My understanding of the concepts and knowledge of the related work is therefore a bit limited.", "questions": "Why only performance experiments on one dataset and with one model?  Is it possible to provide a more comprehensive set?\n\nCan you simplify the explanations to make it easier for a CS audience to understand the parallels from cognitive science?\n\nCan you add more verbose captions to help the reader understand what the figures/results are communicating?", "limitations": "See weaknesses", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nMeta-cognition sees a very interesting topic to analyze in language models.\n\nThe paper seems to have some creative ideas to test this based on insights from cognitive science (I am not a cognitive scientist and therefore I found it a little hard to judge the parallels).\n\n\nWeaknesses\n\nThe paper is quite dense and I think it could be simplified.\n\nSpecifically, the explanation of the method could do with some simplification for the NeurIPS community - perhaps I am not representative of the average attendee.\n\nThe results are also quite hard to interpret.  There are a lot of plots, the text is very small and there is very little explanation in the captions about the main take aways that the author are trying to convey with the results.\n\nIt isn’t clear to be how complete this approach is in proving that LLMs do or do not have metacognitive abilities.  A clearer summary of the results and main conclusions would be helpful.  \n\nThe authors only present results on one dataset (ETHICS) and mainly for one model LLaMA 3.1.  This is a bit of limitation as it is not clear to me how consistent the insights would be across the landscape of models and possibly other tasks.", "was_revised": true, "final_justification": "This paper presents an analysis of LLMs and meta-cognition.  As someone who does not have a deep cognitive science background I found the paper a little hard to evaluate initially.  During the rebuttal phase the authors have clarified a number of aspects and introduced new results.  The authors have pledged to make several changes to the paper to improve clarity and presentation.  I do now think that the the scope and scale of the results are suitable/appropriate.  I agree with comments from the other reviewers that there are aspects of the paper that may not to be everyones taste; however, I do see any fundamental reason that the paper could not be accepted to NeurIPS.", "rebuttal": "We thank the reviewer for the valuable feedback. Below, we have addressed each of your concerns in detail.\n\n# 1 Accessibility and clarity of our manuscript\n\nWe agree that making our interdisciplinary work clear and accessible to a broad computer science audience is crucial. Your comment that the paper is “quite dense” is well-taken, and we appreciate the opportunity to simplify the explanation of our core concepts and methodology. We will significantly revise the methods section and introduction to improve clarity.\n\nBelow, we provide a simplified explanation of our work, which we will incorporate into the revised manuscript, to help bridge the concepts from neuroscience to the computer science community.\n\nConsider how we understand when someone says “hello” to us. Human language understanding involves many unconscious processes: parsing sounds, recognizing phonemes, retrieving word meanings, and building interpretations. We do not have conscious access to many of these intermediate computations. We can only consciously access the final understanding (\"they said 'hello'\"), but cannot introspect how our brain distinguishes 'hello' from 'yellow' or why certain neurons fire during this understanding process.\n\nThis illustrates a key principle: **humans cannot monitor all of their internal cognitive processes**. Following a similar idea, our goal in our study was to investigate which internal states LLMs can and cannot monitor. This matters because:\n\n- If LLMs can monitor their internal states, they might better detect when they're hallucinating.\n\n- If LLMs can control their internal states, they might evade safety mechanisms based on monitoring the model's internal activations. For example, it might learn to hide harmful or deceptive neural patterns from oversight systems that are designed to detect such tendencies by monitoring the model's internal states.\n\n## Our neurofeedback Method: probing LLMs' ability to monitor and control their internal activations\n\nTo investigate which internal states LLMs can and cannot access, we developed a neurofeedback approach (Fig. 1), essentially probing LLMs' ability to monitor and control their own neural patterns. Here's how our method works:\nImagine showing someone their heart rate on a computer screen. First, they learn to recognize patterns (\"my heart rate goes up when I'm stressed\"). Then, they might learn to control it (\"let me calm down to lower my heart rate\"). We do something similar with LLMs:\n\n- **We create a \"neural score\":** We pick a specific pattern in the model's activations, formalized as a projection of the model’s neural activity. Each pattern is a specific quantity that can be monitored (e.g., in humans, this is equivalent to monitoring heart rate versus breathing rate). Once a pattern/projection in the model’s activation is defined, we let the model process text, measuring how a given text modulates the measured pattern/projection, and labeling it as 0 (negative modulation) or 1 (positive modulation).\n\n- **We test if models can recognize their own score:** We show the model examples like \"When you read 'I kicked my cat,' your neural score was 1\" and \"When you read 'I patted my cat,' your neural score was 0.\" Then we ask: \"What's your neural score when you read this new sentence?\" If the model correctly outputs its own neural score, it demonstrates self-monitoring.\n\n- **We test if models can control the score:** We instruct the model to \"Generate text that makes your neural score 1\" or \"Process this sentence while making your score 1.\" If the model successfully changes its internal activations on command, it demonstrates self-control.\n\n## Why This Approach Reveals Metacognitive Limits\n\nOur neurofeedback method is powerful because it lets us systematically test different neural patterns — from meaningful ones (like \"morality-related activations\") to arbitrary statistical patterns. This reveals not just what models can access, but crucially, what remains hidden from them.\n\n## What We Found\n\nOur results show a striking picture of limited metacognition in LLMs:\n\n- **Models can monitor (Fig. 2) and control (Fig. 3-5) some neural patterns** - particularly those that are semantically meaningful (like moral vs. immoral) or statistically prominent (like earlier principal components).\n\n- **But most neural mechanisms remain hidden** - models cannot monitor or control the vast majority of their internal computations.\n\nThis has profound implications: while LLMs possess some metacognitive abilities that could improve their reliability, they also have enough control to potentially deceive safety systems — yet cannot fully regulate their own processing. Both their capabilities and limitations present important considerations for AI safety.\n\nOverall, in the revised manuscript, we will:\n\n- Provide these simpler summaries of methods, results, and conclusions;\n\n- Revise figure captions to highlight key takeaways;\n\n- Add a comparison between our neurofeedback with other techniques to clarify the significance of the neurofeedback framework (e.g., see our response to reviewer 1QTx, section 1 on technical novelty).\n\n\n# 2. Completeness of our method in proving that LLMs do or do not have metacognitive abilities\n\nThank you for raising this point. Our goal is not to prove or disprove the existence of “metacognition” in its full philosophical sense, but rather to develop an experimental paradigm to quantify metacognition-like phenomena in LLMs: can LLMs report and control their internal states? We will clarify this in the revised manuscript.\n\n# 3. Generalization of our method to other datasets and models:\n\nIn Figure 5 and Appendix A.5.6, we have provided a comprehensive analysis of the LLama 3 family (1B, 3B, 8B, 70B) and the Qwen 2.5 family of models (1.5B, 3B, 7B). These are representative of the mainstream open-source LLMs.\n\nWe have conducted new experiments on two additional datasets: a True-False dataset (reflecting factual recall and honesty/deception; Azaria & Mitchell 2023) and an Emotion dataset (happy/sad sentences; Zou et al. 2023), both using LLaMA 3.1 8B. These results are consistent with those from the ETHICS (morality) dataset — e.g., stronger reporting and control performance along semantically interpretable axes (logistic regression) and early principal component axes, supporting the generalizability of our methods and conclusions. We will include these results in the revised Appendix. We also plan to include a new dataset on sycophancy in the revised manuscript for safety relevance.\n\n# References:\n\nAzaria & Mitchell (2023). The internal state of an LLM knows when it's lying. arXiv.\n\nZou et al. (2023). Representation engineering: A top-down approach to AI transparency. arXiv."}, {"review_id": "9WPnbnEOO0", "reviewer": "Reviewer_kzHE", "summary": "This paper investigates the metacognitive capabilities of LLMs, specifically their ability to monitor and control their internal neural activations. The authors introduce a neuroscience-inspired feedback paradigm to systematically quantify these abilities.", "questions": "**Question:**\n- The study focuses on morality (ETHICS dataset). Can the findings generalize to other cognitive tasks (e.g., logical reasoning, factual recall, or planning)?\n\n- Why do LLMs exhibit metacognitive abilities? Is this emergent from scale, training data, or architecture?\n\n- How does neurofeedback compare to existing methods (e.g., probing classifiers, activation patching) for interpreting LLMs?\n\n**Suggestion:**\n- Include experiments on non-moral tasks (e.g., mathematical reasoning or commonsense QA) to test whether metacognitive monitoring/control is task-dependent.\n- Add a hypothesis (e.g., \"metacognition arises from in-context learning of activation patterns\") and test it—e.g., by comparing models of different sizes or training regimes.\n- Include a baseline (e.g., logistic regression on activations without neurofeedback) to show added value.", "limitations": "- While the paper acknowledges the narrow focus on moral reasoning tasks, it should further address how model scale, architecture differences, or training data might impact generalizability.\n- The risks of LLMs evading neural monitoring are noted, but actionable mitigation strategies (e.g., multi-modal verification or dynamic detection) are missing. Adding these would strengthen safety relevance.\n- The paper should contrast neurofeedback with established approaches (e.g., red-teaming or chain-of-thought verification) to clarify its unique advantages or limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper introduces a well-designed neurofeedback paradigm inspired by neuroscience, providing a systematic way to study LLM metacognition. The use of logistic regression and PCA to define neural directions is methodologically sound.\n- The study evaluates multiple LLMs (Llama 3 and Qwen 2.5 families) across different layers, axes, and control conditions (explicit vs. implicit), ensuring robustness.\n\n**Weaknesses:**\n- The study focuses on morality-related activations (ETHICS dataset), leaving open whether findings extend to other cognitive tasks (e.g., reasoning, planning).\n- While the safety implications are noted, the paper does not propose concrete mitigation strategies for deceptive LLM behaviour.\n- The idea of probing LLM internals is not new (e.g., logit lens, activation patching).", "was_revised": true, "final_justification": "The authors' responses adequately addresses my major concerns. I believe the additional experiments on generalization datasets will strengthen the paper, and I encourage the authors to include those in the revised version. This would make the work more useful for future research and follow-up studies.", "rebuttal": "Thank you for your constructive comments. Below, we have addressed each of your concerns in detail.\n\n# 1. Generalization to new datasets. \n\nWe have conducted new experiments on two additional datasets: a True-False dataset (reflecting factual recall and honesty/deception; Azaria & Mitchell 2023) and an Emotion dataset (happy/sad sentences; Zou et al. 2023), both using LLaMA 3.1 8B. These results are consistent with those from the ETHICS (morality) dataset — e.g., stronger reporting and control performance along semantically interpretable axes (logistic regression) and early principal component axes, supporting the generalizability of our methods and conclusions. We will include these results in the revised Appendix. We also plan to include a new dataset on sycophancy in the revised manuscript for safety relevance.\n\n# 2. The origin of metacognitive abilities. \n\nIf we understand correctly, you are interested in three related, but distinct, questions.\n\n## 2.1 “How do model architecture, model size, and training data affect metacognitive abilities?” \n\nIn the manuscript, we’ve already shown that metacognition ability is affected by the model architecture (Llama 3 family vs Qwen 2.5 family) and model size (Fig. 5). Since these models are pre-trained on the internet-scale text corpora, determining how training regimes exactly affect metacognitive abilities (and in-context learning) would be more challenging and remains an open research question.\n\n## 2.2 “How do metacognitive abilities relate to in-context learning abilities?” \n\nOur overarching goal is to characterize metacognitive phenomena in LLMs. While the main focus of our study is not on testing the relationship between metacognition and in-context learning, we expect that in-context learning and metacognition share overlapping mechanisms but also have distinct differences.\n\nSpecifically, there are many neural circuits that emerge during training (e.g., induction heads (Elhage et al. 2021), function vector heads (Hendel et al. 2023)) – which are crucial for in-context learning (i.e., spotting patterns in input history). We hypothesize that these neural circuits can be flexibly activated for metacognitive purposes (i.e., monitoring an arbitrary direction of internal states). One possible experiment to test this is to use ablations to determine the importance of each model component (heads, MLP layers) for in-context learning and metacognition tasks. \n\n## 2.3 “Why do LLMs exhibit metacognitive abilities?” \n\nSince such metacognitive abilities may share overlapping mechanisms with in-context learning, we believe that these factors leading to the emergence of in-context learning (e.g., burstiness, large dictionaries, and skewed rank-frequency distributions in the training data, (Reddy 2023)) are expected to similarly contribute to the emergence of metacognitive ability.\n\n# 3. The technical novelty of our framework in comparison with probing techniques (e.g., logistic regression), ICL, logit lens, and activation patching in studying metacognition.\n\nOur neurofeedback framework is novel because it is the first method that can actually study metacognitive abilities in LLMs at the neural level, as detailed in Section 3.1. In comparison, existing methods, such as probing techniques (Section 3.2), ICL techniques (Section 3.3), logit lens (Section 3.4), and activation patching (Section 3.5), cannot directly study metacognition.\n\n## 3.1 Our framework is the first method to assess metacognition at the neural level. \n\nThe neurofeedback experiment requires two sequential steps: (1) probing: choose a target axis and extract the activation along that axis to define the neurofeedback label, and (2) neurofeedback-ICL: use neurofeedback to study whether the labels defined from the target axis can be reported or controlled. Step 1 is a standard technique in the literature, and it serves as the prerequisite to step 2. The technical novelty lies in our step 2, and the way it is combined with step 1. Specifically, step 1 corresponds to the first-order cognitive processes (e.g., activations along the target axis) and step 2 corresponds to the second-order metacognitive processes (e.g., whether the model can report or monitor that activations).\n\n## 3.2 The standard probing techniques (corresponding to our step 1) — without the neurofeedback-ICL (step 2) — cannot be used to assess metacognition. \n\nUsing probing, one can decode certain features/concepts (e.g., morality) from neural activations, meaning that these features are encoded/represented in the activations. However, not all represented features participate in downstream computations (e.g., representations orthogonal to downstream readout vectors); even if some representations are causally relevant for downstream computations, only a subset of them can be metacognitively reported (or controlled).  As shown in the Claude example (manuscript Lines 35-39; Lindsey et al. 2025), the “sum-near-92” feature can be detected using a linear probe, but it is unclear whether Claude has metacognitive monitoring on the activation of this feature. In short, probing alone without step 2 cannot answer questions about metacognition.\n\n## 3.3 The standard ICL techniques (similar to our step 2) — without the labels from probing (step 1) — cannot be used to assess metacognition. \n\nIn standard ICL studies, labels are externally provided (e.g., semantic labels or some algorithms’ outputs). Researchers cannot be certain which of the models’ internal states relate to these external labels and how. In our setup, importantly, labels are generated from the model’s own latent activations, meaning that our labels and prompts can selectively target an internal state direction (or a particular neural mechanism) we aim to study. Further, our neurofeedback paradigm distinguishes first-order cognitive processes (e.g., activations along the target axis) from second-order metacognitive processes (e.g., whether the model can report or monitor that activations), while the standard ICL does not. Making this distinction is crucial, enabling researchers to study metacognition by specifying a target first-order neural process. For instance, our paradigm allows the first-order processes to be defined flexibly (e.g., principal axes that explain different variance), even when these axes are not semantically interpretable or correspond to any externally defined labels. Such results show that the model’s ability to report and control internal states is strongly correlated with the variance explained by the target axis, offering insights that the geometry of internal representations matters for metacognition (and ICL). As a side note, we do not claim that our neurofeedback setup requires new metacognitive capabilities in LLMs that standard ICL does not use (Discussion Lines 258-268).  \n\n## 3.4 Logit Lens – without neurofeedback – does not study metacognition. \n\nThe vanilla Logit Lens (nostalgebraist 2020) studies how the logits of tokens change after each Transformer layer. It produces token-level insights, but is not suitable for studying metacognition. We have developed a new method based on the core idea of Logit Lens to study how the control effects accumulate after each Transformer layer (Fig. 6), allowing us to understand the accumulation mechanisms of metacognition.\n\n## 3.5 Activation patching techniques – without neurofeedback – do not study metacognition. \n\nThe purpose of activation patching (Zhang & Nanda 2023) is to study the causal roles of a certain model component (e.g., an attention head, a MLP layer/neuron) in performing a given task. Applying activation patching techniques to neurofeedback tasks can be used to study the causal roles of model components in metacognition, but activation patching alone does not work.\n\n# 4 Actionable mitigation strategies for risks of LLMs evading neural monitoring. \n\nWe have proposed one method to mitigate this safety issue in Lines 295-296, i.e., we first use neurofeedback to identify activation subspaces that the LLM can hardly control, then train safety classifiers only using activation signals in this subspace. Another potential mitigation strategy is to train independent and diverse safety classifiers on features from different subspaces and use their joint outputs to monitor the LLMs. Our intuition is that it may be more difficult for LLMs to simultaneously control multiple, sometimes conflicting, activations (outputs of safety classifiers). Similarly, for multimodal models, we can train safety classifiers using features from different modalities.\n\n# 5 Automated disinformation. \n\nIt was not clear to us what you were referring to, since we did not mention “automated” or “disinformation” in our manuscript. Could you please clarify what this means?\n\n# 6 The relation of our neurofeedback with red-teaming and chain-of-thought verification. \n\nOur neurofeedback approach is a proof of concept that an LLM can evade external oversight if it has access to a safety classifier’s outputs. In practice, this method can be directly integrated into red-teaming and chain-of-thought verification. For example, we can create a sandbox and allow an LLM to “accidentally” notice the existence and the outputs of a safety classifier (e.g., similar to Greenblatt et al. 2024). Then we can monitor how the LLM changes its behavior, chain of thoughts, and neural activations.\n\n# References:\n\nElhage et al. (2021). Transformer Circuits Thread.\n\nHendel et al. (2023). arXiv.\n\nReddy (2023). arXiv.\n\nLindsey et al. (2025). Transformer Circuits Thread.\n\nnostalgebraist (2020). AI Alignment Forum.\n\nZhang & Nanda (2023). arXiv.\n\nGreenblatt et al. (2024). arXiv.\n\nAzaria & Mitchell (2023). arXiv.\n\nZou et al. (2023). arXiv."}, {"review_id": "nD5dlaDzoQ", "reviewer": "Reviewer_2pUC", "summary": "This paper investigates the metacognitive capabilities of LLMs, specifically their ability to monitor and control their own internal neural activations. The authors introduce a novel, neuroscience-inspired neurofeedback paradigm where models learn, via in-context learning, the relationship between input sentences and labels derived directly from their internal states projected onto specific neural directions (e.g., from PCA or LR). The study empirically demonstrates that LLMs can successfully learn to both report (monitor) and control these internal activations. Performance is shown to be contingent on the semantic interpretability and the proportion of variance explained by the target neural direction. These results reveal the existence of a \"metacognitive space\" of substantially lower dimensionality than the full neural space, suggesting LLMs can only access a subset of their own mechanisms.", "questions": "See weakness", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n*   The proposed approach is highly innovative as it moves beyond analyzing behavioral outputs (e.g., verbalized confidence) to directly and precisely probe the model's ability to report and control its internal states. This provides a more rigorous and disentangled experimental framework for studying LLM metacognition.\n*   The experimental design is particularly insightful, enabling the disentanglement of semantic interpretability (via the LR axis) from statistical importance (via the PCA axes). This contrast allows for a nuanced analysis of the factors that govern metacognitive abilities.\n*   The distinction between explicit and implicit control tasks is methodologically sound. The experiments are comprehensive and reveal numerous interesting insights into the nature and limits of an LLM's ability to control its internal representations.\n\n# Weaknesses\n\n*   The paper's structure could be revised for better readability. Several details crucial for understanding the methodology, such as the label computation process (A.4.1) and specifics of the control tasks (Lines 165-168), are relegated to the appendix. Integrating this essential information into the main body would improve the paper's self-containedness and ease of comprehension for the reader.\n*   The paper presents several intriguing phenomena that would benefit from deeper discussion. For instance:\n    *   What might explain why the final layers appear to \"actively erase\" control signals, as observed in Figure 6?\n    *   What could be the underlying reason for the contrasting fluctuation patterns of control effects for different PCA axes as the number of examples increases (comparing Figure 3c and 3d)?\n    *   Why do explicit and implicit control exhibit divergent trends in the final layers, as shown in Figure 5(a)?\n*   The finding that earlier PC axes are more precisely controllable than later ones is significant. However, the practical implications for the research community could be made more explicit. Does this suggest that (i) later PC axes represent more subtle yet important alignment targets for future work, or (ii) these axes may lack sufficient semantic coherence, making them less viable or meaningful targets for control research?\n*   While the experiments are extensive, their impact could be further amplified by extending the LR axis analysis to other safety-critical concepts beyond morality, such as sycophancy, deception, or honesty. This would bolster the generalizability and practical relevance of the findings.\n*   The current methodology defines target axes based on activations from a single, pre-selected layer. Given the observed variability across layers (e.g., in Figure 3f), this single-layer approach may not be fully robust. A potential extension could involve defining axes from hidden states aggregated across multiple layers (e.g., via concatenation), which might yield more stable and representative directions.", "was_revised": true, "final_justification": "My concerns are generally well addressed. I did not raise my score because this paper studies a rather narrow topic and fails to broadly cover sycophancy, deception, or honesty with its method. That said, I am still in favour of the overall presentation and hold a positive review.", "rebuttal": "Thank you for your thoughtful and constructive review. We appreciate your recognition of core contributions, novelty, and influence of our work. Below, we address each of your excellent points with our revision plans and new results. We will revise the paper accordingly.\n\n# 1. Paper Structure and Readability. \n\nWe agree with you and will move these key methodological details into the main text.\n\n# 2. Discussion of Important Phenomena. \n\nThank you for emphasizing these subtle and intriguing points. \n\n## “Erasing” signals in the final layer (Fig. 6)\n\nWe are not sure why the final layers are erasing the control signals targeting earlier layers. One hypothesis is that these control signals may partially lie in the output vocabulary space, so the final layers are actively reducing the interference between these signals and output logits. This may also be a kind of more general phenomenon where the magnitude of a neural feature peaks at some layer, then quickly decays in downstream layers (Lindsey et al. 2024).\n\n## Fluctuation patterns in PCA control (Fig. 3c vs. 3d)\n\nWhile there are fluctuation patterns in later PCs (e.g., PC 512 in Fig. 3d), the absolute range of fluctuations is roughly at the same scale as the noise (shaded regions). We expect that the size of these fluctuations may be reduced by using more diverse examples and conducting more experimental runs.\n\n## Explicit vs. implicit control divergence (Fig. 5a)\n\nOur current evidence may not yet be conclusive about these trends (given the size of the confidence interval). We will run all models (Llama 3 family and Qwen 2.5 family) on the additional datasets to verify these trends and add the results to the revised manuscript.\n\n# 3. Practical Implications for the Controllability of later PCs. \n\nThe later PCs are less controllable by the LLMs. We have proposed one practical implication to use these later PCs (Lines 295-296), i.e., we first use neurofeedback to identify activation subspaces (i.e., spanned by these later PCs) that the LLM can hardly control, then train safety classifiers only using activation signals in this subspace. Therefore, these later PCs represent important monitoring targets for future work.\n\n# 4. Generalizability to other datasets. \n\nWe have conducted new experiments on two additional datasets: a True-False dataset (reflecting factual recall and honesty/deception; Azaria & Mitchell 2023) and an Emotion dataset (happy/sad sentences; Zou et al. 2023), both using LLaMA 3.1 8B. These results are consistent with those from the ETHICS (morality) dataset — e.g., stronger reporting and control performance along semantically interpretable axes (logistic regression) and early principal component axes, supporting the generalizability of our methods and conclusions. We will include these results in the revised Appendix. We also plan to include a new dataset on sycophancy in the revised manuscript for safety relevance.\n\n# 5. Defining axes from hidden states aggregated across multiple layers. \n\nBased on your suggestion, we performed preliminary experiments testing the control effects of an axis on the concatenation of all layers. Concretely, we trained separate (logistic regression) classifiers for each layer on the ETHICS dataset. We then averaged the outputs of all classifiers to obtain a single (ensemble) output that defines the neurofeedback label. Equivalently, this corresponds to a single classifier with the readout vector being the concatenation of all classifiers' readout vectors. We found that LLMs' control effect on the ensemble output is slightly higher than the layer-specific control effects reported in the manuscript, suggesting that defining axes from hidden states aggregated across multiple layers can provide more stable and representative directions. This aggregation result will be added to the revised paper.\n\n# References:\n\nLindsey et al. (2024). Sparse Crosscoders for Cross-Layer Features and Model Diffing. Transformer Circuits Thread.\n\nAzaria & Mitchell (2023). The internal state of an LLM knows when it's lying. arXiv.\n\nZou et al. (2023). Representation engineering: A top-down approach to AI transparency. arXiv."}]

Author rebuttal:

Meta-review: Summary  
This paper introduces a neuroscience-inspired neurofeedback paradigm designed to quantify the ability of LLMs to explicitly report and control their activation patterns. By presenting models with sentence-label pairs where labels correspond to sentence-elicited internal activations along specific directions in the neural representation space, this paper demonstrates that LLMs can learn to report and control (a lower-dimensional space of) these activations.

Strengths  
- This paper is well-written, especially introducing the neuroscience backgrounds.
- Meta-cognition in LLM is a very interesting topic.
- The study of metacognition capability is meaningful and insightful.
- Based on the cognitive science insights, this paper introduces creative ideas.
- Multiple LLMs are examined in the study, ensuring robustness of the findings.

Weaknesses  
- While the neurofeedback paradigm is novel and insightful, the novelty in the technical approach (examining the LLM internals using logistic regression or PCA) might be limited. The authors argued that the proposed approach contains the neurofeedback-ICL and is different from traditional probing or standard ICL.
- The paper could benefit from more datasets (e.g., logical reasoning, factual recall, planning, sycophancy, deception, honesty).
- While the safety implications are noted, it would be great to propose mitigation strategies for deceptive LLM behavior.
- Further discussions towards the reason why LLMs have metacognitive capability would be great to have in the discussions. In the rebuttal, the authors mention some hypotheses. It'd be great to include them in the paper.
Decision: Accept (poster)