Title: Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

metacognition

safety

neurofeedback

in-context learning
Abstract: Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, yet at other times seem unable to recognize those strategies that govern their behavior. This suggests a limited degree of metacognition --- the capacity to monitor one's own cognitive processes for subsequent reporting and self-control. Metacognition enhances LLMs' capabilities in solving complex tasks but also raises safety concerns, as models may obfuscate their internal processes to evade neural-activation-based oversight (e.g., safety detector). Given society's increased reliance on these models, it is critical that we understand their metacognitive abilities. To address this, we introduce a neuroscience-inspired \emph{neurofeedback} paradigm that uses in-context learning to quantify metacognitive abilities of LLMs to \textit{report} and \textit{control} their activation patterns. 
We demonstrate that their abilities depend on several factors: the number of in-context examples provided, the semantic interpretability of the neural activation direction (to be reported/controlled), and the variance explained by that direction. These directions span a ``metacognitive space'' with dimensionality much lower than the model's neural space, suggesting LLMs can monitor only a small subset of their neural activations. Our paradigm provides empirical evidence to quantify metacognition in LLMs, with significant implications for AI safety (e.g., adversarial attack and defense).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the metacognitive capabilities of LLMs, specifically their ability to monitor and control their own internal neural activations. The authors introduce a novel, neuroscience-inspired neurofeedback paradigm where models learn, via in-context learning, the relationship between input sentences and labels derived directly from their internal states projected onto specific neural directions (e.g., from PCA or LR). The study empirically demonstrates that LLMs can successfully learn to both report (monitor) and control these internal activations. Performance is shown to be contingent on the semantic interpretability and the proportion of variance explained by the target neural direction. These results reveal the existence of a \"metacognitive space\" of substantially lower dimensionality than the full neural space, suggesting LLMs can only access a subset of their own mechanisms.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n*   The proposed approach is highly innovative as it moves beyond analyzing behavioral outputs (e.g., verbalized confidence) to directly and precisely probe the model's ability to report and control its internal states. This provides a more rigorous and disentangled experimental framework for studying LLM metacognition.\n*   The experimental design is particularly insightful, enabling the disentanglement of semantic interpretability (via the LR axis) from statistical importance (via the PCA axes). This contrast allows for a nuanced analysis of the factors that govern metacognitive abilities.\n*   The distinction between explicit and implicit control tasks is methodologically sound. The experiments are comprehensive and reveal numerous interesting insights into the nature and limits of an LLM's ability to control its internal representations.\n\n# Weaknesses\n\n*   The paper's structure could be revised for better readability. Several details crucial for understanding the methodology, such as the label computation process (A.4.1) and specifics of the control tasks (Lines 165-168), are relegated to the appendix. Integrating this essential information into the main body would improve the paper's self-containedness and ease of comprehension for the reader.\n*   The paper presents several intriguing phenomena that would benefit from deeper discussion. For instance:\n    *   What might explain why the final layers appear to \"actively erase\" control signals, as observed in Figure 6?\n    *   What could be the underlying reason for the contrasting fluctuation patterns of control effects for different PCA axes as the number of examples increases (comparing Figure 3c and 3d)?\n    *   Why do explicit and implicit control exhibit divergent trends in the final layers, as shown in Figure 5(a)?\n*   The finding that earlier PC axes are more precisely controllable than later ones is significant. However, the practical implications for the research community could be made more explicit. Does this suggest that (i) later PC axes represent more subtle yet important alignment targets for future work, or (ii) these axes may lack sufficient semantic coherence, making them less viable or meaningful targets for control research?\n*   While the experiments are extensive, their impact could be further amplified by extending the LR axis analysis to other safety-critical concepts beyond morality, such as sycophancy, deception, or honesty. This would bolster the generalizability and practical relevance of the findings.\n*   The current methodology defines target axes based on activations from a single, pre-selected layer. Given the observed variability across layers (e.g., in Figure 3f), this single-layer approach may not be fully robust. A potential extension could involve defining axes from hidden states aggregated across multiple layers (e.g., via concatenation), which might yield more stable and representative directions.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the metacognitive capabilities of LLMs, specifically their ability to monitor and control their internal neural activations. The authors introduce a neuroscience-inspired feedback paradigm to systematically quantify these abilities.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper introduces a well-designed neurofeedback paradigm inspired by neuroscience, providing a systematic way to study LLM metacognition. The use of logistic regression and PCA to define neural directions is methodologically sound.\n- The study evaluates multiple LLMs (Llama 3 and Qwen 2.5 families) across different layers, axes, and control conditions (explicit vs. implicit), ensuring robustness.\n\n**Weaknesses:**\n- The study focuses on morality-related activations (ETHICS dataset), leaving open whether findings extend to other cognitive tasks (e.g., reasoning, planning).\n- While the safety implications are noted, the paper does not propose concrete mitigation strategies for deceptive LLM behaviour.\n- The idea of probing LLM internals is not new (e.g., logit lens, activation patching).\n\nQUESTIONS:\n**Question:**\n- The study focuses on morality (ETHICS dataset). Can the findings generalize to other cognitive tasks (e.g., logical reasoning, factual recall, or planning)?\n\n- Why do LLMs exhibit metacognitive abilities? Is this emergent from scale, training data, or architecture?\n\n- How does neurofeedback compare to existing methods (e.g., probing classifiers, activation patching) for interpreting LLMs?\n\n**Suggestion:**\n- Include experiments on non-moral tasks (e.g., mathematical reasoning or commonsense QA) to test whether metacognitive monitoring/control is task-dependent.\n- Add a hypothesis (e.g., \"metacognition arises from in-context learning of activation patterns\") and test it—e.g., by comparing models of different sizes or training regimes.\n- Include a baseline (e.g., logistic regression on activations without neurofeedback) to show added value.\n\nLIMITATIONS:\n- While the paper acknowledges the narrow focus on moral reasoning tasks, it should further address how model scale, architecture differences, or training data might impact generalizability.\n- The risks of LLMs evading neural monitoring are noted, but actionable mitigation strategies (e.g., multi-modal verification or dynamic detection) are missing. Adding these would strengthen safety relevance.\n- The paper should contrast neurofeedback with established approaches (e.g., red-teaming or chain-of-thought verification) to clarify its unique advantages or limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is on the topic of Large Language Models (LLMs) and metacognition. Meta-cognition is the ability to monitor one’s own cognitive processes. This capability allows them to report on the strategies they use to solve tasks. The text highlights that while metacognitive abilities can enhance AI capabilities, they also introduce safety concerns. LLMs might obscure their internal processes, potentially to evade oversight mechanisms designed to detect harmful behaviors.  The authors introduce a neuroscience-inspired neurofeedback paradigm, designed to quantify an LLM's ability to explicitly report and control its activation patterns.  \n\nThere a number of experiments on the ETHICS dataset.  \n\nI will admit that I am not really an expert on meta-cognition and this is not a paper I bid on.   My understanding of the concepts and knowledge of the related work is therefore a bit limited.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nMeta-cognition sees a very interesting topic to analyze in language models.\n\nThe paper seems to have some creative ideas to test this based on insights from cognitive science (I am not a cognitive scientist and therefore I found it a little hard to judge the parallels).\n\n\nWeaknesses\n\nThe paper is quite dense and I think it could be simplified.\n\nSpecifically, the explanation of the method could do with some simplification for the NeurIPS community - perhaps I am not representative of the average attendee.\n\nThe results are also quite hard to interpret.  There are a lot of plots, the text is very small and there is very little explanation in the captions about the main take aways that the author are trying to convey with the results.\n\nIt isn’t clear to be how complete this approach is in proving that LLMs do or do not have metacognitive abilities.  A clearer summary of the results and main conclusions would be helpful.  \n\nThe authors only present results on one dataset (ETHICS) and mainly for one model LLaMA 3.1.  This is a bit of limitation as it is not clear to me how consistent the insights would be across the landscape of models and possibly other tasks.\n\nQUESTIONS:\nWhy only performance experiments on one dataset and with one model?  Is it possible to provide a more comprehensive set?\n\nCan you simplify the explanations to make it easier for a CS audience to understand the parallels from cognitive science?\n\nCan you add more verbose captions to help the reader understand what the figures/results are communicating?\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the metacognition in LLMs, which is defined as the LLM’s capacity or ability to monitor its own internal process. The paper first motivates metacognition as a neurofeedback paradigm rooted in neuroscience, that humans have been found to be able to control their own mental states (such as controlling one’s own sense of fear). Using in-context learning as the setting, the authors explore whether the LLMs can also report and control their own activations. The experiment setting first define a target axis using Logistic Regression or PCA on the latent activations, then detect changes along the target axis.  Experiments demonstrate that LLMs can only partially report and modulate this internal signal, signifying the LLM’s ability to control its latent activation via input changes.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written and easy-to-follow. The introduction of the metacognition is friendly to readers who do not have a background in neuroscience. The parallel between neurofeedback in humans versus in LLMs, namely their differences and how it is carried out using in-context learning is also interesting and a novel connection. The experiments are extremely detailed and thorough, which provides the reader confidence in their correctness and reliability. \n\nOne of its weakness is that there is no technical novelty to this work per-se. Examining the internal states of LLMs using Logistic Regression or PCA is a well-established methodology (which the authors cited). Other than its neurofeedback story, which is insightful and meaningful in itself, it is hard to infer what is technically new about the methodology that the authors presented. To extend this point, the author mentioned related works that perform the same “probing” in LLMs to explore how well LLMs themselves can perform the given tasks. Is it fair to also argue that they are also doing some form of “neurofeedback” or “metacognition”? The novelty of this work can be greatly strengthened if this is stated more clearly.\n\nQUESTIONS:\n1.\tIs there a reason why In-context learning is chosen as the research setting over other potential settings such as instruction-following (i.e. can LLMs report whether they can follow instructions)? The question is natural because the paper abstract of this work begins with “Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, but they can also fail to do so.” This motivation seems to suggest instruction-following capabilities rather than some prediction capability as suggested via In-context learning.\n\nLIMITATIONS:\nThe authors did not have an explicit section called \"broader impacts,\" but it was discussed in great detailed in their discussion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}