Title: Concept-Guided Interpretability via Neural Chunking
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

LLM

cognitive science
Abstract: Neural networks are often described as 
black boxes, reflecting the significant challenge of understanding their internal workings and interactions. We propose a different perspective that challenges the prevailing view: rather than being inscrutable, neural networks exhibit patterns in their raw population activity that mirror regularities in the training data. We refer to this as the \textit{Reflection Hypothesis} and provide evidence for this phenomenon in both simple recurrent neural networks (RNNs) and complex large language models (LLMs).
Building on this insight, we propose to leverage cognitively-inspired methods of \textit{chunking} to segment high-dimensional neural population dynamics into interpretable units that reflect underlying concepts.
We propose three methods to extract these emerging entities, complementing each other based on label availability and neural data dimensionality. Discrete sequence chunking (DSC) creates a dictionary of entities in a lower-dimensional neural space; population averaging (PA) extracts recurring entities that correspond to known labels; and unsupervised chunk discovery (UCD) can be used when labels are absent. 
We demonstrate the effectiveness of these methods in extracting entities across varying model sizes, ranging from inducing compositionality in RNNs to uncovering recurring neural population states in large language models with diverse architectures, and illustrate their advantage to other interpretability methods. 
Throughout, we observe a robust correspondence between the extracted entities and concrete or abstract concepts in the sequence. Artificially inducing the extracted entities in neural populations effectively alters the network's generation of associated concepts.
Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data to reveal the hidden computations of complex learning systems, gradually transforming them from black boxes into systems we can begin to understand.

Implementation and code are publicly available at _https://github.com/swu32/Chunk-Interpretability_


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new framework for interpretability, inspired by cognitive science. The main claim is that neural networks’ activations mirror the statistical properties of the distributions they model, termed the Reflection Hypothesis. The authors propose several dimensionality reduction algorithms to extract interpretable concepts from the activations and show that the concepts causally impact NN behavior.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper introduces a novel framework for conducting future interpretability experiments that is grounded in cognitive science, rather than more mechanistic or representational approaches. The framework offers a human-in-the-loop approach to interpretability that may pair well with AI agents.\n- The paper shows the hypothesis across multiple network architectures and that it can lead to interpretable concepts, such as narrative structures in passages. These appear to be novel insights that can drive NN behavior.\n\n## Weaknesses\n- Some of the results seem to be reformulations of previously known results in interpretability, e.g., that \"Recurring chunk patterns emerge, such as similar activations for commas and adjectives in early layers\" is similar to the findings in [1] \n- Two of the proposed methods seem somewhat impractical. Discrete sequence chunking is mostly tenable when the NN’s hidden dimension $d$ is somewhat small. The authors then propose Neural Population Averaging, but this requires some hypothesis as to the concepts learned by the network. Specifically, the authors write that “this method relies on prior knowledge of recurring patterns in the input sequence”. Thus the main practical approach is unsupervised chunk discovery, but the only algorithm the authors give here is a variant of a k-means clustering algorithm. \n\n[1] https://karpathy.github.io/2015/05/21/rnn-effectiveness/\n\nQUESTIONS:\n1. Could there be more discussion and exploration of other possible unsupervised algorithms for chunking? I would consider raising my score if there were interesting, novel results there.\n\nLIMITATIONS:\nno, please discuss possible limitations of when the reflection hypothesis might break down or when it might be difficult to extract insights from the activations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel interpretability framework based on the “Reflection Hypothesis,” proposing that neural networks reflect structured regularities in their training data. The authors develop cognitively inspired chunking methods to extract recurring neural entities from RNNs and LLMs. Empirical results across architectures demonstrate that these extracted chunks correspond to meaningful concepts and can causally modulate model behavior.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe proposed Population Averaging (PA) method demonstrates superior performance in predicting concept occurrences compared to pretrained Sparse Autoencoders (SAEs), which is impressive. It is also a strong point that these results generalize across diverse model architectures, including Mamba and others.\n\n### Weaknesses\n- The core claim of the paper—the reflection hypothesis (i.e., that neural networks' internal representations reflect the structure of their training data)—does not appear particularly novel. Prior work such as [1] suggests that neural networks implicitly acquire syntactic structure; [2] shows that the geometric structure of SAE features reflects underlying data structure; and [3] demonstrates that cyclic structures in time-series data (e.g., weeks and months) are mirrored in internal representations. These studies collectively indicate that the reflection hypothesis is not new, and the contribution here may be more confirmatory than novel.\n\n- The result in Figure 1(a), where replacing the hidden state of an RNN alters its output, seems rather expected—what is the new insight here?\n\n- It is unclear why the authors use RNNs for toy tasks and Transformer-based LLMs for real-world scenarios. Why are there no toy experiments with Transformers to support consistency across architectures?\n\n- In line 258, the paper states: “Figure 2a shows normalized activations of the first 50 neurons (unsorted) across all LLaMA3’s layers for the word ‘cheese’.” However, the result shown is from an RNN, not LLaMA3.\n\n[1] Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs.    \n[2] The Geometry of Concepts: Sparse Autoencoder Feature Structure.    \n[3] Not All Language Model Features Are One-Dimensionally Linear.\n\nQUESTIONS:\n- What are the practical advantages of this chunking method over existing interpretability tools such as SAE?\nFor example, SAEs have been used for unlearning [4] and steering [5]. Can chunking be used similarly?\n- Is the idea of chunking applicable beyond the language modality?\n\n\n[4] Applying Sparse Autoencoders to Unlearn Knowledge in Language Models.    \n[5] A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the hypothesis, motivated by findings from human cognitive science, that artificial neural networks (ANNs) memorize recurring structural patterns in training data. The authors propose methods to identify such recurring patterns, or “chunks,” by clustering activation vectors. An activation vector $h \\in \\mathbb{R}^d$ is considered part of a chunk if its subvector $h_C$ lies within a threshold distance of a chunk centroid. For low-dimensional settings, chunk extraction is done by discretizing neuron activations into strings and forming a dictionary of recurring patterns. For large-scale networks, the authors use a supervised method, Neural Population Averaging (PA), to identify concept-specific chunks, and an unsupervised method that optimizes a dictionary to ensure each input is close to at least one entry. Through experiments on RNNs and transformer LLMs, they show that the extracted chunks correspond to interpretable and recurring input patterns, can be manipulated to affect model predictions, and capture hierarchical or contextual input structure.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper is generally well-written and clearly structured.\n* It presents a novel perspective on interpretability, drawing inspiration from human cognitive science, which adds an interesting conceptual framing to the work.\n* The proposed method has demonstrated utility in downstream applications, by enabling causal interventions on model outputs through chunk manipulation.\n\n\n**Weaknesses**\n\n* While the work aligns with the broader literature on concept-based interpretability and mechanistic interpretability, the related work section lacks coverage of key concept discovery methods beyond sparse autoencoders (SAE). Additionally, these competing methods are not included in the quantitative evaluation \\[1–6].\n* Most quantitative results represent an exploratory analyses rather than a rigorous, systematic evaluation.\n* There are several clarity issues throughout the paper:\n\n  1. The Reflection Hypothesis is introduced in a vague and imprecise manner.\n  2. It is not stated whether a single input can be associated with multiple chunks.\n  3. Key terms such as *grafting* and *freezing* are not properly introduced or formally defined.\n* The decision process behind the chunking strategies in Section 3.1 is not thoroughly explained. The overall methodology appears somewhat ad hoc and lacks strong theoretical grounding.\n\n**Minor Comments:**\n\n* Missing space after “(SAEs)” (line 69), “at the time of the concept-specific index” (lines 156–157), and “Figure9” (line 282), double period typo in “neural activities..” (line 234).\n* Figure 2a (line 258) reference does not point to the correct figure.\n* Figure 3 (left side): unclear how different tokenizations are represented. \n* Some variables are not defined or are ambiguous:\n\n  * $\\overline{h_C}$ (line 128) \n  * $h_1, \\ldots, h_d$ (lines 130–131)\n\n\nReferences:\n\n[1] Ghorbani, A., Wexler, J., Zou, J.Y. and Kim, B., 2019. Towards automatic concept-based explanations. Advances in neural information processing systems, 32.\n\n[2] Vielhaben, J., Bluecher, S. and Strodthoff, N., Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees. Transactions on Machine Learning Research.\n\n[3] Fel, T., Boutin, V., Béthune, L., Cadène, R., Moayeri, M., Andéol, L., Chalvidal, M. and Serre, T., 2023. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems, 36, pp.54805-54818.\n\n[4] Black, S., Sharkey, L., Grinsztajn, L., Winsor, E., Braun, D., Merizian, J., Parker, K., Guevara, C.R., Millidge, B., Alfour, G. and Leahy, C., 2022. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312.\n\n[5] Tamkin, A., Taufeeque, M. and Goodman, N., 2024, July. Codebook Features: Sparse and Discrete Interpretability for Neural Networks. In International Conference on Machine Learning (pp. 47535-47563). PMLR.\n\n[6] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J. and Viegas, F., 2018, July. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning (pp. 2668-2677). PMLR.\n\nQUESTIONS:\n1. Please describe how the proposed chunking method compares to other concept discovery approaches (e.g., \\[1–6]). What are the conceptual differences and potential advantages?\n2. Provide a more formal and precise description of the Reflection Hypothesis.\n3. Clearly define the terms *grafting* and *freezing* within the paper.\n4. Explain the theoretical or empirical rationale behind the chunk clustering algorithm. Which theory, if any, guided the algorithm design choices?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the Reflection Hypothesis - the idea that the raw neural population activity of neural networks exhibits patterns that reflect the statistical regularities of the training data. To interpret these dynamics, the authors propose several cognitively-inspired 'chunking' methods to segment high-dimensional neural trajectories into recurring, interpretable units. These include Discrete Sequence Chunking (DSC) when $d$ is small (low-dimensional RNNs), Neural Population Averaging (PA) when $d$ is very large (transformers), and Unsupervised Chunk Discovery (UCD) when labels are unavailable. Interestingly, across architectures (RNNs and LLaMA-3), the authors demonstrate that these methods recover recurring internal states aligned with concrete and abstract input concepts. Furthermore, intervention-style experiments (e.g., grafting neurons to discovered chunks) demonstrate that some discovered chunks have a systematic causal effect on model behaviour, providing evidence that chunks are not just spurious features, but correspond to *functionally* meaningful internal representations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: \n- The paper addresses a central challenge in modern deep learning - how to interpret the internal structure and reuse of representations in large models. \n- Introduces a compelling conceptual shift in how we approach neural interpretability. Rather than focusing on individual neurons or attention heads, the authors argue that the dynamics of the neural population should be the primary unit of analysis. Crucially, the authors also challenge the dominant focus on linear decodability (e.g., probing), and instead emphasises functional and causal efficacy of the units of analysis. This reframing aligns interpretability research more closely with cognitive neuroscience, where population dynamics are a central unit of analysis. \n- Methodological breadth: the paper introduces three chunking methods for use across different settings (supervised vs unsupervised, low/high-dimensional settings), making the framework quite applicable to many scenarios. \n- The use of causal interventions (e.g., chunk grafting) demonstrate the chunks are not just correlated with behaviour, but causally implicated.  This kind of interventional evaluation strengthens the paper's empirical claims. \n- Architecture agnostic evaluation: chunking framework validated across diverse architectures, supporting the claim that chunk-like internal structure is a general emergent property arising from statistical regularities of the training data, rather than any specific model architecture.\n\n**Weaknesses** \n- Chunking methods are insensitive to temporal dynamics: only one of the three proposed methods (DSC) explicitly incorporates temporal information by merging adjacent symbols in the sequence. The other two methods are snapshot based, which effectively ignores the trajectory-level structure that is essential to sequential computation (especially in models e.g., transformers processing long sequences).   \n- Context invariance assumption for PA: assuming that a concept can be represented by a single population vector obtained by averaging across occurrences may be invalid in context-sensitive models (e.g., a concept like 'cat' could produce different neural activations depending on its context, and averaging across all instances could obscure the underlying structure). This compounds the issue of using static snapshots, since the temporal and contextual dependencies are not preserved in the averaged chunk.  \n- Evaluation: for the LLM case, there are only qualitative examples (e.g., for grafting and freezing) instead of quantitative metrics in the interventional test to causally implicate chunk identity in the model's functional behaviour. This makes difficult to assess whether the claim that chunks are functionally useful actually generalises to these larger models. Similarly, in the LLM case, for quantitative metrics (e.g., decoding in Fig 3), the evaluation setting is quite narrow (3 words), again making it difficult to assess whether the chunk behaviours are exceptions or consistent patterns that hold for larger models.  \n- Lack of generalisation analysis for chunks: Relating to point 2, while chunks are learned in a context-agnostic way, it is unclear that the chunks can be consistently reused across different contexts. I would be curious to see some evaluation of a learned chunk's (e.g., 'cheese') consistency across different contexts, perhaps by e.g., grafting the chunk across multiple contexts and quantitatively measuring that the impact across all contexts is consistent.\n\nQUESTIONS:\nClarification on chunk minimality: I'm curious if the discovered chunks are minimal, in the sense of being irreducible functional units. E.g., could the chunk for 'cheesecake' be decomposable into smaller sub-chunks ('cheese' and 'cake') with independent causal effects? \n\nJustification for PA methodology: the PA method assumes averaging hidden states across multiple contexts yields a meaningful, general 'chunk' for the concept. Could the authors maybe provide some evidence or justification for this assumption, especially given the context-sensitivity of internal representations in transformers? \n\nQuantitative evaluations: the paper presents compelling qualitative demonstrations to implicate chunks in the functional behaviour of larger models, but offers limited large-scale quantitative evaluation of chunk causal efficacy. Have the authors considered more systematic evaluation strategies - e.g., measuring the consistency of chunk-induced output changes across a large, diverse dataset? \n\nUnsupervised setting and chunk evaluation: in this setting, how do the authors determine whether learned chunks are semantically distinct or functionally redundant? Related to this, have the authors thought about any systematic way to assess the 'quality' of a chunk? Or what determines a 'good' chunk, e.g., semantically distinctness, functionally consistency, minimality?\n\nLIMITATIONS:\nI would like to see potentially some discussion on the context-agnostic assumptions underlying the PA and UCD approaches, or a broader discussion about this in the supplementary, perhaps.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for interpreting neural network computations through the cognitive lens of chunking, proposing the Reflection Hypothesis: well-trained neural networks exhibit internal neural population patterns that mirror the regularities in their training data. The authors develop three complementary methods—Discrete Sequence Chunking (DSC), Neural Population Averaging (PA), and Unsupervised Chunk Discovery (UCD)—to extract recurring, interpretable neural activity patterns (chunks) from both small RNNs and large language models (LLMs). These chunks capture concrete or abstract concepts present in input sequences. The authors empirically validate the hypothesis through comprehensive experiments spanning synthetic RNN tasks and modern LLMs like LLaMA-3, showing that chunk activations robustly align with recurring words and abstract narrative schemas. Grafting and freezing these chunks causally affect the models’ predictions, offering mechanistic interpretability grounded in cognitively inspired principles.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The Reflection Hypothesis reframes interpretability by connecting neural population trajectories to human-like chunking, a fresh perspective beyond token-level or neuron-level probes.\n\n- The three proposed chunk extraction techniques effectively handle a wide range of settings, from low-dimensional RNNs to high-dimensional LLMs.\n\n- The paper presents extensive experiments demonstrating chunk consistency across architectures, scalability to LLMs, and the causal influence of extracted chunks on model behavior.\n\n- The ability to graft or freeze identified chunks and observe systematic changes in model output provides compelling evidence of interpretability.\n\n- Validation on diverse architectures (RNNs, transformers, state-space models) supports the robustness and applicability of the proposed methods.\n\nWeaknesses:\n\n- The paper could better contextualize how its framework integrates with or contrasts against recent neuron-centric interpretability methods like SAE circuits or representation engineering.\n\n- Some details on the computational cost of applying chunk extraction at scale in large models are missing.\n\n- While synthetic and narrative-driven datasets are useful, real-world downstream tasks (e.g., factuality or bias assessment) would further strengthen practical significance.\n\nQUESTIONS:\n1. How do the authors envision chunk discovery scaling to production-sized LLMs (e.g., 70B+) given the need for large embedding datasets and iterative dictionary learning?\n\n2. Could chunk-based interpretability complement or enhance existing sparse autoencoder approaches or linear probe techniques? Any preliminary experiments combining them?\n\n3. Have the authors explored whether the extracted chunks remain stable or shift when the same base model is finetuned on different tasks? This could reveal insights into concept drift during finetuning.\n\n4. Beyond the included narrative schema experiments, could future work assess whether chunk activations correlate with human attention or comprehension on tasks like QA or summarization?\n\nLIMITATIONS:\nYes. The authors clearly acknowledge the potential dependency on training data structures, the challenges in chunking highly dynamic contexts, and limitations of relying solely on chunk-based explanations for interpreting complex behaviors. They also highlight promising directions for future research to formalize theoretical underpinnings, which aligns with best practices in interpretability research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}