Concept-Guided Interpretability via Neural Chunking
Authors: Shuchen Wu
Stephan Alaniz
Shyamgopal Karthik
Peter Dayan
Eric Schulz
Zeynep Akata
Venue: NeurIPS 2025 poster ()
Keywords: interpretability
LLM
cognitive science
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A cognition grounded interpretability method

Abstract: Neural networks are often described as 
black boxes, reflecting the significant challenge of understanding their internal workings and interactions. We propose a different perspective that challenges the prevailing view: rather than being inscrutable, neural networks exhibit patterns in their raw population activity that mirror regularities in the training data. We refer to this as the \textit{Reflection Hypothesis} and provide evidence for this phenomenon in both simple recurrent neural networks (RNNs) and complex large language models (LLMs).
Building on this insight, we propose to leverage cognitively-inspired methods of \textit{chunking} to segment high-dimensional neural population dynamics into interpretable units that reflect underlying concepts.
We propose three methods to extract these emerging entities, complementing each other based on label availability and neural data dimensionality. Discrete sequence chunking (DSC) creates a dictionary of entities in a lower-dimensional neural space; population averaging (PA) extracts recurring entities that correspond to known labels; and unsupervised chunk discovery (UCD) can be used when labels are absent. 
We demonstrate the effectiveness of these methods in extracting entities across varying model sizes, ranging from inducing compositionality in RNNs to uncovering recurring neural population states in large language models with diverse architectures, and illustrate their advantage to other interpretability methods. 
Throughout, we observe a robust correspondence between the extracted entities and concrete or abstract concepts in the sequence. Artificially inducing the extracted entities in neural populations effectively alters the network's generation of associated concepts.
Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data to reveal the hidden computations of complex learning systems, gradually transforming them from black boxes into systems we can begin to understand.

Implementation and code are publicly available at _https://github.com/swu32/Chunk-Interpretability_

Concept-Guided Interpretability via Neural Chunking
Shuchen Wu Stephan Alaniz', Shyamgopal Karthik',
Peter Dayan Eric Schulz', Zeynep Akata*
Allen Institute, UUiiversity of Washington, 'Télécom Paris, Institut Polytechnique de Paris,
IInttitute of Explainable Machine Learning, Helmholtz Munich
Department of Computational Neuroscience,
Max Planck Institute for Biological Cybernetics
Institute for Human-Centered AI, Helmholtz Munich shuchen. wwwalleeinntaaaaneeeemmmmn
org
Abstract
Neural networks are often described as black boxes, reflecting the significant challenge of understanding their internal workings and interactions. We propose a different perspective that challenges the prevailing view: rather than being in- scrutable, neural networks exhibit patterns in their mirror regularities in the training data. We refer to this as the Reflection Hypoth- esis and provide evidence for this phenomenon in both simple recurrent neural networks (RNNs) and complex large language models (LLMs). Building on this insight, propose leverage
population activity that
cognitive tendency of chunking segment high-dimensional neural population dynamics into interpretable units that reflect underlying concepts. We propose three methods extract recurring chunks on a neural population level, complementing each other based on label availabil- ity and neural data dimensionality. Discrete sequence chunking (DSC) learns a dictionary of entities in a lower-dimensional neural space; population averaging (PA) extracts recurring entities that correspond to known labels; and unsupervised chunk discovery (UCD) can be used when labels are absent. We demonstrate the effectiveness of these methods in extracting concept-encoding entities agnostic to model architectures. These concepts can be both concrete (words), abstract (POS tags), or structural (narrative schema). Additionally, we show that extracted chunks play causal role in network behavior, as grafting them leads to con- trolled and predictable changes in the model's behavior. Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data to reveal the hidden computations of complex learning systems, gradually transforming them from black boxes into systems we can begin to understand. Implementation and code are publicly available at btttps///githue com hwtmm//ww/////m Chunk- Interpretability
Introduction
Neural networks known as "black boxes" 88
7
93
56
25
highlighting the inherent
challenge to make sense of a vast number of computing units. Their computations are performed by up to billions of interacting components, in stark contrast to the traditional scientific practice of modeling phenomena through a number of well-defined symbolic entities in physics or mathematics.
A substantial portion of the interpretability challenge is cognitive [1 60) and demands asking the question: what makes high-dimensional data meaningful for cognition? And what constitutes the basic units and entities for perception that underlie the constitution of the symbols that we are adept at studying in physics or math?
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We present a novel perspective on interpreting artificial neural networks by referencing how cognitive entities arise from sensory data. Similar the vast neural activities, perceptual data that flood into our sensory stream is similarly high-dimensional and complex. However, there is an inherent tendency for cognition to rapidly discern recurring patterns in perceptual sequences as chunks and entities (991 |
process that even infants do with minimal exposure [75 76 3 Our perceptual and
cognitive system adeptly organizes the the overwhelming flow of the sensory stream into a structured
representation of entities, relations, and events over time [37 35 23
26
46
[4 89 88).
Psychologists have long postulated that identifying patterns in perception serves as an inverse model to uncover structured reality. in which recurring elements often reflect shared causal mechanisms [64] 77] We termed this the "reflection hypothesis" " and examined if the neural network's 's hidden activity reflects the regularities in the data it learns to predict, first in RNNs trained to predict sequences with known regularities, and then in large language models. Evidence that structured information exists in high-dimensional neural data suggests the possibility of leveraging our cognitive tendency to chunk information and discern recurring patterns and entities directly from neural population activities. Hence, we developed several methods to extract these recurring entities from low to high dimensions of neural population activities, complementing each other in their usage cases. We show the advantage of this simple method over popular interpretability approaches. Our findings suggest novel interpretability framework that leverages the cognitive principle of chunking to find emerging entities within artificial neural activities, paving the way for future work that decomposes network computation into interactions and relations between symbolic entities.
2
Related Work
Most interpretability approaches hold salient agreement on what is interpretable and what to interpret. What interpretable influenced by models in physics and mathematics, where operations and derivations are framed around the manipulation of small set of well-defined symbols. Hence, interpretable concepts are confined to word-level or token-level description, and methods try to learn
mapping between the neural activities and the target interpretable concept 33 94
9
10
69 20).
Current methods on what to interpret to understand the computations inside a neural network are heavily influenced by neuroscience: either on the level of neurons as a computation unit or in low-dimensional neural activity descriptions. The carliest interpretability approaches, inspired by discoveries of "grandmother cells"" and "Jennifer Aniston neurons" inside the brain, focused on understanding the semantic meanings that drive the activity of individual neurons. Similarly, studies in ANNs, from BERT to GPT, have identified specific neurons and attention heads whose
43 Sparse autoencoders (SAEs)can be seen an intermediate step that encourage the hidden neurons to be more 15 Thereby, an autoencoder maps neural activities a hidden unit layer to much larger number of intermediate hidden units while encouraging small number of them to be active. This way, the activity of the target hidden layer can be represented as a superposition of several individual neurons within the SAE. Other approaches such as representation engineering captures the distinct neural activity corresponding to the target concept or function, such as bias or truthfulness 194). Then, it uses linear model to identify the neural activity direction that predicts the concept under question or interferes with the network behavior.
activations correlate with semantic meanings in the data [68] 25
85 57
36 65 63
72
monosemantic 22
The current interpretability approaches that study language- based descriptions conceptual entities in terms of individual neurons or low-dimensional projections suffer from limitations on both ends: meanings are finite, and individual neurons are limited in their expressiveness and may not map crisply to these predefined conceptual meanings. Just like the failure in physics to have a closed- form description of motion beyond two interacting bodies [79]. confined, symbolic definitions of interpretation have inherent limitations in precision. This cognitive constraint, i.c., our reliance on well-defined symbolic entities for understanding, has made deciphering the complexity of billions of neural activities an especially daunting task. It underscores a fundamental trade-off between the expressiveness of model and our ability to understand it [86].
Focusing solely on individual neurons also insufficient to capture the broader mechanisms under- lying neural activity across network. Monosemantic neurons, which respond to single concept, (5]. Empirically, especially for transformer models (25), neurons are often observed to be polysemantic, i.c. associated
make up only a small fraction of the overall neural population 722 25

---

with multiple, unrelated concepts 663 25 67 which complicates the task of understanding how
neural population activity evolves across layers 2 39. This highlights the need for more holistic approaches that account for the complex, distributed nature of neural representations.
3 Chunks from Neural Embeddings (CNE)
We effortlessly perceive high-dimensional perceptual signals by segmenting them into recurring, meaningful patterns. i.e. chunks. These patterns reside in subset of our perceptual dimensions as cohesive wholes [58 48 37 35]. We employ this strategy to simplify the complexity of naturalistic data. As naturalistic data possess compositional structure and contain rich regularities, isolating recurring entities as concepts is an effective way for an agent to decompose their observations into entities and their relations 89 87
Similar to humans, Al systems aim to understand reality from naturalistic data. The regularities in naturalistic data may drive converging representations in diverse AI models. Recent findings suggests that neural networks of different architectures, scales, and sizes learn representations that are remarkably similar 61] even when trained on different data sources [52). This phenomenon is more pronounced in larger, more robust models [6] 21 47 [4),, and across multiple data modalities [52 61 47 74]. Recently, it has been hypothesized that such convergences reflect underlying statistical model of reality (28).).
The Reflection Hypothesis: We posit that the underlying cause of these observations is that, like human cognition, neural networks reflect the structures and regularities present their training data through their internal computations. Specifically, we hypothesize that well-trained neural network should exhibit trajectories of neural activity that mirror the structure of the data.
Formally, denote the sequence as S = (s², ",.,.,,,."""" s"), indexed by I = f1, ... m.). A pattern the input is defined as a recurring subsequence carrying the same concept P (s', s ..... s'....' s** S S of fixed or variable length k. A neural network with set of neu- rons W ([W)= = d) exerts sequence of population activities S₁ = (h',h² hh. . hhh) in response to the input sequence. Each neural population vector has embedding dimension d: h Rdd
The Reflection Hypothesis posits that such recurring input patterns P are mirrored within the internal neural dynamics of the model, such that their corresponding neural activations (h', h ..... h*++=1 lie within a population-level chunk defined as ball B(b A) C Rd centered at prototypical activation vector he with radius A. Regularities in the input sequence reflected as localized activity patterns in subdimensions neural population dynamics.
If this hypothesis is even partially true, it suggests that neural network activity can be decomposed into recurring "chunks," analogous how human perception segments nigh-dimensiona visual input into meaningful units. Such chunks, elicited by activities of neural subpopulations, may represent distinct computations and serve as interpretable entities. Extracting these entities allows a reduction of the highly complex neural computation to the emergence of these chunks. We develop architecturally agnostic methods to extract chunks and evaluate their efficacy concept encoding, interpretabilityy and their causal role in altering network behavior.
3.1 Extracting Entities
Depending the dimensionality d and the nature of the problem, we develop three chunk extraction methods: discrete sequence chunking for small d, neural population averaging for large d and when there is an identifiable pattern in S, and unsupervised chunk discovery when the supervising concept is unavailable.
Discrete Sequence Chunking (DSC) When d is relatively small, we can use cognitive- inssireed method to extract recurring patterns 887, which is also used for text compression B1 92
2 or tokenization. The idea is to cluster the neuron activations individually, use the cluster indices to represent the network activity as strings, and then further group frequently occurring string sequences into chunks. For this purpose, we convert each hidden neural activity vector h E Rd into string of discrete integers, and transform the sequence of vectors S₁ into one-dimensional sequence of (L' (2,, C.,,... 1²). Each 1 has length d and contains the nearest cluster index for each of the
strings S*
3

---

Chunks from Noural Embeddinge (CNE)
a.
b.
Population Activiy S (b..)
Vocabulary
Biological Mind
Continusus
Aphaaata
HIT
Grammat
Symbolic Popuislion Aptivly
VP
Verb
Del.
Youn
Discnatu I
Vocabulary
Artificial Mind
Input Sequence Predicton Based on Symbolic Population Activly
Churks
Sequence
Figure Naturalistic data are highly redundant and compositional, e.g. in language sequences, cognitive systems learn dictionary of recurring concrete and abstract entities. b c. In simple networks that contain a small number of neurons, chunking methods can be used to learn a dictionary of frequently recurring population trajectories. This discrete representation can reliably predict the input in the sequence and the network's predictions on the next character.
neurons, i.e., concat(M((h)) the closest cluster index to the activity of the neuron.
M a clustering function that assigns
We then apply chunking method on S extract vocabulary of patterns of string combinations, which represent patterns in neural state trajectories. The vocabulary of chunks is initialized with the set of unique strings in S This vocabulary grows iteratively by merging the top N frequent adjacent chunk pairs above threshold. Within each new iteration, the updated vocabulary is used reparse the sequence, repeating until convergence (see [O). This identifies recurring neural population trajectories that segment the activity sequence. The resulting dictionary D can then parse S in terms extracted chunks.
Neural Population Averaging (PA) In larger- networks like transformers, can be very big. this case, identifying recurring patterns in the sequence allows us to uncover the internal structures of neural population chunks by averaging the relevant population activities. akin extracting task- induced neural response functions [17 [6]
We assume that there recurring activities of neural subpopulations, which account for the network's computation as elicited by recurring concept s in the sequence (such as particular word 'cheese') Denote the indices of the concept encoding chunks as C(s) W. i.e., subset of the neurons in among the whole neuron population W ([WI d). Denote the set of indices where the pattern appears in the input sequence (s)) T. From the Strong Law of Large Numbers (SLLN), the population mean of concept-encoding neural subpopulation activities ho(e) converges to the true mean as [V()) approaches infinity:
ho(2) = Ejev(,) bo())
and
lim (V()) -
how)
= μ (a.s.)
[V(s]
where f [B(b(()) is the true mean of the subpopulation neural activity.
Given training data containing the sequence S, the corresponding neural population activity Sr, and recurring pattern in S, we aim to estimate the signal-relevant chunk. This includes the mean neural subpopulation activity he(s) within the subpopulation neurons C(s) C W, as well as the range of
deviation A within which the neural activities ho(a)) fluctuate hc(s)-

---

To do this, we first compute the mean population response of signal s by averaging over the token- specific hidden state representations
Liev hag A neuron i is hypothesized to be inside C(s) [V
if its activity at the time of the concept-specific index fluctuates within pre-set tolerance level around the mean signal-relevant activity C(s) = {i E W [has ha1 s tol Vj E V+))
After estimating the neural subpopulation C(s) and h)(s)) we then calculate a maximal deviation acceptable in the training data:
hCG)) hc
max
(2)
JEV(a)
d
We define the deviation A as threshold for acceptable fluctuation around the mean neural subpopu- lation activity hc((s)- An unknown neural activity vector h is classified as belonging to the chunk ho(s) if it lies within the closed ball B centered at ho(s)) with radius A:
[lb))) - bc AA
h B(h A)
iff
d
Using this criterion, we identify instances of an unknown neural subpopulation vector hc(s) in the test data using the chunk identification function
{o: 1, if [ -eccell I A(s),
fchunk (e) bc(s): A(s)
(0, otherwise.
and evaluate its quality as a neural population activity classifier for the occurrence of the signal using true positive rate (TPR) and false positive rate (FPR).
Since the neural subpopulation C(s). the chunk hc(s). and the deviation threshold A depend on tolerance parameter, generate series of increasingly stringent tolerance thresholds: tol, 2 x 0.8², = 0, 1,...... 39. As A is tied to the tolerance threshold, we find the optimal tolerance parameter and the associated A that maximizes TPR while minimizing FPR the training data.
Unsupervised Chunk Discovery (UCD) While discrete sequence chunking applies to small d and population averaging hinges knowing the location of the recurring concepts, we also develop an unsupervised method to discover recurring patterns in the embedding space.
To this purpose, we formulate the chunk finding question as learning dictionary matrix D E Rxdd that includes the recurring chunks embedded in the latent space of dimension d.
Given the embedding data X E Mxd from one single layer of an LLM processing some tokenized input with batch size M and embedding dimension d, we optimize a loss function C(X, D) that encourages each embedding to match its most similar dictionary entry:
L
1 M
Σ
SIM(DR. X.))
(4)
(E. AE(L....KK
Here, SIM(d, x) denotes the normalized cosine similarity between and x: SIM = [a]la ix| The loss function encourages each embedding X,, to align closely with one of the chunks D
x
4 Results
4.1
Evaluating Reflection Hypothesis on Simple RNNs
First, we assessed the reflection hypothesis using simple recurrent neural network (Figure 2 and appendix(D.I D.I) trained on synthetic sequences containing known pattern. We begin with sequences in which ABCD appear periodically (Figure 2p). then gradually increase complexity: first by embedding ABCD sparsely within a background of repeated Es (Figure : and then by placing ABCD amid randomly occurring background symbols (E, F, G) to simulate noise (Figure 2t). In each case, the RNN trained to predict the next character. Across all conditions, the network's hidden states consistently exhibit activity patterns that reflect the pattern occurrences in the sequence.
5

---

a. Menory State E Input
Perturbed State Input
Perturbed State B
Perturbed State
b.
Ingut
Input
Transtor ('ABCD; GHI, UKLMNN > (ABCOLMN)
Model Prodicts:
Predicts:
Model Prodicts:
Predicts:
Petureton
14
0.95
68
Accuracy
66
54
0.85
0.80
Centrol
0.75
Control 95%
Perturb
Penturb 99%
20
40
Transfer Trial
Nurber of extracted chunks from DSC
d.
Number Extracted Chunks Neural Popu latiantion
Sequence Parsing Length
Sequenie Parsing Length
25000
1600
ground truth
600
500
2300
NChunk
2250
30
400
20
300
2000
400
200
200
1800
uncrained
100
20
20
30
Models
Models
Level of Hierarchy
Figure 2: a. Hidden states can grafted to causally change network memory and prediction. Embedding grafting enables faster transfer learning of a compositional vocabulary. When RNNs learn from sequences that contain context-dependent predictions, training creates extra chunks inside the embedding space. d. Left: The neural population trajectory can be parsed by bigger chunks, which translates to smaller sequence parsing length with an increasing level of hierarchy in the training sequence. The number of extracted chunks increases with the level of hierarchy inside sequences.
Understanding the neural state and its mapping to the input We apply Discrete Sequence Chunking to transform continuous hidden unit activities to a symbolic and discretized description. We denote population activities by the alternation of indices that mark the belonging cluster centroid of individual neurons and visualize the population activity by the indices of the assigned nearest cluster. This symbolic description of the neural activities allows decoding the trajectory of the hidden state and its corresponding input via look-up table (appendix [D.2). and reaches a perfect decoding then apply
accuracy the test set. From the symbolic description of neural trajectories, chunk discovery methods to learn a dictionary of symbolized chunks. This dictionary contains the maximally recurring patterns inside the network, which consistently reflect the patterns in the input.
Grafting PA chunks causally alter the network's behavior We then test whether the extracted chunks causally alter the network's output in predictable way. Starting from state predicting B from imput A (with prior input E), we replace the hidden state with those corresponding to A, B, or C, while pairing with imputs B, C, or D, reliably shifts the output predictably to C, D, E (Figure 2 a).
Grafting neural population activity in chunks facilitates compositional learning Humans learn complex sequences by reusing simpler parts [5]] 50). RNNs, by contrast, struggle with reuse and composing the new from the known (49] We test whether the population grafting may artificially induce the network to compose and reuse a previously learned vocabulary to compose new vocabulary. We trained two identical RNNs on synthetic training sequences with randomly sampled words from a dictionary (ABCD, GHI, JKLMN) occurring within null E characters.
We then further train the two RNNs in a transfer sequence with the word ABCDLMN a composite of two-word parts in the training vocabulary. Without perturbation, the control RNN suffers from performance degradation consistent with the observations in the literature. For the other RNN, we learn lookup table that maps its discrete population state to the concurrent input. As this RNN learns on the transfer sequence, whenever the input is D (the network state should be C), we graft the neural population state to the cluster centroid of the hidden state induced by the previous input being J paired with the input character being K, thereby forcing the network to generate the next prediction as I. instead of Shown in Figure 2 are the learning curves between the two networks during the transfer sequence. The perturbed RNN predicts the transfer sequence with higher accuracy than

---

a.
C. Repettion amonget random occumences
Posd Spet
Chancie
Populatton Adtivty
Population
lksden Stre
Clade
d.
Llama-3
Cheese
the versatile ingredients
the cultiaryy world,
cheese
Figure 3: The reflection hypothesis: ANNs' neural actívities can be parsed into entities that reflect the structured regularities training sequences. a-c. Population activity of the initial 5 neurons (unsorted) from an RNN trained to predict the next sequence character, from sequence that contains repeating ABCD patterns and another that contains ABCD embedding in default background noise (random E, F, and G); d. Raw neural activity of the first 50 neurons of LLaMA-3 (unsorted) across all layers (32) processing the prompt up to the last token of each highlighted word.
control, artificially grafting hidden states forcefully induces the RNN to reuse existing representations predict sequence composition.
Training creates additional population states that distinguishes contexts We further hypothe- sized that learning may create additional patterns of neural population activities inside the network to distinguish different contexts. We create sequences with vocabulary that contains context-dependee subsequence parts: vocabulary (CDAB, AB, ABCD) among default sequence of Es. In this sequence, the CD inside ABCD conveys different meaning than that inside CDAB. While the former predicts E shall follow, the latter predicts that AB shall follow. We used DSC to learn dictionary of chunks from an untrained and trained network separately, responding to the input sequence of a fixed length. We then use the learned chunk parse the neural trajectory and measure the length of the parsing. Shown in Figure c a trained RNN contains neural trajectory that can be parsed by a number of chunks similar untrained network. The trained network also processes neural trajectories that contain more diverse set chunks, suggesting that training creates additional neural trajectory patterns to distinguish context-specific meanings.
number of words in the sequence, and higher than the
Hierarchically structured input sequence corresponds to a more diverse number of chunks in neural activities. Another implication of the reflection hypothesis is that when the input sequence contains an inherent hierarchical structure, i.e., alphabets can become word parts, which can become words, etc., then neural network that learns predict the sequence should also exhibit more diverse set of population state to distinguish the ever-more complex context dependency given rise by the hierarchical structure. To test this, we generated synthetic sequences with hierarchical vocabulary following [87]. Starting with an alphabet (A, B, C, DJ/ and default null symbol E, we iteratively built new words by concatenating pairs from the existing vocabulary over steps. Word probabilities were sampled from a uniform Dirichlet distribution, scaled by 0.2, with the remaining 0.8 assigned to E. We sampled from this vocabulary to construct the training sequences. We then independently trained RNNs on sequences with an increasing number of hierarchical depth and extracted neural population chunks using DSC, As shown Figure the number of extracted neural chunks grew with hierarchy level/vocabulary size. Additionally, as the structure of the input becomes more nested, population trajectories could be parsed into increasingly larger chunks of neural trajectories, indicating that the network internalized the nested structure.
4.2 Reflection Hypothesis Evaluated on Large Language Models
We next test the reflection hypothesis on larger models, and then use the population averaging and unsupervised chunk learning method to extract recurring entities that may reflect the network's
7

---

Token
Cake
cale
Cake
Grafting "Hello, how are you doing?
Decoding Performance Across Layers (Test)
0.000
Cheesecake dll
-0.025
PA
recipe very orcning chenseeake
giing you
you
10
25
checaeoaee
Tuses ont sae icheese Cae
Decoding Performance Across Layers (Test)
Cheese
Layer
choose chasetske
cake
Target Concept
Without Grafting
0%
1%
With Grafting
90%
K3%
0%
16%
Freezing
0.00
"Nane three
tarta, cakax and
most
-0.01
dessert
Europe
Cake
beitls, tirowisu
15
20
30
"What is the name of rich, savory food made fron curdied often aged to flavors?
Layer
Tokan Dle ecats Iihs acate
Cheesecake
Decoding Performance Across Layers (Test)
2
stop Italon, French, Spantes, Greek
outiine It's called "Ricoetu" in Italian "Barrata" Itailan and Frwnch. "Queen Blanca" Spantsh, and
0.00
SAE
"eeineal" In Groek.
-0.02
Target Cuncept
No Froczing
Freszing
choose 67%
335
15
20
cale
36%
9%
Layer
Figure 4: The identifiability of the presence of extracted chunks evaluated by signal detection measures, and comparison between SAE. Grafting and freezing word-related population chunks alter the network's sequence generation.
representation of concepts. We started with LLaMA3-8B 22 due to its open-source access and architectural complexity. I.LaMA3 predicts the next token based on the ever-expanding preceding context; hence, it analyzes LLaMA3's hidden states as it incrementally processes tokenized sequences, from the beginning of each sentence to the latest rath token S = s?, ...,.") s"). We recorded the hidden state sequence across all layers Sa (h2 h2 hi,.., h²) to track the evolution of the model's internal activity as new tokens are included. We then examine the existence of neural chunks applying the Neural Population Averaging and Unsupervised Chunk Discovery.
Recurring concepts in a sentence Figure shows normalized activations of the first 50 neurons (unsorted) across all LLaMA3's layers for the word "cheese". Consistent with the reflection hypothe- sis, the two differently tokenized occurrences of "cheese" evoke similar neural activity patterns, more so than a different word like "ingredient," suggesting the presence of common neural subpopulation activities encoding recurring concepts. Using recurring words language as the pattern of interest we applied PA identify the neural subpopulation C'(s), the mean chunk activity hc(s), and the fluctuation radius (ss) the last token of each word bearing a variety of tokenization forms.
We assessed the extracted chunks and thresholds by measuring how well chunk detection predicted the occurrence of the word concept Figure 4 shows that the population averaging method yielded neural
LLaWW-3
Mamba
Space Model)
g
2011 19 twie 20 oe of ss
(Encoder-decoder architecture)
(Large-scale
13
8:
0.0
5-14
Figure 5: Average decoding performance of the top 100 words in English, applying population averaging variety of models. Error bar denotes the standard error of the mean.
8

---

subpopulation chunks that are predictive of the example words "cheese". "cake", and "cheesecake" in each layer and distinct from one another, with low false positives and high true positives across all layers throughout. Additionally, we compared PA with pretrained sparse autoencoder (SAE) on L-LaM-33 (24), using the most active concept-related neurons as decoders. Similar to PA, we selected the activation thresholds of concept encoding neurons from training data and applied them to test data. Overall, PA-extracted population activity predicts the presence of the selected concept better than the individual concept neurons identified by SAE.
We generalized our method for identifying concept-encoding neural population chunks to other large-scale models with distinct architectures and sequence- proocessiiggcnnggggggge mechanisms, including the encoder-decoder model T5 (t5-small [66]), the RNN-based RWKV (rwky-4-166) 1--6m-mmppl)e (70), and the
in the SI for decoding the example concepts). We evaluated chunk quality by testing whether concepts could be predicted from chunk detections on held-out neural recordings corresponding to test prompts. Beyond the illustrated concepts, we extended the concepts to the top 100 most frequent English words. Overall, we found that PA consistently extracts chunks whose activations correspond to meaningful concepts across diverse architectures. The presence of a chunk reliably predicts the presence of its associated concept in the text with minimal false positives.
state-spat space model Mamba (mamba- 1 330mhff 33;; see Figures 24 25
26
Chunks Encoding Abstract Schema and Narrative Structures We evaluated PA's ability to find chunks encoding more abstract narrative structure by training on 20 stories following schema (e.g., visit food location + buy item - cat react) and 13 control stories with narrative structure inconsistent with the schema (examples shown in Figure 6 We extracted shared subpopulation chunks at the end-of-sentence token using PA, and tested on 18 new schema-consister and 15 inconsistent stories. Schema-related chunks are activated at sentence boundaries in structured narratives but not in controls, suggesting that PA captures abstract narrative schemas beyond surface- level concepts or syntax.
Grafting and Freezing Neural Populations We tested the causal roles of these extracted signal- relevant neural subpopulation activities hc(s) by grafting neurons to discovered chunks and gen- erating the subsequent text with LLaMA-3. To do this, fed in a prompt and graft the neural subpopulation C(ss) (s) a specific token position of the prompt. Shown in Figure 4 perturbing the hidden units to embeddings corresponding to words such as "cheese", "cake", or "cheesecake" biases the network towards generating sequences related to the grafted topic. Conversely, we also experimented with freezing the words by setting the corresponding neural subpopulation of chunk support set to zero hc = and used prompts that lure the network to generate the frozen word concept. As shown in Figure 4 freezing the chunk related to topic can cause the model to avoid using the concept- speecffcc word.
We then evaluate the general effectiveness of grafting general concepts on the ROCStories benchmark 622, sampling 2,000 sentences extract neural population chunks corresponding the top 20 frequent concept words (filtered to avoid short or overlapping terms) in addition to the existing concepts collected. To assess context-dependent effects of prompt used when grafting the concept- encoding neural chunks, we used prompts from the TREC Question Classification dataset [[33), which includes six gaausee context categories. For each category, we sampled 50 prompts and evaluated grafting success by measuring the difference in concept occurrence probability between grafted and ungrafted (control) LLaMA-3 generations. To reduce computational cost, we categorized grafting into early (layers 1-9), middle (10-19), and late (20-29) interventions. Results are shown Table [ We found that grafting consistently increased concept occurrence across contexts, with early-layer grafting being the most effective.
Schema
Control
frod
Purchoa
chocolete muffin, walkoud to the nearbay
avent the bakery, bought
took a bite of the survito I broughtt homo, and caid ni to everyone. Trainin Tost
Into
park, bits, smiled.
to coffee shop, ordered latto, sat down, took dip, and loved
Enterousued infrowers, couch, and screlled through my phone....
Roaet perohased
location
Innn
Layer
Figure 6: Example sentences consistent with one narrative structure, and the control sentences which are inconsistent in training and test sets. Decoding accuracy of narrative-consisten structures in the test set.

---

Table 1: Effect of grafting on TREC categories (percentages).
TREC Category
No Graf.
Early Graf.
Middle Graf.
Late Graf.
ABBR (Abbreviations, acronyms)
14.9%
55.9%
30.8%
18.0%
DESC (Descriptions, definitions) 15.6%
49.0%
28.1%
20.7%
ENTY (Entities)
12.6%
48.1%
22.5%
16.9%
HUM
(Human-related)
11.9%
46.7%
21.5%
15.2%
LOC (Locations)
10.7%
47.5%
20.5%
14.4%
NUM (Numeric answers)
11.5%
45.3%
21.8%
16.0%
Unsupervised Chunk Discovery We investigate whether recurring chunks in the neural embedding space can be identified in an unsupervised manner, when labels in text are unavailable. To this end, we train a chunk dictionary D LLaMA3's hidden activity while processing sentence- wise prompts derived from Emma by Jane Austen, sourced from the Project Gutenberg corpus [40], accessed via NLTK [. As this bigger embedding dataset, we then trained D (K 2000, 4096) by minimizing the similarity loss function formulated in 3.1 individually for each layer of LLaMA-3.
can simplify high- dimensional embeddings by represent- ing them the appearance and disap- pearance of identifiable chunks. This enables visualization of the interac- tions between input tokens and the activations of these recurring enti- ties. Figure 7 illustrates chunk in- teractions across LLaMA's layers dur- ing early text processing. Recurring chunk patterns emerge. For example, commas and adjectives show similar activations in early layers (Figure [p)) but their representations later elicit distinct chunks in deeper layers. The full plot can be found shown in SI We also interpret the unsuper- vised learned chunks by correlating them with part-of-speech (POS) tags,
This way,
Mood
Teuse
depacaio
seneed
GRAT
utt
sare
se
n
Laper
3
F5
Figure 7: Upper: Describing the network's processing of tokens as the succession of unsupervised learned chunks. Lower: Layer-wise processing of similar token types.
the result be found in S
Conclusion, Discussion
We propose the Reflection Hypothesis, which posits that neural population activity reflects the structure of the data. We provide evidence in support of this hypothesis and, building on this insight, leverage the cognitive principle of chunking isolate perceptual entities units of interpretation. Using the methods introduce, we identify chunks and test their feasibility across models ranging from small-scale RNNs to number of large-scale LLMs. We find that these chunks correlate strongly with both concrete and abstract concepts spanning recurring words, sentence structure, and POS tags. These results suggest that chunking provides a powerful lens for segmenting high- ditheeiinnii neural activity into structured trajectories. This work invites future research to formalize its theoretical foundations, investigate the learning dynamics that give rise such entities, and apply this perspective to build a mechanistic understanding of the inner workings of complex learning systems.
Acknowledgment
This work was partially funded by the ERC (853489 DEXIM) and Alfried Krupp von Bohlen und Halbach Foundation, for which we thank them for their generous support. Shuchen Wu was in part supported by the Shanahan Family Foundation Fellowship the Interface of Data and Neuroscience at the Allen Institute and the University of Washington. Stephan Alaniz is in part supported by Hi!
10

---

PARIS and ANR/France 2030 program (ANR-23-IACL-000) Peter Dayan is funded by the Max Planck Society and the Humboldt Foundation. Shyamgopal Karthik thanks the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support.
References
[1] Amina Adadi and Mohammed Berrada. Peeking inside the black-hox: A survey explainable artificial intelligence (xai). IEEE Access, 6:52138-522 52160, 2018.
[2] Rakesh Agrawal and Ramakrishnan Srikant. Mining sequential patterns. Proceedings of the Eleventh International Conference Data Engineering, pages 3-14. IEEE, 1995.
[3] Richard N Aslin, Jenny Saffran, and Elissa L Newport. Computation of conditional probability statistics by 8- 8-month-old infants. Psychological science, ((4):321324, 1998.
[4] Judith Avrahami and Yaakov Kareev. The emergence of events. Cognition, 53(3):239-261, 1994.
[5] Randall Balestriero and richard baraniuk. A spline theory of deep learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 374-383. PMLR, 10-15 Jul 2018.
[6] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages
225-236. Curran Associates, Inc., 2021.
[7] Roger Barker. The stream of behavior: Explorations of structure & content. Appleton- Century-Crofis, 1963.
[8] David Bau, Jun-Yan Zhu, al. Understanding the role of individual neurons in deep learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[9] Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1)2007-219 March 2022.
[10] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt, Eliciting latent predictions from transformers with the tuned lens. arXiv preprint, arXiv:2303.08112 2023.
[11] Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python.
O' RRillyy Media, 2009.
[12] Sydney Black, Luke Sharkey, Leo Grinsztajn, Elliot Winsor, Daniel Braun, Jack Merizian, Karina Parker, Carlos Rueda Guevara, Blair Millidge, Ghadi Alfour, and Christopher Leahy. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312. 2022.
[13] Michael R Brent. An efficient, probabillstically sound algorithm for segmentation and word discovery. Machine Learning, 34:71-105, 1999.
[14] W. G. Chase and H. A. Simon. Perception in chess. Cognitive Psychology, 4(1)(55-81), 1973.
[15] Maheep Chaudhary and Attieus Geiger. Evaluating open-source sparse autoencoders on disen- tangling factual knowledge in gpt-2 small, 2024.
[16] Mark M. Churchland, John P. Cunningham, Matthew T. Kaufman, Justin D. Foster, Paul Nuyujukian, Stephen Ryu, and Krishna V. Shenoy. Neural population dynamics during reaching. Nature, 487(7405):51-56 Jul 2012.
[17] Marlene Cohen and Adam Kohn. Measuring and interpreting neuronal correlations. Nature Neuroscience, 14(7):811-81) Jun 2011.

---

[18] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features language models, 2023.
[19] D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493-8502, 2022.
[20] Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. arXiv preprint, arXiv:2303.09443. 2023.
[21] Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta neurons: Mining the common units a model zoo, 2023.
[22] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.21783, 2024.
[23] Dennis E. Egan and Barry J. Schwartz. Chunking in recall of symbolic drawings. Memory & Cognition, 7(2), 1979.
[24] EleutherAI. Sparsify: Interpretable sparse autoencoders for language models. https:/ github. Catiuu...om/Eleetttti www..///eeeeeeerr/arrrrr..t. 2024. Accessed: 2025-05-14.
[25] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Hatdeeld-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[26] Nick C. Ellis. Sequencing in SLA: Phonological memory, chunking, and points of order. Studies in Second Language Acquisition, 18(1), 1996.
[27] Andreas K. Engel, Pascal Fries, and Wolf Singer. Dynamic predictions: Oscillations and synchrony in top-down processing. Nature Reviews Neuroscience, 2(10):004-766, 2001.
[28] Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensyynally linear. arXiv preprint arXiv:2405. /4860, 2024.
[29] Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models, 2024.
[30] Thomas Fel, Victor Boutin, Louise Béthune, Rémi Cadène, Mahdi Moayeri, Louis Andéol, Mikael Chalvidal, and Thomas Serre. A holistic approach unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing 35:tems, 54818, 2023.
[31] Philip Gage. A new algorithm for data compression. Users Journal, 1994.
[32] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093, 2024.
[33] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space, October 2022. arXiv:2203.1 14680 [cs].
[34] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations. Advances in Neural Information Processing Systems, 32, 2019.
[35] Fernand Gobet, Peter C.R. Lane, Steve Croker, Peter C.H. Cheng, Gary Jones, Iain Oliver, and Julian M. Pine. Chunking mechanisms in human learning. Trends in Cognitive Sciences, 5(6).
2001.
[36] Gabriel Goh et al. Multimodal neurons in artificial neural networks. Distill, 2021.
12

---

[37] Ann M. Graybiel. The basal ganglia and chunking of action repertoires. In Neurobiology of Learning and Memory, volume 70, pages 1-2, 1998.
[38] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprini urXiv:2312.00752 2023..
[39] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitri Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023.
[40] Michael S. Hart. Project gutenberg. projectgutenberg.orgg 1971.
[41] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis, 2024.
[42] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages
2019.
[43] Andrej Karpathy. The unreasonable effettvveess of recurrent neural networks. Blog post, May 21, 2015, 2015. https::/kkarpathh:
[44] Adam Karvonen, Can Rager, Samuel Marks, and Neel Nanda. Evaluating sparse autoencoders on targeted concept erasure tasks, 2024.
[45] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, and Fernanda Viegas. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International Conference on Machine Learning, pages 2668 2677. PMLR, 2018.
[46] Iring Koeh and Joachim Hoffmann. Patterns, chunks, and hierarchies in serial reaction-ting time tasks. Psychological Research Psychologische Forschung, 63(1):22-33) 35, March 2000.
[47] Simon Kornblith, Mohammad Norouzi, Honglak Lee,, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 3519-3529. PMLR, 09-15 Jun 2019.
[48] John Laird, Paul S Rosenbloom, and Allen Newell. Towards chunking as a general learning mechanism. In AAAI, pages 188-192 1984.
[49] Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to- seguencc-to-sequenee recurrent networks. In International conference on machine learning, pages 2873-2882 PMLR, 2018.
[50] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[51] Karl Spencer Lashley et al. The problem of serial order in behavior, volume 21. Bobbs-Merrill Oxford, 1951.
[52] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence, 2015.
[53] Xin Li and Dan Roth. Learning question classifiers. In Proceedings of the 19th International Conference on Computational Linguistics (COLING 2002), pages 1-7, 2002.
[54] Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv:1312.5668 2013.
[55] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313-330, 1993.
[56] Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, March 2024. MrXiv:2403.19644 [cs].
13

---

[57] Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs language models, 2024.
[58] George A. Miller. The magical number seven, plus minus two: some limits on our capacity for processing information. Psychological Review, 1956.
[59] Joseph Miller and Clement Neo. We found neuron in gpt-2. Al Alignment Forum, February 2023. Linkpost from https::/ atttps://clementner com
[60] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences, 2018.
[61] Luca Moschella, Valentino Maiorea, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication, 2023.
[62] Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of NAACL-HLT 2016, pages 839-849, Association for Computational Linguistics, 2016.
[63] Jesse Mu and Jacob Andreas. Compositional explanations of neurons, 2021.
[64] Allen Newell, Herbert Alexander Simon, et al. Human problem solving. Prentice-hall Englewood Cliffs, NJ, 1972.
[65] Anh Nguyen, Alexey Dosovitskiy, et al. Synthesizing the prefemed inputs for neurons in neural networks via deep generator networks. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[66] Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2 10X.x:2/00.0887,, 2021.
[67] C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. An overview of early vision in inceptionvl. Distill, 2020. https://distill.pubbbb pub/2020 circuitsrrttt//// earll-
[68] Chris Olah, Arvind Satyanarayan, Shan Carter, et al. Zoom in: An interpretability method for deep neural networks. Distill, 2020.
[69] Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. arXiv preprint, arXiv:23 1.004977 2023.
[70] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305. 2023.
[71] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models, September 2018. arXiv:1806.07427 [cs].
[72] Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discover- ing sentiment, 2017.
[73] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explain- ing the Predictions of Any Classifier, August 2016. 1rXiv:1602.04998 [cs].
[74] Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned repre- sentations. In Marina Meila and Tong Zhang, editors, Proceedings the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9030-9039. PMLR, 18 24 Jul 2021.
[75] Jenny R. Saffran, Richard N. Aslin, and Elissa Newport. Statistical Learning by 8-Month-Old Infants. Science, 274(5294): 1926-1928. December 1996.
14

---

[76] Jenny R Saffran, Elissa L Newport, Richard N Aslin, Rachel A Tunick, and Sandra Barrueco. In- cidental language learning: Listening (and learning) out of the corner of your ear. Psychological science, 802):101-105, 1997.
[77] Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu. and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models, 2025.
[78] Alex Tamkin, Moiz Taufeeque, and Noah Goodman. Codebook features: Sparse and discrete interpretability for neural networks, In International Conference on Machine Learning, pages PMLR, 2024.
[79] Terence Tao. E pluribus unum: From complexity, universality. Daedalus, 141(3):23-34, 07 2012.
[80] Adly Templeton. Word equations: Inherently interpretable sparse word embeddings through sparse coding. In Jasmijn Bastings, Yonatan Belinkov. Emmanuel Dupoux, Mario Giulianelli, Dicuwke Hupkes, Yuval Pinter, and Hassan Sajjad, editors. Proceedings of the Fourth Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 177-191, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[81] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024.
[82] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593 1601-4601,220 2019.
[83] Jan Vielhaben, Sebastian Bluecher, and Nils Strodthoff. Miltii-dimnniionll concept discovery (med): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2024. To appear.
[84] Elena Voita, Javier Ferrando. and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional, 2023.
[85] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022.
[86] Ruochen Wang, Si Si, Felix Yu, Dorothea Wiesmann, Cho-Jui Hsieh, and Inderjit Dhillon. Large language models are interpretable learners, 2024.
[87] Shuchen Wu, Noemi Elteto, Ishita Dasgupta, and Eric Schulz. Learning Structure from the Ground ap--iiirrchiical Representation Learning by Chunking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Process- ing Systems, volume 35, pages 36706-36721. Curran Associates, Inc., 2022.
[88] Shuchen Wu, Mirko Thalmann, and Eric Schulz. Two types of motifs enhance human recall and generalization of long sequences. Communications Psychology, 3(1):3, 2025.
[89] Shuchen Wu, Noémi Éltetō, Ishita Dasgupta, and Eric Schulz. Chunking as rational solution to the speed-accuraye trade-off in a scrial reaction time task. Scientific Reports, 13(1):7680, May 2023.
[90] Jeffrey M Zacks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds. Event perception: mind-brain perspective. Psychological bulletin, 133(2):273, 2007.
[91] Jeffrey Zacks and Barbara Tversky. Event structure in perception and conception. Psycho- logical bulletin, 127(1):3, 2001.
15

---

[92] Mohammed J. Zaki. Sequence mining in categorical domains: Incorporating constraints. In Proceedings of the Ninth International Conference on Information and Knowledge Management. pages 422 429. ACM, 2000.
[93] Quan-shi Zhang and Song-chun Zhu. Visual interpretability for deep learning: survey. Frontiers of Information Technology & Electronic Engineering, 19(1):27-39, January 2018.
[94] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendryeks. Representation Engineering: A Top-Down Approach to AI Transparency. October 2023. arXiv:231000100 [cs].
Supplementary Information (SI)
Implementation and code are publicly available Chunk- Interpetaarrrriityy
stttp:://itthub.ccommmee
A Extended Discussion on Related Work
A.1 Comparison with Other Interpretability Approaches
Our work is different approach from number of other interpretability methods. For example, Black et al. define polytopes as linear regions in input space and study their geometry [12). Vielhaben et al. define concepts as lincar subspaces in feature space, spanned by an unbounded number of basis vectors [83. Our definition of a chunk is compact and bounded subspace, hence it does not tolerate addition or scalar multiplication of some basis vectors. In fact we would expect a certain level of scalar multiplication may run into other chunks. We suggest that chunks themselves are interpretable.
Other approaches, such Tamkin et al., propose architectures that enforce sparsity and interpretability. In particular, they learn a codebook of features through codebook bottlenecks (78), which requires training new models with modified architectures. By contrast, our goal is interpret the raw activity of large models directly, independent of their training procedures or architectures, rather than optimizing a new model.
A.2 Comparison with SAES and Representation Engineering
Our method for grafting and freezing corresponds to steering and unlearning in SAE work [2 57 However, the chunk grafting is simpler and more direct than that of SAES. In SAEs, steering is typically achieved by clamping a feature to specific value- either through a constant multiplier or scaled value relative its maximum activation [81). This clamped feature then probabilistically influences the network's 's output during text generation.
However, SAE features often do not correspond clearly human- interaannttereeebble concepts. The mapping between features and words is indirect, requiring additional steps such as VocabProj or MaxAct to identify associated tokens 29). This makes steering both complex and fragile. Interpreting SAE features remains an open challenge, and developing effective steering methods still an active area of research 77
In contrast, PA offers more interpretable units by design. Chunks are derived from recurring patterns in the model's activations, grounded in actual input semantics. This enables more transparent and effective interventions, such as concept grafting or unlearning, without the need for specialized bigger corpus across multiple contexts, and
heuristics. We also evaluated our steering efficiency verified that the extracted chunks can be an efficient way of steering network behavior.
The other main difference between this approach and SAE is compute efficiency. The population averaging (PA) method is a post hoc analysis that requires no optimization or gradient computation. Neural activations are precomputed, and the only cost lies in computing prototype vectors and evaluating distances across small number of deviation thresholds opperaatonso that scale linearly with the number of tokens and the embedding dimension.
16

---

In contrast, sparse autoencoders (SAEs) require multiple forward and backward passes through both encoder and decoder networks, typically over many training epochs. This results in orders-of- magnitude greater computational cost. PA, by contrast, offers lightweight and accessible alternative for large-scale interpretabilityy research. The unsupervised chunk discovery method also remains computationally feasible. It runs efficiently on single GPU. Given precomputed neural activations, it can be applied broadly to various architectures and datasets.
This work also differs from representation engineering, which isolates neural activity associated with predefined concept or function, such as bias or truthfulness [94] Representation engineering typically fits linear model to identify activity directions that predict or intervene on the target concept. However, this approach assumes the concept of interest is known a priori. In contrast, we assign single chunk to summarize the neural activity of concept, enabling discovery without prior labels. In contrast to grafting with chunks, which directly overwrites neural activity with a given chunk, representation engineering applies a latent direction whose effect depends on a chosen magnitude. Determining this strength a nontrivial hyperparameter choice and not always straightforward.
A.3 Comparison with Concepted-baad Interpretability Literature on Vision Models
Our work different from concept-basede based interpretability methods for image classifiers. In [34], 30 and [45] concepts correspond to visual features in the image dataset (e.g. pixels of a wheel), and they study what part of the image influences the network's classification decision. In contrast, our work primarily studies network interpretability processing sequences of tokens. Our work examines how the internal structure of the network exhibits recurring patterns of entities. We reframe interpretability by connecting neural population trajectories to human chunking.
B Definition of Grafting and Freezing
Grafting refers to the targeted insertion of previously learned chunks into a neural network's internal computation. We modify its internal activations h at specified layer / and neural subset C by grafting the chunk he. <- her replace the original neural subpopulation activities.
Freezing refers to the operation of zeroing out the neural activations that support the chunk. This is im- plemented by replacing the network activation with stored chunk representations at the corresponding neurons. At a given layer I, we enforce: hc. = O.
C Pseudocode of Learning Chunks Using Discrete Sequence Chunking
Algorithm 1: LearnChunks
Input :K, threshohr,hol,, gymbolized_neurree states, n n_jter
Output: sstatttptarseeee vocab
1 vocab + unique seaeeeeeddeeuaaaeeeeeeeeeaaaaree neural_states).
2 null_state + GetMostFrequ reeemmooliressmmmollleeedddd
3 state_parse: + Parse state trajectory by individual units;
for n_iler do
5 ChunkCandidales +
MostCommon((K Counter(zip(( (state parsepparre--eee -1], state parse(1 :)))); merged dict + Merge(eL, CR) ChunkCandidates if (count(cL, CR) 2 freq_threshold and C₁ # null_sstaee eR # mull_state); wocabbupdate (merged, _dict;;;
7
VergeeOvelappingghhnke (hpppnnCChnns(ooaa)):
(vocab);
state_parss se, vocab t- PureehaeSSSllsmmmlllladeeoeonddd
10 return state__pttt parse, vocab;
17

---

Algorithm 2: Neural Population State Chunking
Input =lroiningsequence
Output peliputssaate :ssatsstapppree _prrse, vocab
1 Initialize RNN;
colddnn_stated sequence TrainRNN(sequencee)
3 symbollc aymbolic_hidden. states +
states);
state_parse, vocab LearnChunkslsymmboicc Jaiiden_activityy sequence);
D
RNN
D.1
Recurrent Neural Network (RNN) Architecture
We provide more details on the structure of RNN. The RNN used by this work can be described by the following equation:
Xx
hn Woh
bb
(5)
hn-
Xn
O₂
Weo
hn
+
bo
(6)
y/ = log (softmax(o:))
(7)
The hidden state is initialized all zeros ho O. Xn, by R|01 h b, E RR, Weh [dxx(d+(0)
W co E [(([)(
We implemented a simple RNN with 12 hidden units, consisting of two linear layers: one for updating the hidden state and another for generating output predictions. At each time step. the input vector concatenated with the previous hidden state and passed through these layers. The output is normalized using log softmax to produce probability distribution over the output classes. The RNN is optimized with cross-entropy loss using Adam (learning rate = 0.005). Training is conducted on random subsequences of length 200 per batch, with the hidden state initialized zero the start of training. We track hidden states for analysis.
For the experiment in Figure of the main paper, the training sequence alternates between pattern ("ABCD") and noise ("E") repeated & ~ (1(1, 20) times. The RNN processes one-hot inputs from vocabulary, with inputs concatenated to hidden states and passed through two linear layers to produce log- bog-probabilities. The model was trained for 160 iterations using Adam (Ir = 0.005), with each iteration sampling 200-character sequence for next-character prediction via cross-entropy loss.
To evaluate cross-model consistency, we applied the unsupervised chunk discovery method to RNNs with different architectures. All models exhibited consistent structural regularities (Figure 10), with similar chunk counts and alignment to input patterns (Figure despite having different architectures.
D.2 Example of Lookup Table that maps from symbolic clustered state to the input
We apply Discrete Sequence Chunking to the hidden states of simple RNN which was trained on a sequence of ABCD within default E characters. In table we show the resulting lookup table for the discrete strings corresponding to input signals. Many distinct population states correspond to Es, and for A, B, C, and D, separately, there is distinct population state for each. Presumably, the network creates additional states also to distinguish Es in different contexts.
D.3 Validation of PA on RNNs
also validated that PA works for simpler cases on RNNs demonstrated earlier in the paper by applying PA and UCD to RNNs trained on sequences with repeated "ABCD" patterns. PA revealed recurring chunk states whose activation frequency perfectly matched pattern occurrences in the sequence 8
18

---

Neural Population State
Input
021340200453
E
004042212403
E
0323302122044
E
032410212204
E
032010212204
E
12333333111
A
213211131132
B
144304324321
C
040322404444
D
E
300000000020
E
340432012022
440332412200
E
400040312200
E
402040312202
E
00204031 000040322202
E
032010212202
E
Table Sample Look-up Table. The left column corresponds the string that represents the neural
population state. The right column are the corresponding concurrent input.
akgementt
14-
2
Oceation
Demation 1777
Figure 8: Applying population averaging on RNNs predicting simple sequences. Left: The chunk extracted from population averaging aligns closely with the signal-triggered neural embeddings. Middle: When ABCD does occur in the sequence, embedding deviates strongly from the extracted chunks. The count of chunk occurrence in the neural population (x axis) correlates with the number ABCD occurrences in the sequence. Blue line: perfect alignment.
We look the neural population activity when the network learns from a sequence with repeating ABCD as cohesive chunk amid background noise composed of random occurrences of E, F, and G. We get the population template by averaging the population activity vector E Rdx1 over the occurrences of the chunk ABCD:
((8s, S141. 8143) ABCD) H,
T
(8)
L-4 ((8i, 8i 8142, ABCD) i=
H 1::1
T
dev
size(T)
Figure suggests that neural activities, once adapted to the training data, deviate very little from the neural population template. Setting threshold between the population activity and template deviation suffices to distinguish the occurrences of the input pattern based on the neural trajectory.
D.4 Validate UCD on RNNs
To evaluate cross-model consistency, applied the unsupervised chunk discovery method to the simple example in RNN. UCD learns similar chunks which aligns with the DSC extracted chunks (Figure
19

---

Lookup tab
Unsupervised chunk discovery
N neuron
N neuron
10
10
Figure 9: Applying the unsupervised chunking method simple sequence containing repeated instances of ABCD. Left: recurring population activity extracted from the discrete chunk learning. Right: recurring population activity extracted from the unsupervised chunk discovery method.
Neural Population
0-
Holoon untt
Neural Population Activity (n hidden=24)
Hobden unt
2
Neural Population Activity (n hidden=46)
Hodden unit
Figure 10: Neural population activity of the first neurons (unsorted) across three distinct RNNs. Population activities exhibit regularity reflecting patterns of the input sequence despite different network architectures.
D.5
The Reflection Hypothesis Across RNN Architectures
We show that the reflection hypothesis stays intact, independent of the RNN's size. Figure 10 plots RNN's neural activity with an increasing number of hidden neurons (12, 24, and 48, separately). Across all RNNs, the neural activities reflect regularities in the training sequence.
Moreover, applying DSC on the RNNs suggests that the number of chunks extracted stays consistent across RNNs with different numbers of hidden units (Figure II
20

---

Average Number of chunks
Average Lookup Table Size
40
23
30
12
Number min hidden units.
Number of PANN hidden units
Figure 11: The number of chunks stays consistent across RNN with different number of hidden units.
a.
b
C.
d.
ground mruth
400
1350
X.-
802
400
Mines
Modes
Figure 12: Comparison between trained and untrained RNNs on sequences with overlapping words across 10 independent runs and randomly initialized RNNs. Sequence parse length. Dashed line is the ground truth sequence parsing length. b. Number of unique neural population states. c. Chunk dictionary size. d. Chunk dictionary size (filtered desolate chunks (threshold=5)) Error bars represent the standard error of the mean.
D.6 Contrasting Trained versus Untrained RNNs
We provide comprehensive comparison trained and untrained RNNs, focusing on the characteris- ties of their neural trajectory patterns revealed the chunking method.
Figure takes the neural population trajectory as a sequence, uses the learned neural chunks to parse the symbolic neural trajectory, and measures the length of the population trajectory sequence broken into chunks. A comparison of the sequence parsing length suggests that the trained network has a similar amount of neural trajectory chunks to the number of chunks in the ground truth sequence. The untrained network contains more regularities in its neural trajectory. This an indicator training that encourages the network to create new distinct neural trajectory states that help the network distinguish different contexts. This property especially important in this task as overlapping subsequences with different prefixes shall lead to distinct predictions.
Figure 12 suggests that the trained RNN contains more unique neural population states than an untrained RNN. This observation supports the hypothesis that during training, RNN creates more population states that are useful for distinct contextual predictions. Figure 12 c, d. shows the vocabulary size as acquired by the symbolic chunking method. is filtered version of where obsoletely occurring chunks are excluded. The trained RNN's neural trajectory contains more recurring chunks than the untrained RNN. Also note that the size of the vocabulary containing recurring neural trajectories is bigger than the ground truth vocabulary the sequence, pointing to possibility that neural networks acquire multiple neural population chunks that can represent the same word in the sequence. However, the larger vocabulary size can also be an artifact of the clustering algorithm, which has more clusters than proper.
21

---

Perturbation only input
b. Perturb only chunk
Transfer Comparison
Transder Comparison
0.0
0.90
0.55
0.85
0.80
0.00
0.75
0.75
Transfer
Fandder
Figure 13: a. Simply grafting inputs is ineffective and do not encourage reuse. b. Grafting only the neural chunks encourages compositional reuse and transfer.
D.7 Grafting Neural Chunks versus Grafting Input
We examine the effects of grafting at two levels-neural population chunks and input-on the behavior of RNNs when transferring to sequences with compositional components. Chunk-level grafting leads to reliable reuse (Fig. D.7): input grafting proves ineffective. This suggests that merely changing the input is insufficient to alter network behavior; rather, behavior is shaped by the coordinated activity of a larger set of computing units beyond those directly encoding the input.
E
Population Averaging
E.1
Example Layer-wise Statistics PA-extracted Chunks
Beyond using PA to identify concept-encoding chunks for concepts occurring at the end of a prompt sequence, the same approach also be applied to other positions in the sequence. Specifically, PA can isolate chunks that encode concepts appearing at earlier points in the sequence (prior to the end token), well as chunks that encode the future occurrence of target concept. The latter correspond to invariant neural subpopulations whose activity reliably precedes the later appearance of a given concept, which corresponds predictive structure in the network's dynamics.
Figures T4 15 16 and 17 present layer-wise statistics of the learned parameters obtained from training data using the population averaging (PA) method. From left to right, the plots display: Left, the optimal tolerance threshold across layers, Middle the number of neurons contributing to recurring embedding activity, and Right the maximal deviation threshold.
Note that the maximal deviation learned by the method typically increases from earlier to later layers. This trend reflects the architecture of LLLMA, where deeper layers exhibit a broader range of variability their neural activity. In addition, the analysis reveals that concept encoding is generally supported by substantial populations of neurons rather than isolated units. Sparse encodings of concepts are rare, This suggest that concepts emerge through distributed patterns of activity across the network rather than through localized neural encoding.
E.2 Sample Dictionary of Chunks Memorizing and Predicting Signals in Text
We then generalize this method to other locations of the token, such prior to following the occurrences the signal at indices Vss), denote the set of indices undergoing this k step shift as
V-*(a) = (1 / E V(s)), (+*(s) (I + the network's 's neural activity representing memory of the signal, and the latter (k: < 0) examines
E V(=)). The former (k > 0) examines
the network's neural activity that is predictive of signals happening subsequent to the input prompt sequence.
22

---

Optimal Threshold Across Löyers
Nurrber of Encoding Neu ons Across
Maximal Deviation Across Loyers
10"
6000
30-
Optimer Threshad
25
2.0
109
2000
15-
1000
10"1
0.0
Layers
Layers
Layers
Figure 14: Layer-wise statisties of PA extracted chunks commonly preceding the concept "cake".
Optimal Threshold Layers
Nurrber of Encoding Nev Across
Maximal Deviation Across
10"
4000
0.20
Optimel Threstld
0.35
309
2008
0.20
10
Number
1000
0.05
0.00
Layers
Layers
Layers
Figure 15: Layer-wise statistics of PA extracted chunks commonly preceding the concept "in".
Optimal Threshold Across Layers
Nurbeer of Enceding Neu ons Across Layers
Maximal Deviation Across Loyars
107
4000
Opemal Threshold
16l
0.25
Sumbar 0 Encodrig
0.0
10-1
0.20
1000
0.00
Layers
Layers
Txpert
Figure 16: Layer-wise statistics of PA extracted chunks commonly proceeding the concept "of".
Optimal Threshold Across Layers
Nurr of Enceding Neu ons Across
Maximal Deviation Across Loyers
15a
6000
Opšimal Threshold
25
2000
1.5
10
1000
0.0
Layers
Layers
layers
Figure 17: Layer-wise statistics of PA extracted chunks encoding the latest word being "people".
23

---

Cheesecale
Llama3
Decoding Performence Across Layers (Test)
Chetsecale wep 4
15
Cheesecale step-s
Decoding Performance Across Layers
Layer
Decoding Ferformance Across Layers (Test)
10
20
30
10
20
Layer
Chessraskratep
Cheeseall
Decoding Performance Across (Test]
Decoding Performance Across Layers
B
Denn Ben
10
20
30
10
Layer
Layer
Figure 18: Evaluation of the decoding accuracy of the extracted chunks predictive of future elements (right panels) and memorization (left panels) of past elements beyond immediate next-token predic- tions (middle).
Llama3
The, step
Decoding Performance Across Layers (Test)
The, =1 Decoding Performance Across Layers (Test)
Layer
Layer
stop-
Decoding Performance Across Leyers (Test)
The, steps-
Decoding Performance Across Layers (Test)
Q
10
15
20
Figure 19: Another example evaluation of the extracted chunks predictive of future elements (right panels) and memorization (left panels) of past elements.
E.3 Visualizing Concept Encoding Chunks at Different Sequence Token Positions
Figure 20 visualizes the subpopulation activity chunks extracted by the population- aaverognnn method responding to the word "cheese", "cake", and "cheesecake" with their variety of tokenized form, ending with the current, previous, and subsequent signal indices. One observation is that the information about context is represented in the latter rather than the earlier layer of the network. Meanwhile, information about the memory the neurons in the network than the most recent signal. Additionally, the temporal coding of the signal not uniform, representing the same word 'cheese' as memory, as the latest token, or as prediction manifested in distinct neural population actívities. There is smaller neural population responsible for predicting future signal than encoding for current signal or the past signal. Evaluation the decoding performance using the extracted chunks is shown in Figure E.3 and B.3
signal was represented much more sparsely among

---

cheese, sleps-2
cheese, steps-1
cheese, step=0
cheese, step=1
cheese, step=2
stepm-2
step--1
step-1
step=2
Postdiction
Prediction
Word
Word
Wind
Whod
Q
Q
Q
Figure 20: Visualization of the extracted neural subpopulation for the words "cheese", "cake", and "cheesecake", respectively, in both directions.
E.4 Generalizing to Other Concepts
We used the population averaging method to extract neural subpopulation activity chunks of llama accounting for the 100 most frequently occurring words in English (as can be seen in main text Figure adapting the threshold parameters on hidden activities collected from training prompt and evaluating hidden activities collected from a test prompt (appendix H). For each word, we study subpopulation activities ranging from the two steps prior and two steps subsequent the last sequence token position. We observed subpopulation chunks that represent information in both directions.
We also show more examples of decoding accuracy to evaluate how indicative the extracted sub- population chunk is of the signal's existence in Figure 2T in the Oth layer is usually perfect, as the Oth layer is the embedding layer having a fixed dictionary mapping tokens to the token embeddings. In subsequent layers, the presence of the extract indicates the occurrence of the recurring tokens in the input sequence. Having the recurring token in the input at the last sequence location elicits the network's hidden activity to lie within the range of the extracted embedding chunk.
22 23
Note that decoding performance
Generally, the population averaging method learns chunks that are more predictive of the signal in the input sequence with words that have specific meanings than words that serve as prepositions; this can be caused by the network creating many neural population states to distinguish a preposition in different contexts, similar to the observation with RNNs (population averaging would be fragile in this case). We included more examples of chunk evaluation of this type.
25

---

Decoding Performance Across Layers (Test)
TP
20
word people, step=-2
Decoding Performance Across Layers (Test)
20
25
Layer
(b) word people, step=-1
Decoding Performance Across Layers (Test)
R
-1
5
10
15
20
25
30
Layer
(c) word people, step=0
Decoding Performance Across Layers (Test)
TTP
0
5
10
20
25
Layer
(d) word people, step=1
Decoding Performance Across Layers (Test)
Layer
(e) word people, step=2
Figure 21
26

---

Decoding Performance Across Layers (Test)
TP
20
word it, step=-2
Decoding Performance Across Layers (Test)
20
25
Layer
(b) word it, step=-1
Decoding Herformance Across Layers (lest)
20
Layer
(c) word = it, step=0
Decoding Performance Across Layers (Test)
0
10
15
20
25
30
Layer
(d) word = it, step=1
Decoding Performance Across Layers (Test)
0
10
20
25
Layer
(e) word = it, step=2
Figure 22
27

---

Decoding Performance Across Layers (Test)
20
(a) word and, step=-2
Decoding Performance Across Layers (Test)
20
25
Layer
(b) word and, step=-1
Decoding Performance Across Layers (Test)
0
10
15
20
25
30
Layer
(c) word = and, step=0
Decoding Performance Across Layers (Test)
0
5
10
20
25
Layer
(d) word and, step=1
Decoding Performance Across Layers (Test)
0
10
15 Layer
20
25
30
(e) word and,step=2
Figure 23
28

---

E,5
Extracting Chunks from Range of LLMs
We here present more details of results evaluating the extracted PA chunks on other large models. shows the PA chunks corresponding to the sample concepts extracted from mamba (large-scale state-space model), T5 (encoder-decoder architecture), and RWKV (large scale RNN). From the left to the right are the decoding accuracy (divided into true positives and false positives) on the neural activities recorded from the test prompt; the optimal threshold extracted by PA across the layers: the number of concept-eneoding neurons across layers; and the maximal deviation from the mean chunk prototype across layers.
Figure 24
25 26
mamba
Cake
Oetiral
Hyrrber Eroodes Nezzr Accoss Lewe
Devidies Seveo Lover
Cheese
Layer
Deceding Performance Layers
OEENA TREPEMEE ECNESS LAgao
1aa7sa RECOSIN NARRIS AATOOS Layent
MAUTER DIVADY GOTOO agges
Cheesecake di
Performance
Oetirel
Laver
Figure 24: Evaluation of the extracted chunks in mamba (25 layers, embedding dimension 768, large-scale RNN) on the three example words illustrated in the main paper.
T5
Decoding Performance Acress Layers (The))
Optraal Thaanoli Aczas Layers
Numoee Encoding Naaton Across Layara
Plac inal Deniatio Norua Lapers
4000
Cheese
Decoding Rerformance Across Layers (Test)
0-
Cheesecake
Laver
Octiral Acres Lavers
Number Croodine Nearta Actos Levers
Montrel
Figure 25: Evaluation of the extracted chunks in T5 (7 encoder layers, embedding dimension 512, encoder-decoder architecture) on the three example words illustrated in the main paper.
29

---

RWKV
Cake
Decoding Performance Laye's
Cheese
Decoding Performance Acreas (Rest)
Detind Longs
Excrcino Neamm
Cheesecake
Layer
layt
ayers
Lyys
Figure 26: Evaluation of the extracted chunks in RWKV (13 layers, large-scale RNN) on the three example words illustrated in the main paper.
E.6 Quantification Causal Intervention
We quantified the effectiveness of causal evaluation by the percentage of concept occurrence in the generated sequences by LLM, comparing intervention (grafting/frrezing)) with control (no interven- tion) for the illustrated example concepts.
Table 3: Percentage of target concept occurrence for grafting and control using the prompt: "Hi, how are you doing?" (N 100).
Target Concept
Without Grafting
With Grafting
cake
1%
83%
cheese
0%
90%
cheesecake
0%
16%
Table 4: Percentage of target concept occurrence for freezing and control across two prompts (N 100). Row Freezing results for the prompt: "What is the name rich, savory food made from curdled milk, often aged enhance its flavors?" Target word: cheese, Perturbation Layers: 2-10, N = 100. Row 2: Freezing results for the prompt: "Name three most common desserts in Europe." Target concept: cake, N 100.
Target Concept
No Freezing
Freezing
cheese
67%
33%
cake
36%
9%
Generally, grafting is more effective than freezing in influencing target concept occurrence. However, both types of intervention have measurable effects on influencing the model's generated text.
E.7 Comment on Context Sensitivity of PA Chunks
Population averaging assumes that averaging hidden states across multiple contexts yields mean- ingful, general 'chunk" for the concept. We study context-sensitivity of PA in extracting the internal representations in transformers. Indeed, modern language models such as Transformers are known to exhibit highly context-sensitive internal representations, and given word (e-g., "bank") may evoke distinct activations depending on its context (c.g. a river bank versus a financial bank).
30

---

However, our goal is not to model context-specific usage, but rather to extract the core structural regularities that persist across contexts- analogouss approaching semantic prototypes in word2vec and Ve We assume that the mean of many contextual embeddings approximates central semantic prototype, and averaging over many diverse instances of the identical concepts serves to extract the common representation shared across contexts.
This assumption on the algorithm then finds its empirical validation in the experiments. We observe that PA and UCD-derived chunks correlate well with many concepts. This suggests that for at least subset of words, there exists a stable representation despite contextual variations. And this is the first step towards interpretability at a population dynamic level.
We acknowledge that this would be the case for some concepts, but sometimes multiple concepts map to the same word, such as 'bank' contains a different meaning on different occasions. these cases there may be two chunks of neural population activities encoding for river bank and financial bank. And for some words such as prepositions, they have much more contextual nuance, we would expect this method to be less effective.
Word school
With Perturbation
Control (No Perturb.)
(Perturb Control)
78.7%
0.7%
78.0%
christmas
74.3%
0.3%
74.0%
she
71.7%
0.0%
71.7%
family
71.3%
1.7%
69.7%
Table
Words with the highest perturbation effectiveness (percentages).
Word
With Perturbation Control (No Perturb.)
A (Perturb Control)
if
7.3%
6.7%
0.7%
11.7%
7.7%
4%
on
28.7%
23.0%
5.7%
Table 6: Words with the lowest perturbation effectiveness (population averaging chunks).
Indeed we observed that the grafting success of the chunks dependent on the type of words. We noticed grafting can be very successful concrete words, examples the concepts most effective by grafting include school, christmas, family (t. For non-specific words such as prepositions and conjunctions, grafting can be ineffecctive. Examples are 'if', 'at', 'on' (table This is often correlated with PA quality which can also reflected the PA decoding evaluation metrics for those specific concepts. In short, despite contextual dependency, empirical evidence suggests that this assumption works well for at least a substantial subset of all existing concepts, and this paves a way for future investigation expand the interpretability step-wise towards more contextual subtlety.
F
Unsupervised Chunking
F1
Attempted Architectures that Led to the Current Design
We also experimented with SAE-like variants for chunk discovery, using a reconstruction loss with continuous hidden activations and tying the decoder weights to the transpose the encoder weights. However, this approach yielded many hidden units to interpret meaningfully, and the continuous, additive nature of the reconstruction did not align well with our definition of chunks. To improve interpretability, we also tried variations that binarized the hidden layer, but this variant struggled to train effectively. Moreover, even with binarized activations, the additive reconstruction process still diverged from our intended definition of neural chunks as discrete, reusable population patterns.
F.2 Evaluating Chunk Qualities
To evaluate the chunks extracted from UCD, we have explored a number of measures to assess the quality of the extracted chunks.
good chunk needs to approximate the embedding. Therefore we check the normalized cosine similarity between the maximally matching chunk in the dictionary with each data point, and ensure
31

---

a.
3000
690
440
C.
d.
Oestiuzton vakas
Figure 27: a. Similarity scores between the maximally similar chunk and the embedding it explains are largely unimodal, with a peak near one due to predictable tokens (e.g., start/end markers). b. Most embeddings are explained by a single dominant chunk, with low similarity to other chunks. c. Chunk identification frequency is unimodal; chunks are identified similar number of times. d. Learned chunks lie sparse subspace, with weight distributions centered near zero.
that the cosine similarity between embedding and its matching kernel are reasonably similar. Figure 27 a shows the distribution of cosine similarities between embedding chunks and the embeddings in the 10th layer. The average cosine similarity between the maximally similar dictionary chunk and the embeddings is concentrated around 0.5.
To verify semantic distinctness, for one embedding, it needs to be similar to a few chunks while being different from most chunks in the dictionary. Figure 27 b shows the distribution of cosine similarities between embedding chunks and the embeddings in the 10th layer.
Additionally, the number of times that each chunk is identified to be the most similar chunk to the current embedding should be distributed in a reasonable way. There should not be a few chunks that identified. This identification diversity is illustrated in Figure 27 c. Chunk usage is also unimodal, with most chunks selected a similar number of times.
Chunks lie in sparse subspace with weights centered near zero (Figure 27d). t), and each embedding 29 also shows two example
typically explained by a single dominant chunk (Figure 27p). Figure 29
chunk representations and the embeddings they explain, compared to control embeddings.
On top of that, we visually inspect samples of the extracted chunks, of its matching embedding, and non- non-matching embedding as a control. Two sets of these examples are shown in Figure 28
F.3
Visualization
illustrates the most frequently identified chunk in the 10th layer (visualized as vd x vd image, where d is the embedding dimensionality), alongside two embedding examples where this chunk is identified as the maximally similar. Additionally, include a control embedding where the same chunk is not identified as maximally similar. The extracted chunk is visually more similar to the two embedding examples than to the control embedding, demonstrating that this method extracts visually similar recurring neural population activities.
Figure 28
F,4 The influence of hyperparameter K on Unsupervised Chunk Discovery
We tested the impact of varying K (500-8000) on early, middle, and late layers shown in 30 Increasing does not consistently reduce loss, and the method's performance does not dramatically differ; overall, this method's effectivvnees remains stable across range of hyperparameter choices.
32

---

Chunk embedding examples which chunk maximally similar, compared a control embedding
a.
Chunk
Embedding 1
Embedding 2
control
b.
Chunk
Embedding Sample
Embedding Sample
control
Figure 28: Two example chunk representations and the embeddings they explain, compared to randomly selected control embeddings. Chunk-associated embeddings exhibit high similarity and structure, while control embeddings show low similarity.
Example chunks learned unsupervised chunk discovery Example
Example
Example
Example
Figure 29: Example chunks learned by the unsupervised chunk discovery method.
F.5 Full plot on layer-wise chunk processing in LLaMA-3
UCD simplifies nigh-dimensional activations into interpretable patterns of chunk appearance and disappearance. Figure 31 illustrates chunk interactions across LLaMA-3's s all layers processing the beginning Emma.
F.6 Correlating sentence POS tags with chunk activities
We can interpret the unsupervised learned chunks by mapping them to interpretable linguistic structures, such part-of-speech (POS) tags. To achieve this, extracted the POS tags for the corpus using the averaged perceptron tagger [11], following the Penn Treebank POS Tagset [55] We then computed the correlation between each chunk with each POS tag for every layer.
Figure 32 visualizes the maximum correlation between each POS tag and its most correlated discovered chunk across network layers (excluding the embedding layer). Our findings align with prior research showing that certain POS tags are processed the earlier layers of the network and clearly demonstrate that there are chunk activities purely responsible for certain pos tags (possessive nouns, for example), while others hold very strong correlations.
42
82
Additionally, the POS-tag-correlated correlated neural activities peak in earlier layers but also persist in later layers, indicating a sustained representation syntactic information throughout the network. These findings suggest that chunk activities learned without supervision can serve candidates for inter- preting computational components within the network that are responsible for processing abstract concepts.
33

---

-0.30
Kamsag
-0.34
Effect of Maximal Correaation with POS tags Across
Layer 4
Layer 16
Layer 30
0.4
1000 2000
3000
4000
3000
6000
7000
5000
3000
3000
4000
3000
8000
7000
8000
1.000
20000
3000
4200
8000
7000
8000
Figure 30: Effect hyperparameter choice on the unsupervised chunk discovery method. The effect of K on the loss (upper plots) and maximal correlation with POS tags (lower plots) across layers 4, 16, and 30 (early, middle, and late). Increasing does not consistently reduce loss, and the method's performance do not dramatically differ across choices of hyperparameters.
Max Correlation
RSR
RU5
VBP
MN4
MRR
22
MPPS
MNP
M
FRB
VBG
10
11
12
13
15
16
17
18
19
22
23
24
25
25
27
28
29
30
31
34
Figure 32: Part-of-speech tags and their maximal correlation across all layers in LLaMA-3.

---

Tokens
comfortable
handsome
disposition
seemed
blessings
existence
Emmm
Wood
house
clever
rich
home
and
happy
unite
some
the
best
of
of
1
2
3
4
5
7
8
10
11
14
17
18
20
21
23
24
26
27
30
31
Figure 31: Interactions of neural population chunks upon parsing the beginning sentence of Emma (all layers).
35

---

G Computational Cost of Applying Chunk Extraction at Scale
All interpretability methods that analyze neural population activities need access to LLM neural activities, which can be acquired in single forward pass on the data. This is required for SAE, representation engineering, and chunking methods alike.
For PA that extracts chunks from shorter input, taking a 1,000-token sequence, for example, our matching and tolerance sweep are applied to hidden state arrays of -16MB per layer, resulting in total memory usage under 200MB. Even across all 32 layers, the complete decoding procedure finishes in under 1 minute on standard CPU. This makes our analysis tractable even in constrained environments,
method is lightweight. The cosine-similarity-aasee
The unsupervised chunk learning model consists of -8.2 million parameters (learning a matrix of shape [2000, 4096]) and operates primarily through matrix multiplication for cosine similarity computation. With input sequences comprising ~200,000 tokens (e.g.. Austen-Fmma),, the batch- based training regime (batch size 32) requires under 100MB of GPU memory per step. On a single NVIDIA RTX 3090, training for epoch takes 30-60 seconds, and 100 epochs takes about 1-2 hours. All layers will take 32-64 hours if done sequentially. In comparison to the SAE variate, where the latent layer has 131,072 neurons (i.e., 32x the input dimension of 4096). This increases the parameter count to ~1.07 billion. The same setting completes 100 epochs in approximately 6-10 hours per layer. All layers will take 192-320 GPU-hours.
Envisioning Chunk Discovery Scaling to Production-Sized LLMs (e.g., 70B+)
When applied to production-scale model like ILLMA-70B,, the proposed methodology remains tractable. For PA over sequences such as one containing 1000 tokens, each hidden state array occupics -32MB, and the total memory usage across all 80 layers is approximately 2.5GB if done simultaneously. The total runtime will be a few minutes on CPU per concept across all layers. The unsupervised chunk learning model, which learns 2000 x 8192 matrix (-16.4 million parameters), will require under 1GB of GPU memory per step and trains in approximately 2-3 minutes per epoch, totaling 3-5 hours for 100 epochs on a single NVIDIA RTX 3090, with batch size 32 and 200,000 input tokens. All 80 layers shall take 240 400 GPU-hours. In contrast, the sparse autoencoder variant with 262, 144-dimenlional latent layer (32x overcomplete) increases the parameter count to -4.1 I billion, requiring -50GB of memory and 12-20 hours to train over 100 epochs per layer. All 80 layers will take 860- 1600 GPU-hours.
H Prompt Bank
We show the prompt used to extract the demonstrated word in the main manuscript using the population averaging method:
prompttcheesecaak cheesecake, train = "Cheese is one of the most versatile ingredients in the culinary world, and cheese can be used in everything from savory dishes to desserts. Cheese lovers often enjoy pairing cheese with crackers, wine, or fruit, but cheese also shines in baking. Cake, on the other hand, is the quintessential dessert, with cake being staple at celebrations. Cake comes in many forms, such as chocolate cake, vanilla cake, or even carrot cake. However, when you bring cheese and cake together create cheesecake, a magical transformation happens. Cheesecake is a dessert like no other, with cheesecake offering the creaminess of cheese and the sweetness of cake in perfect harmony. Cheesecake can be topped with fruits like strawberries blueberries, or cheesecake can be flavored with chocolate or caramel. Some people prefer classic cheesecake, while others enjoy a more decadent cheesecake loaded with toppings. Regardless of the variation, cheesecake remains one of the most beloved desserts worldwide. The crust of cheesecake, often made from crushed biscuits or graham crackers, complements the smooth filling, making cheesecake irresistible. Cheese plays a central role in cheesecake, while the influence of cake ensures that cheesecake is always delightful dessert. Whether you love cheese, crave cake, or are obsessed with cheesecake, this dessert proves that the combination of cheese and cake is truly extraordinary. Every bite of cheesecake reminds us that cheese and cake, when united cheesecake, are match made in heaven. Cheesecake aficionados often debate whether baked cheesecake or no-bake cheesecake is superior, but all agree that cheesecake a dessert worth savoring. With so many variations, cheesecake enthusiasts never tire of exploring new ways to enjoy their favorite dessert. Cheese and cake come together seamlessly
36

---

in cheesecake, showing how cheese and cake can create something greater than their individual parts. Cheesecake is, without a doubt, the ultimate testament to the greatness of cheese and cake in unison."
The extracted chunks are then evaluated on the recorded hidden activity of the following prompt:
batrrcakkkeettt==aee = Chheee is culinary treasure that has delighted taste buds for centuries. Whether it's creamy, tangy, or sharp, cheese offers endless possibilitics. Cheese finds its way into countless dishes, from savory casseroles to gooey pizzas, and its versatility knows no bounds. Cake, too, is a universal favorite, with cake symbolizing joy, celebration, and indulgence. Cake comes in every flavor imaginable- chocolate cake, vanilla cake, red velvet cake- and each cake brings its own special charm. But when cheese and cake are combined to form cheesecake, something truly extraordinary happens. Cheesecake is a dessert that transcends expectations, merging the velvety richness of cheese with the sweet, airy allure of cake. Cheesecake can be baked or chilled, simple or elaborate, yet every cheesecake captures the perfect balance of flavors. Classic cheesecake recipes highlight the creamy taste of cheese, while fruit-topped cheesecake adds a burst of freshness. Some people swear by chocolate cheesecake or caramel-drizzie zesty lemon cheesecake, No matter the variation, cheesecake consistently proves that the union of cheese and cake is match made in heaven. The crust of cheesecake, typically crafted from crushed cookies or graham crackers, provides the ideal foundation for the smooth and luscious cheese layer. Every bite of cheesecake reminds us why this dessert has stood the test of time. Fans of cheese, cake, and cheesecake alike agree that cheesecake combines the best of both worlds. Whether you'l re indulging in a slice of classic cheesecake, exploring new cheesecake flavors, or savoring the rich taste of a perfectly baked cheesecake, it's clear that cheese and cake reach their pinnacle when united in cheesecake. Cheesecake is a testament to how cheese and cake, when brought together, create something greater than the sum of their parts. From the first bite to the last, cheesecake is a celebration of everything wonderful about cheese, cake, and, of course, cheesecake itself.'
cheesecake, while others can't resist a
We show the prompt used to extract the top 100 frequent words using the population averaging method:
prompt_ _requent_ words _atiii-rrin--- = "In today's 's world, people often feel the push and pull of con- nection and solitude. With technology and social media the rise, we now have countless ways to stay in touch with those we know and love. However, the question of whether this digital world can truly satisfy our need for real connection remains. To be truly connected is to share experiences, to understand and to be understood. This kind of connection goes far beyond a screen. The desire to connect is universal, and people have searched for it throughout history. In ancient times, communities formed to support one another, to live together, and to build bonds that could help them through challenges. Today, people may still crave this closeness, yet it is not always easy to find in our fast-paced world. While the internet gives us access to almost anyone, anywhere, it does not always give us the depth of connection that true friendship and family relationships can offer. Consider a family spending time together. For many people, family is a source of strength, a place where they feel safe and understood. But as life becomes busier, it can be easy to let work, hobbies, or other commitments pull us away from family time. Many people find that they must make a conscious effort to set aside time for their loved ones. To sit down for a meal together, to talk about the day, to share thoughts and laughter- - these moments are priceless. They are what remind us of who we are and who we want to become. Friendship, too, plays a significant role in life. Friends are often the people we choose to spend time with, the ones who share our interests and support us. To have friends is to feel understood in a unique way, to laugh together, to encourage each other, and sometimes just to sit in silence knowing someone is there. A true friend listens without judgment, stands by us in hard times, and celebrates with us in good times. However, maintaining friendships can require work and commitment. With busy lives, people can sometimes lose touch, only to realize later how much those friendships meant. In the workplace, relationships are equally important. Many people spend significant amount of time at work, so having good connections there can make big difference. Working with others requires collaboration, understanding, and respect. When people feel connected their colleagues, they tend to work better together, share ideas freely, and support each other. A positive work environment can foster not only productivity but also well-being. People are more likely feel satisfied in their work when they know they are valued and understood by those around them. Of course, technology plays a large role in modern connections. Platforms like social media allow people to connect across distances, to share life events, and communicate instantly. For some, these tools make it easier to stay in touch with family and friends, to share news, and to express themselves. However, while technology can bring people closer, can also create a sense of
37

---

distance. Seeing the lives of others through screen is different from experiencing life together in person. The highlights of life shared online may not always show the full picture, leaving people to wonder if they are missing out. To have genuine connection, people often need to go beyond what is easy and convenient. Sometimes, this means reaching out, making an effort, and being open. True connection requires understanding and empathy. It asks us to listen, to be present, and to care. In a world that often moves fast, taking the time to connect deeply can feel like a challenge, but is also incredibly rewarding. The value of connection is evident in difficult times. When people go through challenges, it is often those close connections that help them through. Whether it's a friend who listens, family member who offers support, or a colleague who steps in to lend hand, these connections give people strength. Knowing that someone else understands or is there to help can make all the difference. Moreover, the abilitt to connect also fosters compassion. When people share experiences, they begin to see the world from each other's 's perspectives. This understanding can lead to greater kindness and less judgment. People who feel connected are often more empathetic, more understanding, and more willing to help others. This creates a positive cycle, as kindness and empathy tend to inspire more of the same. For people to have balanced and fulfilled life, connection is essential. But to nurture these connections takes effort. It is not always easy to set aside time, to reach out, or stay in touch. Life can be busy, and distractions are everywhere. However, those who make the effort to connect often find that their lives are richer and more satisfying. The joy of shared laughter, the comfort of understanding, and the strength of support are all things that make life meaningful. As we move through life, the connections we make help to shape who we are. We learn from others, grow with them, and find new perspectives. Each person we meet adds to our experience and helps us to see the world in new ways. Sometimes, people find that their most valuable lessons come from those who are different from them. To connect with people from various backgrounds and with different life experiences is to broaden our view of the world. In conclusion, to live fully to connect meaningfully. Whether through family, friends, colleagues, or even strangers, these bonds enrich life. They offer joy, comfort, and understanding, and they remind us that we are not alone. While technology may change the way we communicate, it cannot replace the depth of real connection. To make time for those who matter, to share moments, and to care is to live life of purpose and love."
The extracted chunks on the top 100 frequent words are then evaluated on the recorded hidden activity of the following prompt:
prompt_ frenupt_requettt_r_ prnquent_wordds fills_tet==--- = "In the world we live in, each day is filled with choices we all make, big small. The way approach these choices can be what shapes not only our own lives but also the lives of those around us. To be the kind of person who reflects on what they do, who they are, and who they want to become, is to take a meaningful step toward self-awareness and growth. One of the first things to know about making decisions is that they are all interconnected. When we choose do one thing, it often means we cannot do another. This may seem obvious. but to understand the full impact of this reality. it helps to look at the ways in which choices play out in real life. We are always presented with options, and while some decisions may seem trivial at first, they add up over time. We can think about it like this: to choose a path, even if small, can set in motion a series of events that shape our lives in ways we could never fully predict. For example, people make decisions on how to spend their time. Time is one of the most valuable resources we have. There is no getting more of it, and once it's gone, it's gone for good. How we choose to spend it whether working, relaxing, being with family, or helping others- says lot about what we value. Some people may spend time worrying about things that, in the end, are not as important as they seem, while others may put time into making themselves or others better. This shows the difference in what people find to be meaningful or valuable in life. When we look at those around us, we see that everyone is trying to figure out what it means to live good life. Some believe that to have successful career is key, while others might think that family or friendships are the foundation of a fulfilling life. Whatever the focus may be, it's clear that we all want life filled with purpose. The concept of purpose is one that everyone seems to look for, although it can mean different things to different people. People sometimes find that purpose comes from the roles they play in the lives of others. For instance, many parents feel that to have children and raise them well is one of the most meaningful things they can do. They look to guide their children, to give them the tools they need to make their own choices. The idea of helping others extends beyond family, as people also contribute to their communities in many ways. Volunteering, supporting friends, and giving back are just some ways people find meaning in their lives. In work, too, people seek purpose. It's common to find that people want to do something that they can be proud of, something that allows them to use their
38

---

talents and contribute to society. This desire to work well is what drives many people forward and keeps them motivated. Yet, work can also become overwhelming, especially when people forget to balance their time between work and other areas of life. Balance is essential in any good life. We often have to remind ourselves not to let any one part of life take up all of our attention. It's not easy, but it is essential if we want to live fully. This balance extends to how people think about success. For some, success is about achieving certain goals, like owning a home, getting a promotion, or earning a certain amount of money. For others, success is about having good relationships, feeling at peace, and being happy. Each person will have their own idea of what it means to succeed. Some may be quick to compare themselves to others, thinking that if someone else has something they don't tt, they somehow lacking. But to make comparisons is not often helpful. We each have our own journey, and to look too much at what others have can take away from the joy of our own experiences. Another important part of life is facing challenges. There are times when things don't go as planned, and it's easy to feel frustrated. However, these moments are often when we learn the most about ourselves. Challenges can show us what we are capable of and remind us that we are stronger than we think. People often say that to know hardship is to know strength, and it's true that the challenges can make us wiser and more resilient. People also face choices about how they treat those around them. Kindness, empathy, and patience are qualities that many strive to have, but it's not always easy to be kind in a busy, fast-paced world. However, those who make a habit of treating others with respect and understanding often find that they are happier and more fulfilled. Relationships, whether with family, friends, or colleagues, require effort and care. By giving time to nurture these connections, we make life richer not only for ourselves but also for others. In moments of reflection, people often think about their lives and ask themselves. "Am I doing what want to do? Am I living the life I want to live?" These questions can be difficult to answer, but they are important. To pause and look within is to take stock of what matters. This reflection can help guide future decisions and keep people on the path that is right for them. At the end of the day, life is made up of moments, decisions, and interactions. To be mindful of how we choose to spend these moments, of who we choose to spend them with, and of what we put into the world, is to live with purpose. While each person's 's path may be unique, we all share the desire to find happiness and meaning. In seeking to make each day count, to treat others well, and to pursue goals that matter to us, we contribute to a world that is better for all. It may not always be easy, and we may not always make the right choices. But to try, to reflect, and to grow from each experience, is to live a life well-lived. We each have the power make choices that, in time, build story of who we and what stand for. Let that story be one of kindness, purpose, and joy, shared with those who mean the most to us."
face
I Compute Disclosure
All experiments were conducted on a shared internal cluster equipped with NVIDIA Quadro RTX 6000. Collecting data on neural activities from LLaMA-3-8B residual streams took approximately 25 GPU hours, and from SAE neurons took GPU hours. Training the unsupervised chunk discovery across 32 layers took about 90 GPU hours. Additional exploratory and ablation experiments, which were not included the final version of the paper, consumed approximately another 50 GPU hours. The total compute used for the experiments reported in the paper is estimated at around 170 GPU hours. We include these details the supplementary material to facilitate reproducibility.
J Ethics Disclosure and Broader Impact Statement
This work contributes to the interpretability of large-scale neural networks by identifying recurring structured patterns ("chunks") in their internal representations. On the positive side, enhancing the transparency of these models can aid in better debugging, auditing, and alignment with human expectations, potentially reducing harmful outputs and supporting trustworthy AI deployment in high-stakes settings such healthcare, education, or legal -ecision-making.
Furthermore, our chunk-based approach can facilitate more sample-efficient or modular fine-tuning by enabling structured interventions in network behavior, which may reduce computational costs and environmental impact in downstream applications.
However, we also acknowledge that improved model interpretability may lower the barrier to misuse by enabling more precise model manipulation, adversarial targeting, or extraction of private information from model internals. Additionally, interpretability methods may be incorrectly assumed
39

---

to confer safety or faireess guarantees when, in reality, they offer only partial insights. These risks underscore the importance of using interpretability tools in contextual and ethical manner, especially in deployment.
We encourage responsible application and further research into robust, human-aligned interpretability frameworks that include social considerations beyond technical performance.
40

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introductions make claims that substantiated by the method and result part of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a section in the paper that discusses the limitations of this work and encourages future work investigate further on the topic.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
41

---

Answer: [Yes]
Justification:
We provide complete proofs in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in SI or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have disclosed detailed information in the main text as well the supple- mentary information, which are necessary to reproduce this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
42

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided open access to the data and code to produce the main experimental finding and supplementary material in the attached zip file of this submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.co cc/
public/guides/ wttaeees//ooeeSeeeiiiiiinoiiiee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccccpblcccguuuddooooo Cooee//mdeeeSbmmmaia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included in the paper all the training and test details necessary to understand the results. Sometimes we do need to be brief in the main paper due to space limit, but in such cases we add sufficient detail in the SI.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in SI, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars and confidence intervals that clearly defined.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
43

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
All experiments were run shared internal cluster using NVIDIA A100 GPUs. Collecting data on neural activities from LLaMA-3- residual streams took approx- imately 25 GPU hours, and from SAE neurons took 5 GPU hours. Training the unsupervised chunk discovery across 32 layers took about 90 GPU hours. Additional exploratory and ablation experiments, which were not included in the final version of the paper, consumed approximately another 50 GPU hours. The total compute used for the experiments re- ported in the paper is estimated at around 170 GPU hours. We include these details in the supplementary material to facilitate reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics attpp://neurri wwweeurrrpsccppuuliieeppc wwtpccppccpccteeeeeeeeee itt.eceeeeeccciieeeeeieeee
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and can confirm that our research conforms to the code of ethics as provided by the conference guideline.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
44

---

Justification: This work contributes to the interpretability of large-scale neural networks by identifying recurring structured patterns ("chunks") in their internal representations. On the positive side, enhancing the transparency of these models can aid in better debugging. auditing, and alignment with human expe potentially reducing harmful outputs and supporting trustworthy Al deployment in high-stakes settings such as healthcare, education, or legal reciison-making
Furthermore, our chunk-baut based approach can facilitate more sample-effciiet or modular fine-tuning by enabling structured interventions in network behavior, which may reduce computational costs and environmental impact in downstream applications.
However, we also acknowledge that improved model interpretability may lower the barrier to misuse by enabling more precise model manipulation, adversarial targeting, or extraction of private information from model internals. Additionally, interpretability methods may be incorrectly assumed confer safety or fairness guarantees when, in reality, they offer only partial insights. These risks underscore the importance of contextual and ethical use of interpretability tools, especially in deployment.
We encourage responsible application and further research into robust, human- alinned interpretability frameworks that include social considerations beyond technical performance.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not think our paper poses such risks, as this paper do not release pretrained language models scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
45

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best-faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly available models and datasets whose creators are properly credited in the main text and references. Specifically, rely on pretrained language models such as LLaMA-3, T3, RWKV, and Mamba, and we acknowledge their respective authors and original repositories. All models used in accordance with their respective licenses and terms of use (e.g., non-commercial academic usage for LLaMA-3), Any additional assets (e.g., codebases for sparse autoencoders or tokenizers) similarly cited and used under their permissible terms. We include relevant license information and attributions in both the main text and supplementary material.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com /dttaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
paperssithcode
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [No]
Justification: We do not currently release new assets (e.g., code or models) with this submission. We intend to make them publicly available upon acceptance, at which point will include comprehensive documentation to support reproducibility.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
46

---

Answer: [NA]
Justification:
This research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: This paper does not contain studies participants, hence do not involve the approval process of Institutional Review Boards.
Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development introduced by this paper does not involve LLMs. Rather, this work studies LLMss
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs any important, original, non-standard components.
Please refer to LLM policy oct:pe///baaureeccc/o/ooorrrreee for what should or should not be described.
47


Reviews:
[{"review_id": "1GcY4WiySv", "reviewer": "Reviewer_86W2", "summary": "This paper introduces a novel framework for interpreting neural network computations through the cognitive lens of chunking, proposing the Reflection Hypothesis: well-trained neural networks exhibit internal neural population patterns that mirror the regularities in their training data. The authors develop three complementary methods—Discrete Sequence Chunking (DSC), Neural Population Averaging (PA), and Unsupervised Chunk Discovery (UCD)—to extract recurring, interpretable neural activity patterns (chunks) from both small RNNs and large language models (LLMs). These chunks capture concrete or abstract concepts present in input sequences. The authors empirically validate the hypothesis through comprehensive experiments spanning synthetic RNN tasks and modern LLMs like LLaMA-3, showing that chunk activations robustly align with recurring words and abstract narrative schemas. Grafting and freezing these chunks causally affect the models’ predictions, offering mechanistic interpretability grounded in cognitively inspired principles.", "questions": "1. How do the authors envision chunk discovery scaling to production-sized LLMs (e.g., 70B+) given the need for large embedding datasets and iterative dictionary learning?\n\n2. Could chunk-based interpretability complement or enhance existing sparse autoencoder approaches or linear probe techniques? Any preliminary experiments combining them?\n\n3. Have the authors explored whether the extracted chunks remain stable or shift when the same base model is finetuned on different tasks? This could reveal insights into concept drift during finetuning.\n\n4. Beyond the included narrative schema experiments, could future work assess whether chunk activations correlate with human attention or comprehension on tasks like QA or summarization?", "limitations": "Yes. The authors clearly acknowledge the potential dependency on training data structures, the challenges in chunking highly dynamic contexts, and limitations of relying solely on chunk-based explanations for interpreting complex behaviors. They also highlight promising directions for future research to formalize theoretical underpinnings, which aligns with best practices in interpretability research.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The Reflection Hypothesis reframes interpretability by connecting neural population trajectories to human-like chunking, a fresh perspective beyond token-level or neuron-level probes.\n\n- The three proposed chunk extraction techniques effectively handle a wide range of settings, from low-dimensional RNNs to high-dimensional LLMs.\n\n- The paper presents extensive experiments demonstrating chunk consistency across architectures, scalability to LLMs, and the causal influence of extracted chunks on model behavior.\n\n- The ability to graft or freeze identified chunks and observe systematic changes in model output provides compelling evidence of interpretability.\n\n- Validation on diverse architectures (RNNs, transformers, state-space models) supports the robustness and applicability of the proposed methods.\n\nWeaknesses:\n\n- The paper could better contextualize how its framework integrates with or contrasts against recent neuron-centric interpretability methods like SAE circuits or representation engineering.\n\n- Some details on the computational cost of applying chunk extraction at scale in large models are missing.\n\n- While synthetic and narrative-driven datasets are useful, real-world downstream tasks (e.g., factuality or bias assessment) would further strengthen practical significance.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your insightful suggestions. We clarify and contrast our method with alternative interpretability methods a bit more, especially on computation. We also ran more evaluations on the grafting experiment and examined the grafting effectiveness in multi-modal models. \n\n> Some details on the computational cost of applying chunk extraction at scale in large models are missing.\n\nAll interpretability methods that analyze neural population activities need access to LLM neural activities, which can be acquired in a single forward pass on the data. This is required for SAE, representation engineering, and chunking methods alike.\n\nFor PA that extracts chunks from shorter input, taking a 1,000-token sequence for example, our method is lightweight. The cosine-similarity-based matching and tolerance sweep are applied to hidden state arrays of ~16MB per layer, resulting in total memory usage under 200MB. Even across all 32 layers, the complete decoding procedure finishes in under 1 minute on a standard CPU. This makes our analysis tractable even in constrained environments. \n\nThe unsupervised chunk learning model consists of ~8.2 million parameters (learning a matrix of shape [2000, 4096]) and operates primarily through matrix multiplication for cosine similarity computation. With input sequences comprising ~200,000 tokens (e.g., Austen–Emma), the batch-based training regime (batch size 32) requires under 100MB of GPU memory per step. On a single NVIDIA RTX 3090, training for 1 epoch takes 30-60 seconds, and 100 epochs takes about 1-2 hours. All layers will take 32-64 hours if done sequentially. In comparison to the SAE variate, where the latent layer has 131,072 neurons (i.e., 32× the input dimension of 4096). This increases the parameter count to ~1.07 billion. The same setting completes 100 epochs in approximately 6–10 hours per layer. All layers will take 192-320 GPU-hours.  \n\nWe will add a discussion of these computational costs, along with concrete runtime estimates and scaling behavior in the supplementary material to clarify the tractability of our approach at scale.\n\n> How do the authors envision chunk discovery scaling to production-sized LLMs (e.g., 70B+) given the need for large embedding datasets and iterative dictionary learning?\n\nWhen applied to a production-scale model like LLaMA-70B, our methodology remains tractable. For PA over sequences such as one containing 1000 tokens, each hidden state array occupies around 32MB, and the total memory usage across all 80 layers is approximately 2.5GB if done simultaneously. The total runtime will be a few minutes on CPU per concept across all layers. The unsupervised chunk learning model, which learns a 2000 × 8192 matrix (around 16.4 million parameters), will require under 1GB of GPU memory per step and trains in approximately 2–3 minutes per epoch, totaling 3–5 hours for 100 epochs on a single NVIDIA RTX 3090, with batch size 32 and 200,000 input tokens. All 80 layers shall take 240 - 400 GPU-hours.  In contrast, the sparse autoencoder variant with a 262,144-dimensional latent layer (32× overcomplete) increases the parameter count to around 4.1 billion, requiring around 50GB of memory and 12–20 hours to train over 100 epochs per layer. All 80 layers will take 860 - 1600 GPU-hours. \n\n\n> While synthetic and narrative-driven datasets are useful, real-world downstream tasks (e.g., factuality or bias assessment) would further strengthen practical significance.\n\nWe appreciate the reviewer’s suggestion and agree that evaluating in real-world contexts is important. To this end, we extended our analysis to the ROCStories benchmark—a naturalistic narrative corpus—and evaluated grafting effectiveness using the TREC Question Classification dataset, which spans six real-world categories. For each category, we sampled 50 prompts and compared generation outcomes with and without grafting. We noticed that averaging all collected words, grafting consistently improved word occurrence probability across all context categories: +41% (ABBR), +33% (DESC), +36% (ENTY), +35% (HUM), +37% (LOC), and +34% (NUM). These results highlight the method’s practical potential to steer model behavior across practical settings. (more detailed discussion about this experiment can be found in our response to reviewer 1jwn). \n\n\n> Could chunk-based interpretability complement or enhance existing sparse autoencoder approaches or linear probe techniques? Any preliminary experiments combining them?  \n\nWe have not thought about combining chunking with SAE. Can the reviewer clarify what an example of a combination could be? And what can be the specific usage case of such a combination? In general, we view chunk-based interpretability as an alternative to SAEs.\n\n> Have the authors explored whether the extracted chunks remain stable or shift when the same base model is finetuned on different tasks? This could reveal insights into concept drift during finetuning.\n\nOur RNN experiment goes in this direction. What we did was to train the RNN with a training sequence that contains two randomly occurring words, ABCD and JKLMN, and fine-tune it on another sequence with the combined word ABCDLMN. We observed that the network state after receiving D, was not stable, and shifted to another location, as the network is fine tuned on the new ABCDLMN sequence. It looks like the network is not reusing its previous state of L, which leads to the subsequent predictions of M and N, but rather to create a new network state that uniquely represents L and relearns new subsequent states and their associations to M and N. We agree that the stability and efficient reuse of previous network states closely relates to concept drift and catastrophic forgetting. \n\n> The paper could better contextualize how its framework integrates with or contrasts against recent neuron-centric interpretability methods like SAE circuits or representation engineering.\n\nWe propose to go beyond finding single neurons that encode for features, or finding lower-dimensional activities of neural subspaces, but rather look at raw population activities as a subject of interpretation. This is different from sparse autoencoders (SAEs) (Bricken et al. 2023), which trains an autoencoder to map neural activities of a hidden unit layer to a much larger number of intermediate hidden units while encouraging a sparse number of them to be active. In this way, the target hidden layer activity can be represented by a superposition of several individual neurons inside the SAE. However, SAE features often do not correspond clearly to human-interpretable concepts. The mapping between features and words is indirect, requiring additional steps such as VocabProj or MaxAct to identify associated tokens. This makes steering both complex and fragile. Interpreting SAE features remains an open challenge, and developing effective steering methods is still an active area of research.\n\nThis approach is also different from the representation engineering approach that captures the distinct neural activity corresponding to the target concept or function, such as bias or truthfulness. Then, it uses a linear model to identify the neural activity direction that predicts the concept under question or interferes with the network behavior.\n\nWith respect to intervention and steering, our method for steering and unlearning is simpler and more direct than that of SAEs. In SAEs, steering is typically achieved by clamping a feature to a specific value—either through a constant multiplier or a scaled value relative to its maximum activation (Templeton et al., 2024), which probabilistically influences the network’s output during text generation.\n \nIn contrast to our grafting, which overwrites the neural activity with a given chunk, applying a direction through representation engineering requires specifying the strength of the applied latent direction. This hyperparameter is not always straightforward to determine.\n\nWe will improve the contextualization in the updated version of the paper. \n\n> Beyond the included narrative schema experiments, could future work assess whether chunk activations correlate with human attention or comprehension on tasks like QA or summarization?\n\nThis is a great suggestion. In future work, we could relate the neural chunk activation with human perception of the sequences, such as attention or correct comprehension. This could indeed be a testing ground to find neural correlates of successful question answering and relate that to human cognitive tendencies when doing such tasks."}, {"review_id": "mPN6Omgwlf", "reviewer": "Reviewer_1jwn", "summary": "The authors propose the Reflection Hypothesis - the idea that the raw neural population activity of neural networks exhibits patterns that reflect the statistical regularities of the training data. To interpret these dynamics, the authors propose several cognitively-inspired 'chunking' methods to segment high-dimensional neural trajectories into recurring, interpretable units. These include Discrete Sequence Chunking (DSC) when $d$ is small (low-dimensional RNNs), Neural Population Averaging (PA) when $d$ is very large (transformers), and Unsupervised Chunk Discovery (UCD) when labels are unavailable. Interestingly, across architectures (RNNs and LLaMA-3), the authors demonstrate that these methods recover recurring internal states aligned with concrete and abstract input concepts. Furthermore, intervention-style experiments (e.g., grafting neurons to discovered chunks) demonstrate that some discovered chunks have a systematic causal effect on model behaviour, providing evidence that chunks are not just spurious features, but correspond to *functionally* meaningful internal representations.", "questions": "Clarification on chunk minimality: I'm curious if the discovered chunks are minimal, in the sense of being irreducible functional units. E.g., could the chunk for 'cheesecake' be decomposable into smaller sub-chunks ('cheese' and 'cake') with independent causal effects? \n\nJustification for PA methodology: the PA method assumes averaging hidden states across multiple contexts yields a meaningful, general 'chunk' for the concept. Could the authors maybe provide some evidence or justification for this assumption, especially given the context-sensitivity of internal representations in transformers? \n\nQuantitative evaluations: the paper presents compelling qualitative demonstrations to implicate chunks in the functional behaviour of larger models, but offers limited large-scale quantitative evaluation of chunk causal efficacy. Have the authors considered more systematic evaluation strategies - e.g., measuring the consistency of chunk-induced output changes across a large, diverse dataset? \n\nUnsupervised setting and chunk evaluation: in this setting, how do the authors determine whether learned chunks are semantically distinct or functionally redundant? Related to this, have the authors thought about any systematic way to assess the 'quality' of a chunk? Or what determines a 'good' chunk, e.g., semantically distinctness, functionally consistency, minimality?", "limitations": "I would like to see potentially some discussion on the context-agnostic assumptions underlying the PA and UCD approaches, or a broader discussion about this in the supplementary, perhaps.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**: \n- The paper addresses a central challenge in modern deep learning - how to interpret the internal structure and reuse of representations in large models. \n- Introduces a compelling conceptual shift in how we approach neural interpretability. Rather than focusing on individual neurons or attention heads, the authors argue that the dynamics of the neural population should be the primary unit of analysis. Crucially, the authors also challenge the dominant focus on linear decodability (e.g., probing), and instead emphasises functional and causal efficacy of the units of analysis. This reframing aligns interpretability research more closely with cognitive neuroscience, where population dynamics are a central unit of analysis. \n- Methodological breadth: the paper introduces three chunking methods for use across different settings (supervised vs unsupervised, low/high-dimensional settings), making the framework quite applicable to many scenarios. \n- The use of causal interventions (e.g., chunk grafting) demonstrate the chunks are not just correlated with behaviour, but causally implicated.  This kind of interventional evaluation strengthens the paper's empirical claims. \n- Architecture agnostic evaluation: chunking framework validated across diverse architectures, supporting the claim that chunk-like internal structure is a general emergent property arising from statistical regularities of the training data, rather than any specific model architecture.\n\n**Weaknesses** \n- Chunking methods are insensitive to temporal dynamics: only one of the three proposed methods (DSC) explicitly incorporates temporal information by merging adjacent symbols in the sequence. The other two methods are snapshot based, which effectively ignores the trajectory-level structure that is essential to sequential computation (especially in models e.g., transformers processing long sequences).   \n- Context invariance assumption for PA: assuming that a concept can be represented by a single population vector obtained by averaging across occurrences may be invalid in context-sensitive models (e.g., a concept like 'cat' could produce different neural activations depending on its context, and averaging across all instances could obscure the underlying structure). This compounds the issue of using static snapshots, since the temporal and contextual dependencies are not preserved in the averaged chunk.  \n- Evaluation: for the LLM case, there are only qualitative examples (e.g., for grafting and freezing) instead of quantitative metrics in the interventional test to causally implicate chunk identity in the model's functional behaviour. This makes difficult to assess whether the claim that chunks are functionally useful actually generalises to these larger models. Similarly, in the LLM case, for quantitative metrics (e.g., decoding in Fig 3), the evaluation setting is quite narrow (3 words), again making it difficult to assess whether the chunk behaviours are exceptions or consistent patterns that hold for larger models.  \n- Lack of generalisation analysis for chunks: Relating to point 2, while chunks are learned in a context-agnostic way, it is unclear that the chunks can be consistently reused across different contexts. I would be curious to see some evaluation of a learned chunk's (e.g., 'cheese') consistency across different contexts, perhaps by e.g., grafting the chunk across multiple contexts and quantitatively measuring that the impact across all contexts is consistent.", "was_revised": true, "final_justification": "The authors provided a detailed and thoughtful rebuttal that addressed many of my initial concerns. The new results on temporal dynamics, context sensitivity, and chunk evaluation significantly strengthen the empirical contribution. The additional grafting experiments and large-scale evaluations also lend support to the claim that discovered chunks are functionally meaningful.\n\nThat said, some limitations remain, particularly regarding the reliance on early layers, a focus on concrete concepts, and open questions about the robustness of chunk functionality across architectures and linguistic categories. These do not undermine the core contributions, but they do suggest the need for further work to fully establish generality.\n\nOverall, the paper introduces a conceptually interesting and potentially impactful framework, and the authors have made a compelling case for its validity. I maintain my positive recommendation.", "rebuttal": "We thank the reviewer for the very thoughtful and constructive feedback. We are encouraged by the recognition that the work “addresses a central challenge,” “introduces a compelling conceptual shift,” and demonstrates “methodological depth.” Please see our responses to the comments and questions below, focusing on clarification on the temporal dynamics, additional quantitative measures, and contextual sensitivity.\n\n> [W1] Chunking methods are insensitive to temporal dynamics ... \n\nWe would like to clarify that there is indeed a temporal dynamics extracted by both PA and UCD, but the paper only illustrates snapshots to show the smallest chunk granularity. PA is not limited by snapshots, Figures 18 and 19 in Supp. shows the temporal dynamics of PA before and after the temporal snapshot for the illustrated examples. Figure 20 in Supp. illustrates the temporal dynamics preceding and proceeding concepts for other examples, Figures 21 and 22, from pages 23 - 25, evaluate the extracted PA chunks with a temporal component. \n\nFor UCD, condensing neural activity into discrete chunks enables the DSC algorithm to identify underlying temporal chunk patterns. There are rich temporal patterns of chunk interactions—for instance, in the Emma corpus at layer 10, the chunk sequence 1145-136-1300-367 occurs 80 times, and the related pattern 1145-1300-367-367-367 appears 115 times—indicating that neural computations often span multiple tokens.\nWe quantify this by calculating the compression ratio, the number of tokens in the corpus divided by the number of extracted temporal chunks that span the same corpus. This suggests that beyond the level of tokens, there are also neural computation dynamics described by the chunks extracted. \n\n| Layer             | 5    | 10   | 15   | 20   | 25   | 30   |\n|-------------------|------|------|------|------|------|------|\n| Compression Ratio | 0.39 | 0.45 | 0.62 | 0.55 | 0.43 | 0.63 |\n\n\n> [W2] For grafting and freezing, quantification of whether the functionality of chunks generalizes to larger models?\n\nWe measured causal effectiveness by comparing how often a concept appears in LLM-generated text with and without intervention (grafting or freezing).\n\n### Grafting\n**Prompt:** “Hi, how are you doing?”  \n**N = 100**\n\n| Target Concept | Without Grafting | With Grafting |\n|----------------|------------------|----------------|\n| cake           | 1%               | 83%            |\n| cheese         | 0%               | 90%            |\n| cheesecake     | 0%               | 16%            |\n\n\n---\n\n### Freezing\n\n**Prompt:** “What is the name of a rich, savory food made from curdled milk, often aged to enhance its flavors?”  \n**Target concept to freeze:** cheese  \n**N = 100**\n\n| Condition    | Cheese Occurrence |\n|--------------|-------------------|\n| No Freezing  | 67%               |\n| Freezing     | 33%               |\n\n**Prompt:** “Name three most common desserts in Europe.”  \n**Target Concept to freeze:** cake  \n**N = 100**\n\n| Condition    | Cake Occurrence |\n|--------------|-----------------|\n| No Freezing  | 36%             |\n| Freezing     | 9%              |\n\nOverall, grafting was more effective than freezing, though both had an effect.\n\n> [W3] Similarly, in the LLM case, for quantitative metrics (decoding), the evaluation setting is quite narrow (3 words), again making it difficult to assess whether the chunk behaviours are exceptions or consistent patterns that hold for larger models. \n\nBeyond the three illustrated word examples, we extended our evaluation to the top 100 most frequent English words in a larger corpus (Figure 5) and applied the same quantitative metrics across four large models with distinct architectures. In addition to LLaMA-3, we tested Mamba (a state space model), T5 (an encoder-decoder model), and RWKV (a large-scale RNN). We also assessed the model’s ability to extract chunks corresponding to abstract structural schemas (Figure 4). Across these models, we found that the discovered neural chunks consistently and deterministically predicted concept occurrence in held-out test corpora, often achieving 100% decoding accuracy across a number of early layers ( see Supp. Figures 24, 25, and 26) .\n\n> [W4] Lack of generalisation analysis for chunks: ... it is unclear that the chunks can be consistently reused across different contexts, … e.g., grafting the chunk across multiple contexts and quantitatively measuring that the impact across all contexts.\n\nWe extended our quantitative evaluation to the ROCStories benchmark (Mostafazadeh et al., 2016), a narrative corpus of coherent story sentences, and assessed grafting consistency across context categories using the TREC Question Classification dataset (Li and Roth, 2002). From ROCStories, we sampled 2,000 sentences and expanded the existing PA extracted chunks by adding the 20 frequent words (≥5 characters) in the corpus.\n\nTo evaluate context-dependency, we sampled 50 prompts from each TREC category—ABBR, DESC, ENTY, HUM, LOC, and NUM—and measured the percentage success of grafting target concepts in each. To reduce computational load, we grouped grafting interventions into early (layers 1–9), middle (10–19), and late (20–29) layers, and compared concept occurrence probabilities between grafted and control generations.\n\n| TREC Category | Control        | Early Graf.     | Mid Graf.       | Late Graf.      |\n|---------------|----------------|------------------|------------------|------------------|\n| ABBR          | 14.9% ± 24.6%  | 55.9% ± 32.4%   | 30.8% ± 26.9%   | 18.0% ± 25.8%   |\n| DESC          | 15.6% ± 21.5%  | 49.0% ± 26.8%   | 28.1% ± 22.4%   | 20.7% ± 23.0%   |\n| ENTY          | 12.6% ± 21.8%  | 48.1% ± 28.6%   | 22.5% ± 24.1%   | 16.9% ± 22.9%   |\n| HUM           | 11.9% ± 20.0%  | 46.7% ± 28.6%   | 21.5% ± 23.8%   | 15.2% ± 20.7%   |\n| LOC           | 10.7% ± 19.4%  | 47.5% ± 28.6%   | 20.5% ± 22.4%   | 14.4% ± 21.2%   |\n| NUM           | 11.5% ± 18.4%  | 45.3% ± 27.0%   | 21.8% ± 21.4%   | 16.0% ± 20.9%   |\n\nGrafting concept-related chunks increases the likelihood of LLaMA-3 generating the target concept, with little variation across contexts. Grafting early-layer is more effective than in later layers.\n\n> [Q1] Any systematic way to assess the 'quality' of a chunk? Or what determines a 'good' chunk, e.g., semantically distinctness, functional consistency, minimality?\n\nEvaluation in the unsupervised setting was a concern for us, and we explored several measures to assess whether the algorithm achieved its intended outcomes.\n\nA good chunk should closely approximate the embedding. We compute the normalized cosine similarity between each embedding and its best-matching chunk in the dictionary, and ensure the similarity scores follow a reasonable distribution.\n\nWe also check semantic distinctness, i.e., each embedding should be similar to only a few chunks while remaining dissimilar to most others in the dictionary. Figure 27b shows the similarity between a single embedding and the set of learned chunks in layer 10 of LLaMA.\n\nAdditionally, we verify that chunk usage is reasonably distributed—that is, no small subset of chunks should dominate as the most similar matches across embeddings. This diversity in chunk identification is illustrated in Figure 27c. W\n\ne also visually inspect samples of extracted chunks alongside two matching embeddings and a non-matching embedding as a control. Example sets are shown in Figure 28. Further discussion on chunk evaluation is provided in Section E of the Supp.\n\n> [W2, Q2, L] PA assumes that averaging hidden states across multiple contexts yields a meaningful, general 'chunk' for the concept... Could the authors provide some justification for this assumption, given the context-sensitivity of internal representations in transformers? \n\nWe appreciate the reviewer’s insightful comment regarding the context invariance assumption underlying PA and UCD. Indeed, modern language models are known to exhibit context-sensitive internal representations, and a given word (e.g., \"bank\") may evoke distinct activations depending on its context (e.g., a river bank versus a financial bank). \n\nHowever, our goal is not to model context-specific usage, but rather to extract the core commonalities a concept elicits in neural computation activities across contextual variations —analogous to semantic prototypes in word2vec and GloVe.\n\nThis assumption on the algorithm then finds its empirical validation. We observe that PA and UCD-derived chunks correlate with many concepts. This suggests that for at least a subset of words, there exists a stable commonality despite contextual variations. And for other words, such as prepositions, which are contextually more nuanced, we would expect this method to be less effective. \n\nIndeed, we observed that the grafting success of PA-derived chunks depends on the type of word. Grafting proved especially effective for concrete concepts. For example, inserting the chunk for \"school\" increased its occurrence from 0.67% in control generations to 78.7% with grafting; \"Christmas\" rose from 0.33% to 74.3%, and \"family\" from 1.7% to 71.3%. For non-specific words such as prepositions and conjunctions, grafting PA chunks tends to be less effective: inserting the chunk for “if” only increased its occurrence from 6.7% to 7.3%, “at” from 7.7% to 11.7%, and “on” from 23.0% to 28.7%. This limited impact is often correlated with lower PA chunk quality, which is also reflected in the decoding evaluation metrics for these function words.\n\n> [Q4] Could the chunk for 'cheesecake' be decomposable into smaller sub-chunks ('cheese' and 'cake')... ? \n\nThese representations do not reduce to distinct functional subgroups. In the examples shown, over 99% of the neurons encoding cheese, cake, and cheesecake overlap, with only 11, 2, and 34 neurons uniquely encoding each, respectively. This suggests that the same neural population supports multiple meanings."}, {"review_id": "BCD5I82HA6", "reviewer": "Reviewer_dfCF", "summary": "This paper investigates the hypothesis, motivated by findings from human cognitive science, that artificial neural networks (ANNs) memorize recurring structural patterns in training data. The authors propose methods to identify such recurring patterns, or “chunks,” by clustering activation vectors. An activation vector $h \\in \\mathbb{R}^d$ is considered part of a chunk if its subvector $h_C$ lies within a threshold distance of a chunk centroid. For low-dimensional settings, chunk extraction is done by discretizing neuron activations into strings and forming a dictionary of recurring patterns. For large-scale networks, the authors use a supervised method, Neural Population Averaging (PA), to identify concept-specific chunks, and an unsupervised method that optimizes a dictionary to ensure each input is close to at least one entry. Through experiments on RNNs and transformer LLMs, they show that the extracted chunks correspond to interpretable and recurring input patterns, can be manipulated to affect model predictions, and capture hierarchical or contextual input structure.", "questions": "1. Please describe how the proposed chunking method compares to other concept discovery approaches (e.g., \\[1–6]). What are the conceptual differences and potential advantages?\n2. Provide a more formal and precise description of the Reflection Hypothesis.\n3. Clearly define the terms *grafting* and *freezing* within the paper.\n4. Explain the theoretical or empirical rationale behind the chunk clustering algorithm. Which theory, if any, guided the algorithm design choices?", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is generally well-written and clearly structured.\n* It presents a novel perspective on interpretability, drawing inspiration from human cognitive science, which adds an interesting conceptual framing to the work.\n* The proposed method has demonstrated utility in downstream applications, by enabling causal interventions on model outputs through chunk manipulation.\n\n\n**Weaknesses**\n\n* While the work aligns with the broader literature on concept-based interpretability and mechanistic interpretability, the related work section lacks coverage of key concept discovery methods beyond sparse autoencoders (SAE). Additionally, these competing methods are not included in the quantitative evaluation \\[1–6].\n* Most quantitative results represent an exploratory analyses rather than a rigorous, systematic evaluation.\n* There are several clarity issues throughout the paper:\n\n  1. The Reflection Hypothesis is introduced in a vague and imprecise manner.\n  2. It is not stated whether a single input can be associated with multiple chunks.\n  3. Key terms such as *grafting* and *freezing* are not properly introduced or formally defined.\n* The decision process behind the chunking strategies in Section 3.1 is not thoroughly explained. The overall methodology appears somewhat ad hoc and lacks strong theoretical grounding.\n\n**Minor Comments:**\n\n* Missing space after “(SAEs)” (line 69), “at the time of the concept-specific index” (lines 156–157), and “Figure9” (line 282), double period typo in “neural activities..” (line 234).\n* Figure 2a (line 258) reference does not point to the correct figure.\n* Figure 3 (left side): unclear how different tokenizations are represented. \n* Some variables are not defined or are ambiguous:\n\n  * $\\overline{h_C}$ (line 128) \n  * $h_1, \\ldots, h_d$ (lines 130–131)\n\n\nReferences:\n\n[1] Ghorbani, A., Wexler, J., Zou, J.Y. and Kim, B., 2019. Towards automatic concept-based explanations. Advances in neural information processing systems, 32.\n\n[2] Vielhaben, J., Bluecher, S. and Strodthoff, N., Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees. Transactions on Machine Learning Research.\n\n[3] Fel, T., Boutin, V., Béthune, L., Cadène, R., Moayeri, M., Andéol, L., Chalvidal, M. and Serre, T., 2023. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems, 36, pp.54805-54818.\n\n[4] Black, S., Sharkey, L., Grinsztajn, L., Winsor, E., Braun, D., Merizian, J., Parker, K., Guevara, C.R., Millidge, B., Alfour, G. and Leahy, C., 2022. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312.\n\n[5] Tamkin, A., Taufeeque, M. and Goodman, N., 2024, July. Codebook Features: Sparse and Discrete Interpretability for Neural Networks. In International Conference on Machine Learning (pp. 47535-47563). PMLR.\n\n[6] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J. and Viegas, F., 2018, July. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning (pp. 2668-2677). PMLR.", "was_revised": true, "final_justification": "I believe this paper makes a valuable contribution to the interpretability field by introducing a novel concept discovery method grounded in cognitive science. While I initially had concerns about the lack of discussion of closely related work, missing key definitions, and insufficiency of systematic analysis, I appreciate that the authors have addressed these points during the rebuttal. I am hopeful that these improvements will be reflected in the final version of the manuscript.", "rebuttal": "We thank the reviewer for the constructive suggestions. Below we address the reviewer’s comments, focusing on clarifying the literature, quantitative results, and supplementary definitions. \n\n> [W1, Q1] While the work aligns with the broader literature on concept-based interpretability and mechanistic interpretability, the related work section lacks coverage of key concept discovery methods beyond sparse autoencoders (SAE). Additionally, these competing methods are not included in the quantitative evaluation [1–6]. \n... Please describe how the proposed chunking method compares to other concept discovery approaches (e.g., [1–6]). What are the conceptual differences and potential advantages? \n\nWe thank the reviewer for pointing to related concept-based literature, for which we will add a discussion in the related works section. However, most of the mentioned papers study concept-based interpretability methods for image classifiers, sometimes assuming accessibility to a \"class\" label. In Ghorbani et al. [1], Fel et al. [3] and Kim et al.[6], concepts correspond to visual features in the image dataset (e.g. pixels of a wheel), and they study what part of the image influences the network's classification decision. Black et al. [4] define polytopes as linear regions in input space and study the geometry of the input space. Vielhaben et al. [4] define concepts as a linear subspace in the feature space, spanned by a number of basis vectors that is unbounded. Finally, while Tamkin et al. [5] learns a codebook of features, their method proposes training a model with codebook bottlenecks requiring significantly more compute than our methods. Moreover, we intend to interpret the raw activity of large models regardless of their training procedure or architectures instead of optimizing a new model.\n\nIn contrast, our work primarily studies network interpretability processing sequences of tokens. Our work examines how the internal structure of the network exhibits recurring patterns of entities. We reframe interpretability by connecting neural population trajectories to human chunking.This approach to interpretability was noted by reviewer 86W2 as “a fresh perspective beyond token-level or neuron-level probes”; fcFE “a novel framework for conducting future interpretability experiments”, ”grounded in cognitive science” which offers “ a human-in-the-loop approach”; 1jwn “addresses a central challenge”, and “Introduces a compelling conceptual shift” (1jwn).\n\nWhile we agree on the importance of these comparisons, the aforementioned differences make a quantitative comparison a non-trivial problem and out of scope for the rebuttal.\n\n> [W2] Most quantitative results represent an exploratory analysis rather than a rigorous, systematic evaluation.\n\nWe want to make a point about there existing recurring neural population activity patterns in sequence processing models, in response to concrete and abstract patterns in the input sequences, and these neural patterns influence the network behaviors. Due to the nature of this argument, it is not a message that can be easily conveyed by quantitative measures and demands ample examples to support. But we tried to strike a balance between exploratory and quantitative results by testing our claim across architectures (Mamba, RNNs, state space models, T5 in Figure 5, Supp. Figure 24-26); inventories (beyond simple examples to top 100 english words in Figure 5, Supp. Figure 21-24 and abstract POS tags in Figure 31), and corpus (from topic related sentences to sentences with an underlying abstract structure, to literary corpus, in Figure 5, 6). \n\nApart from that, we supplemented the evaluation of grafting effectiveness by quantitative evaluation to the ROCStories benchmark, sampling 2,000 sentences to extract neural population chunks corresponding to the top 20 frequent concept words (filtered to avoid short or overlapping terms) in addition to the existing concepts collected. To assess context-dependent effects of chunk grafting, we used prompts from the TREC Question Classification dataset, which includes six coarse-grained context categories. For each category, we sampled 50 prompts and evaluated grafting success by measuring the difference in concept occurrence probability between grafted and ungrafted (control) LLaMA-3 generations. For example, a +41% increase indicates that the target concept is 41% more likely to appear in the generating output with grafting compared to generating output without grafting.\n To reduce computational cost, we categorized grafting into early (layers 1–9), middle (10–19), and late (20–29) interventions. We found that grafting consistently increased concept occurrence across contexts, with early-layer grafting being the most effective. More details can be found in our discussion with reviewer 1jwn. \n\n| TREC Category | Early Δ   | Middle Δ  | Late Δ   |\n|---------------|-----------|-----------|----------|\n| ABBR          | +41.0%    | +15.9%    | +3.1%    |\n| DESC          | +33.4%    | +12.5%    | +5.1%    |\n| ENTY          | +35.5%    | +9.9%     | +4.3%    |\n| HUM           | +34.8%    | +9.6%     | +3.3%    |\n| LOC           | +36.8%    | +9.8%     | +3.7%    |\n| NUM           | +33.8%    | +10.3%    | +4.5%    |\n\n> [W3] The Reflection Hypothesis is introduced in a vague and imprecise manner.… Provide a more formal and precise description of the Reflection Hypothesis.\n\nLet the sequence be denoted as $\\( S = (s^1, s^2, \\dots, s^n) \\)$, indexed by $\\( I = \\{1, 2, \\dots, n\\} \\)$. A pattern in the input is defined as a recurring subsequence carrying the same concept \n$$P = (s^{i}, s^{i+1}, \\dots, s^{i+k-1}) \\subseteq S $$\nof fixed or variable length $k$, where the same sequence of tokens appears many times in \\( S \\).\n\nThe _Reflection Hypothesis_ posits that such recurring input patterns $ P $ are mirrored in the internal neural dynamics of the model, such that their corresponding neural activations \n$(\\mathbf{h}^{i}, \\mathbf{h}^{i+1}, \\dots, \\mathbf{h}^{i+k-1})$\nlie within a population-level chunk defined as a ball $\\overline{B}(\\overline{\\mathbf{h}}_C, \\Delta) \\subset \\mathbb{R}^d$\ncentered at a prototypical activation $\\overline{\\mathbf{h}}_C$ with radius $ \\Delta $. \n\nRegularities in the input sequence are reflected as localized activity patterns in the neural activities.\n\n> [ W4] It is not stated whether a single input can be associated with multiple chunks.\n\nA single token can be associated with multiple chunks, if this input has a different meaning as in different contexts. For example, the ‘ese’ token may elicit a different chunk in the network when it is used to parse ‘Che’+’ese’ from when it is used to parse ‘Ge’ +’ese’. \n\n> [ W5] Key terms such as grafting and freezing are not properly introduced or formally defined. \n\n__Grafting__ refers to the targeted insertion of previously learned chunks into a neural network’s internal computation. We modify its internal activations $ \\mathbf{h} $ at a specified layer $ l $ and neural subset $ C $ by grafting the chunk $\\mathbf{h}_C \\leftarrow \\overline{\\mathbf{h}}_C$ to replace the original neural subpopulation activities.  \n\n__Freezing__ refers to the operation of zeroing out the neural activations $ C $ that support the chunk. This is implemented by replacing the network activation with stored chunk representations at the corresponding neurons. At a given layer $ l $, we enforce: $\\mathbf{h}_{C,l} = 0.$\n\n> [W6] Explain the theoretical or empirical rationale behind the chunk clustering algorithm. Which theory, if any, guided the algorithm design choices?\n\n\nWe hypothesize that there exist recurring concepts as patterns in the input $P = (s^{i}, s^{i+1}, \\dots, s^{i+k-1}) \\subseteq S $, $\\( S = (s^1, s^2, \\dots, s^n) \\)$. These concepts contain tokens that elicit similar context and distributions (the definition of a word in the dictionary of a language). This follows similar logic to distributional semantics (e.g., word2vec, GloVe), where the mean of many contextual embeddings approximates a central semantic prototype. \n\nThe _Reflection Hypothesis_ posits that such recurring input patterns $ P $ are mirrored in the internal neural dynamics of the model, such that their corresponding neural activations \n$(\\mathbf{h}^{i}, \\mathbf{h}^{i+1}, \\dots, \\mathbf{h}^{i+k-1})$ lie within a population-level chunk defined as a ball $\\overline{B}(\\overline{\\mathbf{h}}_C, \\Delta) \\subset \\mathbb{R}^d$ centered at a prototypical activation $\\overline{\\mathbf{h}}_C$ with radius $ \\Delta $. \nWhen two sequences carry the same concept, we expect them to elicit similar activities for a subpopulation of neurons within the network.\n\nIn our larger-scale experiments, we observe that PA and UCD found neural population clusters and correlate well with concrete and abstract concepts in data from recurring words, to sentence structure, to POS tags, suggesting that for a substantial number of concepts, a stable signal emerges. We will supplement more theoretical grounding in the paper."}, {"review_id": "dUgG8AIFxZ", "reviewer": "Reviewer_Nmtb", "summary": "This paper introduces a novel interpretability framework based on the “Reflection Hypothesis,” proposing that neural networks reflect structured regularities in their training data. The authors develop cognitively inspired chunking methods to extract recurring neural entities from RNNs and LLMs. Empirical results across architectures demonstrate that these extracted chunks correspond to meaningful concepts and can causally modulate model behavior.", "questions": "- What are the practical advantages of this chunking method over existing interpretability tools such as SAE?\nFor example, SAEs have been used for unlearning [4] and steering [5]. Can chunking be used similarly?\n- Is the idea of chunking applicable beyond the language modality?\n\n\n[4] Applying Sparse Autoencoders to Unlearn Knowledge in Language Models.    \n[5] A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\nThe proposed Population Averaging (PA) method demonstrates superior performance in predicting concept occurrences compared to pretrained Sparse Autoencoders (SAEs), which is impressive. It is also a strong point that these results generalize across diverse model architectures, including Mamba and others.\n\n### Weaknesses\n- The core claim of the paper—the reflection hypothesis (i.e., that neural networks' internal representations reflect the structure of their training data)—does not appear particularly novel. Prior work such as [1] suggests that neural networks implicitly acquire syntactic structure; [2] shows that the geometric structure of SAE features reflects underlying data structure; and [3] demonstrates that cyclic structures in time-series data (e.g., weeks and months) are mirrored in internal representations. These studies collectively indicate that the reflection hypothesis is not new, and the contribution here may be more confirmatory than novel.\n\n- The result in Figure 1(a), where replacing the hidden state of an RNN alters its output, seems rather expected—what is the new insight here?\n\n- It is unclear why the authors use RNNs for toy tasks and Transformer-based LLMs for real-world scenarios. Why are there no toy experiments with Transformers to support consistency across architectures?\n\n- In line 258, the paper states: “Figure 2a shows normalized activations of the first 50 neurons (unsorted) across all LLaMA3’s layers for the word ‘cheese’.” However, the result shown is from an RNN, not LLaMA3.\n\n[1] Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs.    \n[2] The Geometry of Concepts: Sparse Autoencoder Feature Structure.    \n[3] Not All Language Model Features Are One-Dimensionally Linear.", "was_revised": true, "final_justification": "My concerns have been mostly addressed, so I will raise my score to 4.", "rebuttal": "We are thankful for the reviewer’s suggestions. We are glad that the reviewer found our work “impressive”, raises a “strong point” and “generalizes across diverse model architectures”. We address the rest of the reviewer’s concerns, focusing on novelty, pragmaticity, and generalization to other modalities. \n\n> [W1] Prior work such as [1] … [2] … [3] …. indicate that the reflection hypothesis is not new, and the contribution here may be more confirmatory than novel. \n\nThanks for pointing to the literature, we looked over these papers: [1] observed that structure onset corresponds to a sudden improvement of model linguistic ability. [2] analyze the low-D projected point cloud of SAE features and observe small-scale crystal-like, mid-scale modular, and large-scale power-law eigenvalue spectrum. [3] found multi-dimensional features in neural activities, and some of them correspond to circular structures in data, such as the day or the week.\n\nThese are empirical findings and do not provide a theory on what causes their observations, in fact, the authors call for investigation on the origin to explain why neural data is structured in such a particular way.\n\nThe reflection hypothesis postulates why we observe the phenomena in these papers, that is, a trained network may have its internal neural activities reflecting and mirroring the patterns in the data. This paper provides additional empirical support for the reflection hypothesis. We will integrate the suggested papers into the motivation for the hypothesis, and thank the reviewer for providing the suggestions. \n\nOn a separate point, we would like to clarify that we do not view the reflection hypothesis as the primary novelty of this work. \n\nThe main point we want to propose from this paper is a perception-grounded interpretability of artificial neural activities. Reviewer 86W2 noted it as “a fresh perspective beyond token-level or neuron-level probes”; fcFE “a novel framework for conducting future interpretability experiments”, ”grounded in cognitive science” which offers “ a human-in-the-loop approach”; 1jwn  “addresses a central challenge”, and “Introduces a compelling conceptual shift” and “aligns interpretability research more closely with cognitive neuroscience”.\n\nSpecifically, instead of visualizing low-dimensional features from compute-heavy SAEs, which depend on the choice of the dimensionality reduction algorithms, we can leverage the chunking property of visual perception to directly discern entities and patterns in a subset of high-dimensional neural dynamics. We propose methods to find these reflections and show the role of these reflections on the network's computation. We tested the hypothesis and the chunk extraction methods from small-scale RNNs, to large-scale transformers, state-space models, and other large-scale networks with diverse architectures. Such general applicability of our method suggests that interpretability can be studied independently of training SAEs, freed from the variability induced by the choice of dimensionality reduction methods, and independently of the neural architectures under study. \n\n> [W2] The result in Figure 1(a), where replacing the hidden state of an RNN alters its output, seems rather expected.\n\nThe hidden states are taken from the dictionary of state patterns extracted by the DSC method, while DSC extracts a dictionary of recurring population state patterns, we are not sure that this correlation contains a causation component and actually induces an alternation of network behavior in a predictable manner.  Therefore, we decided to graft the patterns to test whether it causally alters the network’s output in a predictable way. \n\nThis experiment ensured us that the population activity patterns extracted in the dictionary actually causally influence the network’s behavior, in a predictable way, tightly coupled with the existing pattern in the training sequence. The causal component also implies that one can encourage the network to reuse by artificially altering the network’s neural state and prevent catastrophic forgetting during fine-tuning (Figure 1b, Supp. Figure 13). This experiment also precedes our motivation to perturb the network by grafting the extracted population activity states in the much more complex transformer-type scenarios. \n\n> [W3] It is unclear why the authors use RNNs for toy tasks and Transformer-based LLMs for real-world scenarios. Why are there no toy experiments with Transformers to support consistency across architectures?\n\nRNNs are a much simpler architecture than Transformers, allowing us to design a network with a small number of neurons without requiring hyperparameter choices of Transformers (e.g., positional embedding, number of heads, MLP hidden size) and with a simpler training recipe. We then use standardized large-scale architectures for a comparison between network types including RWKV (RNN), T5, and Llama-3 (Transformers). To ensure consistency across methods, we checked the effectiveness of PA and UCD in discovering the RNN chunks, and discussed them in Supp. section B.3, and Figure 8, 9. With this experimental setup, we are trying to ensure a line of consistency while demonstrating the generality of our claim across diverse architectures.\n\n> [Q1] What are the practical advantages of this chunking method over existing interpretability tools such as SAE? For example, SAEs have been used for unlearning [4] and steering [5]. Can chunking be used similarly? \n\nTo clarify, we refer to unlearning as “freezing” and to steering as “grafting”, both of which we evaluate in our experiments. If the reviewer finds the terminology from the SAE literature more appropriate, we can update our paper accordingly.\n\nOur method for steering and unlearning is simpler and more direct than that of SAEs. In SAEs, steering is typically achieved by clamping a feature to a specific value—either through a constant multiplier or a scaled value relative to its maximum activation (Templeton et al., 2024), which probabilistically influences the network’s output during text generation.\n\nHowever, SAE features often do not correspond clearly to human-interpretable concepts. The mapping between features and words is indirect, requiring additional steps such as VocabProj or MaxAct to identify associated tokens [4]. This makes steering both complex and fragile. Interpreting SAE features remains an open challenge, and developing effective steering methods is still an active area of research [5].\n\nIn contrast, our population-level chunking method offers more interpretable units by design. Chunks are derived from recurring patterns in the model’s activations, grounded in actual input semantics. This enables more transparent and effective interventions, such as concept grafting or unlearning, without the need for specialized heuristics. We also evaluated our steering efficiency on a bigger corpus across multiple contexts, and verified that the extracted chunks can be an efficient way of steering network behavior. More detailed results in a tabular form can be found in our discussion with reviewer 1jwn. \n\nThe other is compute efficiency. The population averaging (PA) method is a post hoc, deterministic \nanalysis that requires no optimization or gradient computation. Neural activations are precomputed, and the only cost lies in computing prototype vectors and evaluating distances across a small number of deviation thresholds—operations that scale linearly with the number of tokens and the embedding dimension and take on a scale of minutes without a GPU, even on production-level LLMs. \n\nIn contrast, sparse autoencoders (SAEs) require multiple forward and backward passes through both encoder and decoder networks, typically over many training epochs. This results in orders-of-magnitude greater computational cost. PA, by contrast, offers a lightweight and accessible alternative for large-scale interpretability research.\n\nOur unsupervised chunk discovery method also remains computationally feasible. It runs efficiently on a single GPU. Given precomputed neural activations, it can be applied broadly to various architectures and datasets. A more detailed comparison of compute can be found in response to 86W2. \n\n> [Q2] Is the idea of chunking applicable beyond the language modality?\n\nYes. We tested whether chunking could apply to vision by steering the vision component of a multimodal transformer (CLIP), which encodes both images and text into a shared latent space. To obtain the neural chunks for visual concepts, we used images with colored text (e.g., the word “RED” written in red or green). We extracted shape chunks from images containing consistent shapes of written words (e.g., “RED”, “BLUE”, 10 categories in total), and color chunks from applying PA to 10 images of uniform colors in each color category (10 categories). After extracting the color chunks and shape chunks, we applied chunk-based interventions to CLIP’s image embeddings. Specifically, given a set of congruent images (image of the shape RED written in red color), we tried grafting the chunk of a target color (e.g., red → blue) or target shape (e.g., “RED” → “BLUE”) to see if CLIP’s image embedding can be steered to align closer with the target concept than the source concept. These were tested on 90 curated examples (where font color matched the word). We measured effectiveness using cosine similarity shift: $\\Delta = sim_{target, after} - sim_{source, after}$, where positive Δ indicates alignment with the target concept. We observed that grafting concept-encoding chunks in VLMs achieved 100% success for grafting shape chunks and 36.7% for color chunks. It suggests that neural subpopulations that encode visual concepts can be extracted and used to steer multi-modal models."}, {"review_id": "OpYSZ74Li5", "reviewer": "Reviewer_fcFE", "summary": "The paper introduces a new framework for interpretability, inspired by cognitive science. The main claim is that neural networks’ activations mirror the statistical properties of the distributions they model, termed the Reflection Hypothesis. The authors propose several dimensionality reduction algorithms to extract interpretable concepts from the activations and show that the concepts causally impact NN behavior.", "questions": "1. Could there be more discussion and exploration of other possible unsupervised algorithms for chunking? I would consider raising my score if there were interesting, novel results there.", "limitations": "no, please discuss possible limitations of when the reflection hypothesis might break down or when it might be difficult to extract insights from the activations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper introduces a novel framework for conducting future interpretability experiments that is grounded in cognitive science, rather than more mechanistic or representational approaches. The framework offers a human-in-the-loop approach to interpretability that may pair well with AI agents.\n- The paper shows the hypothesis across multiple network architectures and that it can lead to interpretable concepts, such as narrative structures in passages. These appear to be novel insights that can drive NN behavior.\n\n## Weaknesses\n- Some of the results seem to be reformulations of previously known results in interpretability, e.g., that \"Recurring chunk patterns emerge, such as similar activations for commas and adjectives in early layers\" is similar to the findings in [1] \n- Two of the proposed methods seem somewhat impractical. Discrete sequence chunking is mostly tenable when the NN’s hidden dimension $d$ is somewhat small. The authors then propose Neural Population Averaging, but this requires some hypothesis as to the concepts learned by the network. Specifically, the authors write that “this method relies on prior knowledge of recurring patterns in the input sequence”. Thus the main practical approach is unsupervised chunk discovery, but the only algorithm the authors give here is a variant of a k-means clustering algorithm. \n\n[1] https://karpathy.github.io/2015/05/21/rnn-effectiveness/", "was_revised": true, "final_justification": "Added discussion about more methods which suggested promising results", "rebuttal": "We thank the reviewer for the comments. We are reassured that the reviewer found the proposed framework novel. We address the rest of the comments below:\n\n> [W1] Some of the results seem to be reformulations of previously known results in interpretability, e.g., that \"Recurring chunk patterns emerge, such as similar activations for commas and adjectives in early layers\" is similar to the findings in [1]\n\nThanks for suggesting related studies. In this blog, Karpathy trained RNNs to generate character-level sequences such as Shakespearean text and Linux source code, and identified individual neurons that responded to features like quotation marks, if statements, or line endings. However, he also noted that the majority of neurons lacked clear interpretability, raising the question of how to understand model behavior at the population level when single-neuron activations are ambiguous. This early investigation inspired many subsequent studies focused on interpreting individual neurons, but progress stalled as it became evident that most neurons are polysemantic (Elhage et al., 2022) and do not correspond to unique, human-interpretable concepts.\n\nIn contrast to Karpathy’s individual neuron approach, our work suggests that concepts are not encoded by the activations of individual neurons, but rather by subpopulations of neurons whose joint activity visits similar regions in the high-dimensional population space. Instead of relying on neuron-wise interpretation, we propose methods to analyze and localize concept-encoding subpopulations—advancing toward interpretable representations at the population dynamics level.\n\n> [W2] Two of the proposed methods seem somewhat impractical. Discrete sequence chunking is mostly tenable when the NN’s hidden dimension is somewhat small. The authors then propose Neural Population Averaging, but this requires some hypothesis as to the concepts learned by the network. Specifically, the authors write that “this method relies on prior knowledge of recurring patterns in the input sequence”. Thus the main practical approach is unsupervised chunk discovery, but the only algorithm the authors give here is a variant of a k-means clustering algorithm.\n\nWe thank the reviewer for raising concerns about the practicality of our methods. We respectfully disagree with the notion that the approaches are impractical and would like to clarify several points. \n\nWhile Neural Population Averaging (PA) indeed relies on recurring patterns in the input (as we state), this is actually a strength, not a limitation. In our examples we also simply searched for the concepts by pattern matching using regular expressions. In many data domains practitioners already know the types of concept they are looking for, and abundant data are labelled by their keywords, abstract schema, intent, and there are tools to extract abstract structures such as pos tags or sentence parsing trees. In a range of corpora, PA allows us to isolate and interpret these reusable representations without retraining the model. Our grafting experiments show that these prototypes generalize systematically (see discussion with reviewer 1jWN for details).\nWe will expand our supplementary material to provide more technical detail on our implementation and clarify how pragmatic the proposed methods are. (Also see discussion with reviewer 86W2 for compute estimate)\n\n> [Q1] Could there be more discussion and exploration of other possible unsupervised algorithms for chunking? I would consider raising my score if there were interesting, novel results there.\n\nYes, we have explored a couple of alternatives. \n\nWe also experimented with SAE-like variants for chunk discovery, using a reconstruction loss with continuous hidden activations and tying the decoder weights to the transpose of the encoder weights. However, this approach yielded too many hidden units to interpret meaningfully, and the continuous, additive nature of the reconstruction did not align well with our definition of chunks. To improve interpretability, we attempted to binarize the hidden layer, but this variant struggled to train effectively. Moreover, even with binarized activations, the additive reconstruction process still diverged from our intended definition of neural chunks as discrete, reusable population patterns.\n\nWe explored several variations of the unsupervised chunk discovery (UCD) loss. One variation is maximal cosine similarity + sparsity regularization, which yielded to us an average of 0.54 cosine similarity from the dictionary. We also tried instead of the max, a softmax weighted similarity with a sparsity regularization, and that yielded to us the best case of 0.58 cosine similarity distribution in the test data. Removing the sparsity regularization and simply using the max gives us 0.60 average cosine similarity. For each configuration, we also evaluated the quality of the chunks by semantic distinctness, identification frequency, identification diversity (illustrated in Figure 27 in Supp.), and visualized and inspected them (Figure 28). The additional component did not contribute to learning a better dictionary, so we decided to report the simplest and the most effective method. We also studied whether the size of the dictionary matters and did not observe a significant effect of K (Supp. E.5 Figure 32).\n\nBut UCD reveals very interesting computational structures in LLMs that call for a closer look in future investigations. Such as the connectivity between chunks across layers: chunk 412 in layer 1 elicits chunk 1621 or chunk 1251 in layer 2 (L1:412 → L2:1621) (L1:412 → L2:1251), but not other chunks in layer 2. This allows the extraction of a graph of neural chunk interaction across all 32 layers of llama. We found that some layers act as hub layers, such as layer 31, where certain chunks receive input from many chunks in the preceding layer, aggregating diverse upstream activity. In contrast, other layers—like layer 30—function as expansion layers, generating a large number of distinct chunks to differentiate activity patterns inherited from the previous layer. Interestingly, we also observed a persistent chain of chunk activations spanning from the first layer to the final layer of LLaMA. This suggests that a specific chunk—likely originating from the input and activated in the first layer—continues to activate a unique chain of chunks across all 32 layers. This behavior may reflect a positional marker in the sequence, such as a beginning-of-sentence or end-of-sentence token. Such a chunk interaction graph gives a discrete description over the neural population interaction mechanisms inside the large models, and can be a subject under investigation and perturbation in diverse usage cases. \n\nBesides that, UCD reveals ample amounts of temporal patterns of chunk interactions, such as in the Emma corpus, chunk 1145 136-1300-367 occurs 80 times, 1145-1300-367-367-367 occurs 115 times. To quantify these extracted patterns of temporal computation dynamics, we computed a compression ratio—the number of input tokens divided by the number of extracted population-level patterns covering the same corpus for each layer. \n\n| Layer | Compression Ratio |\n|-------|--------------------|\n|   5   | 0.39               |\n|  10   | 0.45               |\n|  15   | 0.62               |\n|  20   | 0.55               |\n|  25   | 0.43               |\n|  30   | 0.63               |\n\nThis suggests that neural activity exhibits structure not only at the token level but also across multiple tokens, indicating the presence of higher-order, compressible computation dynamics.\n\nWhile we are still actively developing improved versions of this algorithm, we include our current best iteration in this paper. We believe this method offers a promising starting point for identifying relational entities from neural activity and provides a foundation for future work aimed at uncovering the rules governing neural population interactions in task-specific computations. \n\n> [L] please discuss possible limitations of when the reflection hypothesis might break down or when it might be difficult to extract insights from the activations.\n\nThe reflection hypothesis may fail for networks that are insufficiently trained. This is evident in the RNN experiment with sequences such as ABCD and CDAB. When undertrained, the network does not differentiate between the shared subsequence CD in the two contexts. Instead, it learns a shallow prediction pattern for AB and CD, a tendency that is also observable in its neural trajectories. However, in these two cases, CD should lead to different predictions: in ABCD, it precedes E, whereas in CDAB, it is followed deterministically by A. The network gradually acquires this contextual sensitivity over the course of training. Similarly, we observe that when we make the generative sequence with more nested hierarchies. As subsequences become more contextually dependent, the network responds by developing a greater number of distinct internal chunks. Through training, it gradually learns to reflect the underlying generative structure of the sequence, leading to a richer internal state space that disambiguates subtle contextual differences—representations that do not emerge without sufficient training."}]

Author rebuttal:

Meta-review: In this submission, the authors challenge prevailing attitudes towards the uninterpretability of neural networks by arguing that they exhibit patterns that follow regularities in the training data. The authors propose the Reflection Hypothesis - the idea that the raw neural population activity of neural networks exhibits patterns that reflect the statistical regularities of the training data. Building on this insight, the authors use "chunking", a cognitively-inspired method to extract concepts in three ways: discrete sequence chunking (DSC), population averaging (PA), unsupervised chunk discovery (UCD).

Initial reviews on this paper were mixed, but after discussion, all reviewers converged to support acceptance. Reviewers generally praised the importance of the problem and the novelty of the approach.
Decision: Accept (poster)