Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences
Authors: Shuchen Wu
Mirko Thalmann
Peter Dayan
Zeynep Akata
Eric Schulz
Venue: ICLR 2025 Poster (2025)
Keywords: Abstraction
Chunking
Cognitive Science
LLMs
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: The paper presents a hierarchical variable learning model (HVM) that efficiently abstracts patterns in sequences, outperforming standard compression methods and large language models in mimicking human memory and generalization.

Abstract: Humans excel at learning abstract patterns across different sequences, filtering out
irrelevant details, and transferring these generalized concepts to new sequences.
In contrast, many sequence learning models lack the ability to abstract, which
leads to memory inefficiency and poor transfer. We introduce a non-parametric
hierarchical variable learning model (HVM) that learns chunks from sequences
and abstracts contextually similar chunks as variables. HVM efficiently organizes
memory while uncovering abstractions, leading to compact sequence representations.
When learning on language datasets such as babyLM, HVM learns a more efficient
dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence
recall task requiring the acquisition and transfer of variables embedded in sequences,
we demonstrate HVM’s sequence likelihood correlates with human recall times. In
contrast, large language models (LLMs) struggle to transfer abstract variables as
effectively as humans. From HVM’s adjustable layer of abstraction, we demonstrate
that the model realizes a precise trade-off between compression and generalization.
Our work offers a cognitive model that captures the learning and transfer of abstract
representations in human cognition and differentiates itself from LLMs.

Published as a conference paper at ICLR 2025
BUILDING, REUSING, AND GENERALIZING ABSTRACT REP- RESENTATIONS FROM CONCRETE SEQUENCES
Shuchen Wu
Mirko Thalmann Institute for Human-Centered AI
Helmholtz Munich
Max Planck Institute for Biological Cybernetics shuchen. wu@tuebingen. mpg.
Helmholtz Munich mirko. chailammmehheltttottz-mmmaic...
Peter Dayan
Zeynep Akata Helmholtz Munich
Department of Computational Neuroscience Max Planck Institute for Biological Cybernetics dayan@tuebingen
Technical University Munich
de
zeynep.. akata@he Imholte --mmmin mounich... de
Eric Schulz
Institute for Human-Centered AT
Helmholtz Munich
eric. selulscculz0nn. the www.shhwwwweeeoeeeeeeeeeeeemeeee -oaittzz-unichhee
ABSTRACT
Humans excel at learning abstract patterns across different sequences, filtering irrelevant details, and transferring these generalized concepts to new sequences. In contrast, many sequence learning models lack the ability abstract, which leads to memory inefficiency and poor transfer. We introduce a non-parametric hierarchical variable learning model (HVM) that learns chunks from sequences and abstracts contextually similar chunks as variables. HVM efficiently organizes memory while uncovering abstractions, leading sequence When learning on language datasets such as babyLM, HVM learns a more efficient dictionary than standard compression algorithms such as Lempel-Ziv. recall task requiring the acquisition and transfer of variables embedded in sequences, we demonstrate HVM's sequence likelihood correlates with human recall times. contrast, large language models (LLMs) struggle to transfer abstract variables effectively as humans. From HVM's 's adjustable layer of abstraction, we demonstrate that the model realizes a precise trade-off between compression and generalization. Our work offers a cognitive model that captures the learning and transfer of abstract representations human cognition and differentiates itself from LLMs.
sequence representations.
a sequence
INTRODUCTION
Philosophers traditionally view abstract ideas as formed by identifying commonalities across experiences, distilled from concrete impressions grounded in perception (Kant 1998; Fichtee 2005 STERN 1977). Psychologists suggest that
Abstraction plays key role in intelligence (Konidaris) 2019)
abstraction arises from personal experiences, such as forming the concept of "whiteness" by observing various white objects (Yee) 2019 Barsalou 1999). Abstract concepts are thought to build on concrete concepts and on top of the previously learned abstractions, thereby varying complexity (Cuccio) & Gallese 2018 Van Oers 2001: Collins & Quillian| 1969 Piaget 1964). The ability to abstract, which is often seen as a human- -ppecfic trait, enables reasoning, generalization, and problem-solving in novel contexts (Ohlsson & Lehtinen 1997 Dehaene et al. 2022 Duncker 1945).
We hypothesize that the world contains patterns across scales of time and abstraction. Intelligent agents-facing sequences with nested hierarchical structures- need model these structures store, process, and interact in such environments. As a rational strategy, intelligent agents shall characterize the temporal structure via chunking and characterize the abstract structure via identifying different

---

Published as a conference paper at ICLR 2025
items that play similar roles. To explore these operations, design generative model that produces sequences nested with hierarchical structure and propose an approximate recognition model that conjunctively learns to chunk and abstract.
We go beyond previous proposals that introduce chunking as mechanism for learning to compose complex structures from elementary perceptual units (Gobet al., 2001 Miller 1956 Wu et al.] 2022) and propose a cognitive model that combines chunking and abstraction in one single system. The model abstraction in two key ways: first, by identifying shared features facilitate efficient pattern retrieval; and second, by categorizing different sequential items that appear in the same context, much like variable a computer program. These paired mechanisms enable the model to parse sequences into chunks and form abstract patterns based on both concrete and previously learned abstractions, allowing for compact representations while revealing recurring patterns at the level of abstract categories. Consequentially. increasingly complex abstract patterns can be discovered layer by layer.
We first demonstrate the benefits of abstraction in memory efficiency and sequence parsing by comparing our algorithm with previous chunking models and other dictionary-based compression methods. Then, we show that the model exhibits human-like signatures of abstraction in memory experiment requiring the transfer of abstract concepts. In the same experiment, we contrast the model'ss generalization behavior with large language models (LLMs). Additionally, connection between abstraction level and abstract concept transfer by varying the level of abstraction parameter in the model. Our work offers cognitive model that captures the learning and transfer of abstract representations in human cognition and differentiates itself from the behavior of artificial agents.
demonstrate the
Inilial invertory
Forming stable combinations objects
way
Atomic
Catsgories synthesize sxisting objects
Observational sequines:
Categorias
observed
Figure 1: Generative Model. The inventory is initialized with a set of atomic units, which randomly combine into objects. Some objects are randomly selected to become categories. Categories of existing objects have similar interaction properties. The dashed arrow points from those objects to their corresponding categories. These categories further combine with cxisting objects. An observational sequence is composed of objects randomly sampled from the existing inventory. Categories are latent and never explicitly observed but are manifested one of the denoting objects.
GENERATING SEQUENCES WITH OBJECTS THAT CONTAIN HIERARCHICAL ABSTRACT
STRUCTURES
Hierarchical structures have been long presumed to characterize natural sequences - such as molecules composed of chemical elements the hierarchical nature of language (Abler 1989 Sportiche et al. 2013; yon Humboldt & Losonsky 1999). To study models respect hierarchical structure. we take standard approach starting from conformative generative model (Teh) 2006 Milan et al.). We design probabilistic generative model that generates sequences by sampling objects from an inventory which contains a hierarchical structure.
The generative model creates an inventory of recurring objects over d iterations of expansion. As illustrated Figure[[ [ the inventory starts with a set of atomic units A. On each iteration, a novel object category is created equiprobably. A new category (graded node) is created by pointing to random selection of objects from the inventory up to the moment (objects are treated disjunctively). A new object is created by concatenating random selection of pre-existing objects categories from the existing inventory. After the inventory has been expanded up to the d-th iteration, the independent occurrence probability in the sequence is sampled from flat Dirichlet distribution
2

---

Published as a conference paper at ICLR 2025
B(a) I P(a.)al). a, l and assigned to each object in the inventory. Similarly, a probability sampled from a flat Dirichlet to assign the occurrence probability of the objects within each category. More details can be found in the Appendix A.2
((p(ct)....,p(c a)): a1, "d.......jaa... (la))
|Al
To create an observational sequence, objects are randomly drawn from the inventory one after another with the assigned probability until they reach the desired sequence length. If the sampled object contains an embedded category, one of the objects corresponding to the category is sampled. This process is repeated recursively until all categories are replaced by specific objects, resulting in sequence of discrete atomic units.
3 LEARNING ABSTRACTIONS FROM SEQUENCES
We ask what computational principles could help an agent discover objects and categories from such observational sequences without supervision. We propose that two mechanisms suggested by the cognitive literature are vital: chunk proposal and variable discovery. Chunking concatenates learned objects and forms new ones; variable discovery groups chunks with similar interaction properties into a category. We build on top of the hierarchical chunking model (HCM) (Wu et al., 2022), which learns belief set B of chunk dictionary C from discrete sequences. We expand the model so that it can also learn variables while improving the memory efficiency of the model. By identifying stably recurring entities as chunks and grouping similar entities into categories as variables, the agent can learn a structured inventory of identifiable patterns and use these patterns as entities to parse the sequence, leading
more compressed factorization of perceptual sequences.
HVM learns a belief set B that contains both a dictionary of chunks C and variables V. The variables proposed abstract entities based on the transition and marginal counts. As shown in Figure 2h. each variable E V denotes a set of chunks (lv) = =fcsh Cj E C. The model also learns the probability of each chunk that variable denotes Vu E V. Ej P(u c2) = - e E C.
Parsing the sequence one chunk at a time A fundamental feature of the model is to parse sequences in chunks as the basic cognitive units (Miller 1956). Along with each parse f. the transition counts between the previous cr, and next chunks en are recursively updated: Ti 1) Tip (1) + l ex. = cn. (+) = 1 if the argument is true and O otherwise) and along with the identification frequency of each parsed chunk (t + 1) = (t) - [i = c]. When modeling human behavior, each entry of M and T' multiplies with a memory decay parameter per parsing step. The probability of observing sequence of parsed chunks C12 ...,.. ..... eN then becomes P(c1, 02, cN) = I Pe(e)).
To parse a sequence, HCM iteratively chooses the biggest chunk amongst its learned dictionary C consistent with the upcoming sequence. The end of previous parse initiates the next parse. As the dictionary size (C] increases, searching for the biggest consistent chunk becomes computationally expensive (Schreiber al., (023). In HVM, we introduce one notion of abstraction commonalities amongst memory items to organize memory structure and speed up chunk retrieval during parsing. Memory items the learned dictionary are organized into a hierarchical parsing graph that connects chunks with their common prefixes. Hence, all children chunks are different except for sharing a common prefix from the parent. The parsing graph arranges the chunks in C into prefix Trie structure (Figure 2p), reflecting the cue-based, content-addressable nature of memory retrieval (Cunnings & Sturt 2018 Dotlačil [202]] Anderson 1974). This design reduces search time to retrieve a chunk as arranged in the parsing graph, commonly used in predictive text or auto-complete dictionaries to speed up search steps (Fredkin 1960). At every parsing step, HVM identifies the deepest chunk in the parsing graph that is consistent with the upcoming sequence. The end of the previous parse initiates the next parse. The search process would take the time complexity of O(D), scaling with the depth D of the tree compared to HCM time complexity of O(IC)) scaling with the size of C. The appendix shows a guarantee to reduce the number of parsing steps for chunk retrieval A.4
finding
Learning chunks From parsed sequences, the model estimates the occurrence probability of chunk Mi via the entries of M as Po(ci) Σeyec M, The model starts with the null hypothesis Ho that all
consecutively parsed chunks c, and cn are statistically independent P(c,, cR) = P(cr,) If significant correlation is found between consecutively parsed chunk pairs (with p = 0.05), the null
3

---

Published as a conference paper at ICLR 2025
Representation graph
Parsing graph
Common prate
+
B CUV
Sequence:
Update marghnl and rans mant
Parsing (concrete)
Parsing (abstract)
d.
Propose variables between correlated chunk pairs
Propose chunks based on correisted transition entries
Expand chunk inventory
Covessn aneadiooncci
Expand ropresentation graph
relloded in Iurailon
Concanaration
new
Expand parsing grach
C +
Expand
impresettaion graph
Expand parsing graph
HVM builds up representation graph and parsing graph. A representation graph contains the learned chunks (nodes with solid colors) and the contained variables (nodes with gradient colors) V. Black arrows denote concatenation. Gray and dashed arrows point from chunks of the same category to the variable node that denotes this category. b. Abstraction organizes memory in parsing graph. Nodes shaded by gray are abstract chunks being the common prefix intersection of its children. During parsing, to allocate to the matching chunk in C, the model starts searching from the root of the parsing tree and traverses to the deepest node consistent with the upcoming sequence content. c. Upon completing each sequence parse, HVM updates counts on chunk frequencies and transitions and proposes inventory expansions, enabling layer-wise discovery of recurring patterns iteration, from specific to the more abstract. d. HVM proposes variables amongst correlated consecutive chunk pairs. e. Chunk proposal.
Figure 2:
hypothesis is rejected, and the pair is concatenated into chunk, cr. cn, which is then added to the dictionary C as a new entry. Upon creating each new chunk, an edge from the left parent chunk (sharing the same prefix) connects the newly created chunk in the parsing graph. When no parent chunk can be found, an ancestor node is created in the parsing graph, usually the sequence's atomic units. We prove in the SI section A.3 that under restricted conditions in which the sub-objects constituting the object in the ground truth are parsed faithfully by the model as sub-chunks, then these sub-chunks will be eventually proposed to concatenate into one chunk.
Learning variables A variable denotes distinct observations appearing in the same context (here defined distinct chunks sharing preceding and succeeding chunks). Given consistent parsing of subchunks as object combination components in the generative model, the true chunks that belong the same category will necessarily appear in the preadjacency and postadjacency entries (more explanation in SI AI.).. HVM proposes variables amongst the significantly correlated chunks (p S 0.05) in the transition matrix consistent with this structure. For example, consider the case that the sequence includes A-ABC, A-DC, and ABC-ED, DC-ED. This suggests that ABC and DC play similar roles relative A (coming after) and ED (before). Therefore, ABC and ED can be proposed
variable V represent the abstract property that captures the distinct entities. V denotes ABC and DC, and the model identifies either ABC or DC is identified during parsing, as illustrated in Figure and Figure 7 in SI.
Among the correlated consecutive chunk pairs, the model identifies all eligible transitions by intersecting the post-adjacency columns with the pre-adjacency rows, proposing set of chunks E(u) C₁ C2.... represented by new variable v. This variable proposal is accepted if the set of chunks that satisfy the condition Tmin F(u)] < Tmaz and Ee, M(a) 2 freqr- A variable
4

---

Published as a conference paper at ICLR 2025
denotes set of chunks E() and is identified if any of the chunks that it denotes is parsed (Figure 2 Together, the common preceding chunk, in conjunction with the newly proposed variable, followed by the common succeeding chunk, is concatenated and proposed as novel chunk cr. @ en add to the inventory C. An edge that connects ez to the new chunk ez @ v d en to be added to the parsing graph, illustrated in Figure 2d. Two variables and succeeding chunks. During parsing, a chunk is consistent with the sequence if any of its included variables contain denoting chunk that consistent with the sequence. Upon identifying a chunk during parsing, the marginal and transition count for both the chunk and the immediate variables that the chunk belongs to is incremented.
2)).
merged into one if they share the same preceding
Representation cleaning Upon completion of each sequence parsing, the model uses correlations propose new variables and chunks. It then uses the expanded inventory for the next parse. Unused variables and chunks are deleted upon the completion of each parsing iteration.
4
RESULTS
b.
d.
Coding Efficiency
107-
S00s0ric LAcihoo lights)
1500
Didtoran entrica sequance
Prosing Starch Stops
2250
Pering
750
100
50
230
MCW
MVM 1275
01
MCN MWN 1278 GT
2500
4000
6000
9000
10000
heeachhccl
Nerarchical
SOMMDIFCE longe)
Figure 3: a. The effect of hierarchical memory structure on parsing search steps. b. Comparison between models in length by the generative model. c. Model comparison based on sequence likelihood. d. Model comparison based on coding efficiency. Example model comparison with sequence length [S] 1000, nested hierarchy depth d = 30, atomic set of size |A| 10.
sequence length after parsing. GT denotes the ground truth sequence
4.1 MODEL EVALUATION
HVM is an approximate inverse of the generative model, as in practice, learning the ground truth hierarchical patterns that generate observed data is nonidentifiable problem (Post) 1946 Greibach 1968). Therefore, we use a set of measures for model evaluation, focusing on parsing search steps, sequence length, sequence negative log-likelihood, and encoding efficiency. We trained models sequences generated by the hierarchical generative model until convergence. For each iteration, the models parse the entire sequence using the dictionary updated from the previous iteration and propose novel chunks/abstractions upon completing the parse. While HVM learns both chunks and variables per iteration, HCM learns only chunks and no variables. Previous research has related human memory in specific tasks to lossy compression (Nassar et al.) (2018). with abstraction playing a crucial role in successful learning and transfer in sequence memory recall tasks. For reference, we also compared the models with 1.778,, an off-tf compression algorithm underlying compression schemes such as GIF and PNG, which also parses sequences into chunks and builds dictionary to compress sequence data (Ziv & Lempel 1978). Comparisons shown in Figure
Organizing chunks in a parsing graph based on common prefix reduces parsing search steps We compared HVM with HCM, which does not organize memory into a hierarchical parsing graph. Figure 3 a. shows that organizing chunks a parsing graph dramatically reduces the average number of parsing search steps, i.e., the number of search steps needed for the model locate the biggest chunk per parse. This search step now scales with the depth of the prefix tree compared to the size of the inventory in HCM. More discussions can be found in A.4
5

---

Published as a conference paper at ICLR 2025
Discovering bigger recurring patterns We compared HVM with its counterpart, HCM, which does not learn variables at the end of the learning iteration. Figure 3 compares sequence length [W] after parsing. HVM transforms the sequence [S| into code with much smaller size [W] compared to HCM and 1.2.78 and is on par with the ground truth (GT). Learning variables that denote distinct chunks occurring in similar contexts helps HVM learn bigger patterns underlying the sequence the description level of these variables.
Parsing sequences with higher likelihood Figure 3 compares the negative log likelihood log P(S) II P(c+)) of the parsed sequence upon convergence. HVM parses sequences more efficiently, as variables recur more frequently in sequences than their representing chunks, and therefore, the model that learns variables parses the sequence with higher likelihood.
Encoding more efficiently In many applications, the size of the dictionary affects compression efficiency (Navarro & Mäkinen, 2007: Ferragina & Manzini, 2005). We compare the encoding efficiency, i.e., the ratio between the dictionary size (number of entries) and the original sequence length S. Figure 3d shows the relation between the ratio and the sequence length. To encode sequence of the same length, LZ78 creates a bigger dictionary than HVM and HCM. Models that harness the embedded hierarchical structure in sequences encode sequences more efficiently and learn a smaller dictionary to encode the same sequence length.
4.2 LEARNING FROM REAL-WORLD SEQUENTIAL DATA
Data Domain
Model
Compression Ratio
NLL
Coding Elficiency
LZ78
0.38
2837.50
0.34
CHILDES (MacWhinney, 2000
HCM
0.51
2783.71
0.05
HVM
0.36
1953.01
0.06
LZ78
0.39
3136.67
0.37
BNC BNC Consortium 2007
HCM
0.65
3591.60 60
0.06
HVM
0.50
3108.33
0.08
LZ78
0.39
3156.61
0.37
Gutenherg (Gerlach & Font-Clos. 2020
HCM
0.69
3770,36
0.06
HVM
0.54
3252.84
0.12
LZ78
0.41
3395.09
0.39
Open Subtitles
[Lison Tiedemann, 2016
HCM
0.74
4151.89
0.07
HVM
0.63
3764.48
0.07
Table 1: Model comparison on alternative sequences. NLL stands for negative log- ligeliill
Going beyond artificially generated sequences, evaluate HVM, HCM, and LZ78 in text domains from the BabyLM language dataset, which contain text snippets from a collection data domains such conversational child-directed speech, narrative stories, and other linguistically rich environments Warstadt et al.] (2023). For each data domain, we took random snippets of 1000 characters and calculated evaluation metrics, including compression ratio (the number of tokens before compression divided by the number of tokens after compression), sequence complexity (negative log-likelihoo)) and compression efficiency (the length of the compressed sequence divided by the number of dictionary entries).
As shown in table I LZ78 performs well in terms of compression efficiency, which expected given its design purpose. However, the HVM model exhibits notable advantages when considering alternative evaluation metrics. Specifically, for text across the four data domains analyzed, cognitive models that incorporate the hierarchical structure of data outperform traditional compression methods in terms of encoding efficiency. This because LZ78, which does not make strong assumptions about sequence structure, tends to create redundant entries in its dictionary. While this redundancy reduces the sequence length after compression, it introduces many infrequently used entries. some data domains, HVM outperforms LZ78 in compression ratio and negative log likellhooood well. Amongst all domains, HVM compresses the sequence further and parses the sequence with lower complexity than HCM.

---

Published as a conference paper at ICLR 2025
a.
Training
Tranefer
reuse
Variable
B X
B
40 trials
Group
B
B
trials
Control
D B
E
40
b.
0.5
Humeen Training
Recal Time (Human)
16000
R- 0.86, 222-16.
d.
0.4
Human Transfer
Recal Time (Human)
8500
0.7,7
R-Squared
0.4
12000
R-Squared
0.3
8000
0.3
0.2
7500
0.2
7000
0.1
6500
HCM HVM
10
20
HCM
HVM
Model
Sequence Likelihood (HVM)
Model
Sequence Likellhood (HVM)
Group
Trial
Figure 4: a. A sequence memory task that demands transferring a variable x from the training to the test block. b., c.. d., e., Complexity (negative log-likelihood) progression of HVM correlates with human memory recall time in the training (b., c.) and the transfer block (d., e.). Memory decay parameter 8 0.996. Marker size increases with trial number.
4.3 HVM RESEMBLES HUMAN SEQUENCE MEMORIZATION AND TRANSFER
We take the relation between transfer to learning abstractions to the setting of a memory experiment that demands learning and transferring variables (Wu et al., 2023). We ask whether HVM resembles aspects of human learning while enjoying the advantage of learning interpretable dictionary.
In the experiment, 112 participants were instructed to recall a sequence of presented colors from memory (Figure a a). Participants were randomly assigned to variable group and a control group. During the training block (40 trials), the control group was instructed to remember a fixed sequence BADF DCBF FEDB for each trial (each letter denotes distinct color). In contrast, the variable group was instructed to remember sequences overlapping in 9 of the 12 colors: BXDF DXBF FXDB. A variable X, however, appeared at serial positions 2, 6, and 10, which could be occupied with the letters A, C, or with equal probability for each trial. After the training block, both groups were then tested on the transfer block. The transfer sequence overlapped with the training sequence only at the variable positions: DXBF FXDB BXFD. The time participants took to recall the entire sequence for each trial was recorded. Previous studies suggest human recall time relates logarithmically to the perceptual predictability of the sequence (Carpenter & Williams 1995; Anderson & Milson 1989: Smith & Levy 2013 Elman 1990p. We compare human recall time the model's negative log- probability (likelihood) evaluated on the instruction sequence given.
To simulate the sequence likelihood for trial n, we used the instruction sequences up trial n This allowed the HVM to learn in an online manner, i.e. abstractions and chunks were proposed upon each parsing completion. For comparison purposes, we also simulated HCM with the same parameter setting, and an associative learning model (AL) that updates first order transitions between atomic sequential units. The sequence likelihood was evaluated as log P(S) = - log P(c,) Ik= P(c//c-1). C1 (2, ...; On are the units used to parse the sequence (chunks for HCM and HVM and atomic units for the associative learning (AL) model).
Figure 4 shows the relation between model sequence likelihood and human recall time during the training (b, c) and the transfer block (d, e). The size of the dot represents the trial number, and the shape of the dot represents the group (the triangle being the variable group, the circle the control group). Figure 4b and d show the R-squared goodness-of-ff regressing the various models' sequence likelihoods onto human subjects' sequence recall times. The sequence negative fog-likelhoodd from the HVM correlates with human recall time during the training (R 0.86, p & 0.001) and transfer
7

---

Published as a conference paper at ICLR 2025
blocks (R = 0.7.p p - 0.001). The latter suggests that HVM can generate behavior that resembles that of human knowledge transfer in the memorization of novel sequences with embedded variables.
4.4 COMPARING ABSTRACTION LEARNING AND TRANSFER IN LLMs
Pleade simmmbrr the to tollowing roquarcc: EMORDOBFFEDS, BCDFDCB FADD
AL
HCM
10/M
CPT2
Liana 2-700
L
Figure 5: Comparison between human, variations of cognitive models, and AI models on the transfer block of the memory recall experiment. Bar plot shows the average human sequence recall time, and the sequence negative ligeliillhood evaluated by the various cognitive and large language models.
In recent years, large language models (LLMs) have demonstrated their emergent abilities (Wei) iet al. 2022). raising the concern that their intelligence will soon exceed that of humans, while at the same time. LLMs exert severe limitations in set of tasks that demands abstract thought (Fleuret
et all 2011 Odouard & Mitchell, (022). Given the critical role that abstraction plays in reasoning and generalization, becomes increasingly important to delineate the aspects of L.M that most significantly deviate from human cognition, and find out what features inside cognitive model would provide a missing behavioral aspect inside LLMs. The missing feature will inform the limitations of LLM's computational component that is predictive of its ability generalize and transfer. Specifically, we ask whether LLMs abstract similarly to humans and how their behavior compares to cognitive models incorporating abstraction. To explore this question, conduct the same sequence recall experiment on LLMs.
We adapt the instructions from the human experiment into prompts for in-context learning in LLMs. The LLMs are tasked with predicting the next subsequent tokens based on the context of previously seen instruction sequences. The specific prompt is taken from the instruction given to the human participants: "Please remember the following sequence: BADFDCBFFEDB, BCDEDCBFFADB: We then calculate the conditional probability of the next token, 24, given the instruction sequence history up until the current token P(2 propptrompt). After obtaining this probability, the subsequent token in the instruction is added the prompt, and the LLM'ss conditional probability for the following token is updated accordingly. Using this prompt-chaining method, we can calculate L.M''s negative likeliodl for the next tokenized instruction sequence: llog P(S) = - -log P(2 (promptt) II:=..... P(z (prompt;-1), analogous to the cognitive models. We apply this approach to evaluate the token prediction likelihood in four large language models: GPT-2 (Radford jet al.) 2019) and three variations of the Llama model (Touvron et al. 2023) and two variations of
the Llama 3 model.
Shown in Figure the transfer behavior across humans, L.MMs, and various cognitive models for comparison. Supplementary section A. also shows the R-square value regressing model negative log-likelihoods on human recall time. For humans, the group trained on sequences with embedded variables recalls transfer sequences faster than the group trained control sequences. Relating sequence recall time with sequence negative log- ligeliil,ooodd the cognitive models HCM and HVM, on average, exhibit lower sequence negative log- likelihoood after learning from a training block that shares variables with the transfer block. In contrast, the associative learning model and all variants of large language models do differentiate between training sequences that share variables with the transfer block and those from a training block without transferrable variables.
4.5 ABSTRACTION, DISTORTION, AND GENERALIZATION
With a model that can learn interpretable abstraction, we can delve into the relationship between learning layers of abstraction, compression, and uncertainty. HVM updates its dictionary each iteration
8

---

Published as a conference paper at ICLR 2025
Training
Transfer
a.
b.
1
Forsing Longm
2400
Lunce
1830
22.6
Repposoriation Complexty
Figure 6: a. New chunks or variables are created from previously learned ones, adding increasingly nested structures to the representation graph. b. As the allowable distortion increases (moving right on the x-axis), the required rate to encode the data in terms of chunks decreases (moving down on the y-axis). Introducing variables enables a more compressed sequence, which trades off with higher representational complexity introduced by the variables embedded inside the chunks. c. Abstraction and transfer: The higher the abstraction layer, the higher the likelihood (lower negative log-likel)hood) that HVM parses novel sequences.
by proposing chunks and variables, which are used to parse the sequence for the next iteration. New chunks variables are created from previously learned ones, adding increasingly nested structures to the representation graph. This feature affords a controlled assessment of the relation between the level of abstraction, the amount of distortion that abstract representation introduces, and its relation to transfer and generalization to novel sequences. We measure nestedness using the representation complexity RC(V) Evey EveR(e) - log P(ule), which is closely related to the encoding cost of the representation graph(A.. We also measured the uncertainty carried by the embedded variables in the chunks via entropy formulated as Ecec P(c) Erev(c) E(e - -P(wlv) log P(ulv). Figure 6h shows the relation between the three factors as a consequence of increasing abstraction learning iterations. As the representation graph becomes increasingly nested, the newly Iearned chunks explain longer part of the sequence, and these chunks embed more uncertain variables. This trade-off' between compression and uncertainty reflects rate-distortion theory specifying that the best possible compression of a signal X contains lower bound quality loss specified by the Rate- Distortion Function (R(D)): R( R(D)= info R.d( X)] S D (Shannon 1959. Cover & Thomas, 201 2). Taking representation complexity as a distortion function, as the level of abstraction increases, HVM learns patterns that span and predict longer parts of the sequence while more distortion and uncertainty are introduced. This observation on distortion with the level of abstraction resonates with previous findings relating mental errors with learning more abstract representations in humans (Lynn et al.. 2020).
Higher levels of abstraction implies more flexible transfer If learning higher abstraction layers implies more distortion, what would be the benefit apart from compression? We suggest that the other side of the coin lies in generalization and demonstrates the effect of abstraction on model transfer. We trained HVMs with increasing abstraction layers and evaluated the models' transfer likelihood upon parsing a novel sequence. As shown in Figure 66, the higher the abstraction layer, the higher the likelihood (lower negative log-likelihood) that HVM parses the transfer sequence. To compare, we also evaluated the transfer performance of the chunking model (HCM) with an increasing level of abstraction iteration. While the transfer performance for HCM improves with more layers of chunk learning, it converges at a higher negative logllikeihood. Learning an increasing abstract representation by the HVM enables it compress novel transfer sequences in succinet way. Furthermore, the more abstraction layers are acquired, the higher the relative advantage for HVM that learns abstraction holds over the HCM that does not learn variables.
5 RELATED WORK
Previous modeling work abstraction be divided into two categories. The first is cognitive models, which characterize abstraction as searching for commonalities in explicitly symbolic systems (Lu et al. 2021); these models apply to explain human problem solving, finding problem solutions

---

Published as a conference paper at ICLR 2025
based on its conceptual analogies with the previously-solved problems (Hofstadter & Mitchell 1994). These models describe knowledge using symbolic, explicit graphs but do not address how the representation arises from perceptual data. Other models similar in taste include the recent wake-sleep Bayesian models such as DreamCoder Ellis et al. (222). Our work proposes mechanism for learning explicit abstraction from sequences. HVM starts learning with empty dictionary and does not assume program primitives. Additionally, the inference/recooniiion in the HVM is sufficiently simple and can be simple without extensive machinery.
The second model category implements abstraction in implicit connectionist systems. Works on meta- meta-learning models and LLMs suggest their ability to generalize across contexts and solve problems in way similar to humans in some tasks (Wei et al. [022] Binz & Schulz, 2023) while failing short on other abstract reasoning tasks (Fleuret et al. 201 1.. Meanwhile, abstraction has been argued to be implicitly present in artificial neural networks 2019 Kozma et al. 2018 Johnston & Fusi, 2023] Ito et al. 2022) and biological neural activities (Bernardi et al. 2020 Goudar et al. 2023). albeit challenging to interpret (Fan al. 2021). ) Relative to this approach, our work provides a reductionist cognitive model that explicitly specifies the minimal components needed to learn interpretable abstract
structures.
Besides cognitive models, our work relates to and differentiates from other sequence learning approaches. Our generative mechanism relies on chunk-based recursive generation and inventory growth rather than formal grammar rules (Jelinek et al. 1992. Johnson et al. 2006). Zipfian distributions (Teh, 2006), abruptly changing statisties (Milan et al.), , or sequences from stochastic chain Willems et al. (1995). We suggest that the perceptual sequence contains nested hierarchical structures, with chunks as a central generative force.
The hierarchical nature of HVM shares with models including topics models and Helmholtz Machine; however, differs from topics models Blei et al. (2003); Teh al. (2006) aiming at sequences rather than "bags of words'. The Helmholtz Machine also contains hierarchical generative and recognition components, but its focus is similarly not on sequences, and its sequential extension Hinton al.) (1995) did not work well partly because a lack of iterative processing that HVM employs. Finally, the goal of HVM as a computational cognitive model differs from neural models such as CSCG (George al., (222]) which has an implementation level focus Marr (1922). However, we encourage future work to investigate the neural underpinnings of abstraction.
DISCUSSION
Our work has limitations. One that variables are only proposed to be embedded between chunks and cannot come at the beginning or end of sequences, restricting the location of discoverable variables. Secondly, representation learned later iteration depends on those earlier acquired. However, this aspect aligns with the order and curriculum dependency of human learning. Children demand a simple-to-complex curriculum and utter more complex compounds later in their development (Friedmann et al., 2021 Snow 1972 Fernald et al. [989). There is limit to HVM's expressivity: it won't t be able to capture a pattern of two chunks being a variable number of chunks apart. Sometimes, be arbitrarily nested depending
the variable structure moment. Future work may look at optimizing the structure during learning. Finally, our work focused on the cognitive resemblance of HVM and has yet to explicitly optimize the model for computational efficiency, leaving space for future work.
the representations learned up until that
Our work opens up future directions both in cognitive science and machine learning. Previously, grammar learning (Chomsky 2013), chunk learning (Gobet et al. 2001), and statistical/associative learning (Saffran al.) 1999) were studied as distinct characteristics of human sequence learning. We propose via HVM that the three components can be treated as finding invariant recurring chunks from sequences via statistica//associatve learning, resulting in learning grammar- like structures. Our work suggests normative origin of concrete and abstract chunk learning as a learning agent uncovers the underlying entities that constitute perceptual sequences. Future work can relate this model to developing concepts or compare the model with human behavior and LLMs on more nested datasets. Meanwhile, the close tie between abstraction and generalization also urges future hypothesis-drive driven research regarding preconditions for learning abstract concepts to acquire more complex representations. This can help to illuminate the emergence of abstract representation through excessive training and its relation to the generalizability of connectionist models (Power al. 2022
10

---

Published as a conference paper at ICLR 2025
Miller et al. 2024). Finally, work can act in the tokenization role to fuse with transformers SSM to extract interpretable representations from complex sequence data while enjoying the power of history-dependent sequence prediction.
7
REPRODUCIBILITY STATEMENT
Detailed information about the HVM algorithm, proof, generative model, test and experimental details and results can be found in the supplementary information section. The code used for the algorithm and experiments is available under this link
REFERENCES
W Abler. On the particulate principle of self-diversifying systems. Journal of Social and Biological Systems, 12(1):1-13, January 1989. ISSN 01401750. doi: 10010006001411155099999000000000 URL
https: petps://Lie httpps//liikiin http:///innnnnghhb wwcaiuubbeseeeeee wsevier...... com httpo//wwr....ooeeeeeeeeeeee///a
John Anderson and Robert Milson. Human memory: An adaptive perspective.. Psychological Review,
96:703-719, 10 1989. doi: 10. 01177/00333-95..664444.
John R. Anderson. Retrieval of propositional information from long-term memory. Cognitive Psychology, 6:451-474, 1974. doi: 10. 10.10160010002554449000101
Lawrence W. Barsalou. Perceptual symbol systems. Behavioral and Brain Sciences, 22(4):577-660,
1999. doi: 10.1017750140555599900
Silvia Bernardi, Marcus K. Benna, Mattia Rigotti, Jerome Munuera, Stefano Fusi, and C. Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 183: 954-967, 2020. doi:
200.0.0.03..0
Marcel Binz and Eric Schulz. Using cogmitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 1206)):22188221100 2023. doi: 10. 10..107/pma.... htt.//a.....22111000 URL
https: /www wwwwwwwwwwwwwppw..w..w...www...w phas or /ttg//////b///1.101/...
221 8523120
David M. Blei, Andrew Y. Ng. and Michael I. Jordan. Latent dirichlet allocation. Journal of Muchine Learning Research, 3:993-1022, 2003.
BNC Consortium. The british national corpus, xml edition, 2007. URL htttp://hd..hanet handle net/20.500, 12024/2552
R. H. S. Carpenter and M. L. Williams. Neural computation of log likelihood in control of saccadic eye movements. Nature, 377(6544):59 1995. doi: 10.1038/377059a0. URL
https: ncbi. nlm. nih. gov hovv/pubme//765111
Noam Chomsky. Problems of projection. Lingua, 130:33-49, 2013. ISSN 0024-3841. doi:
URL htRbbttps:: www.ww.eeeeeedree............... com/
SI: Syntax and cognition: core ideas and
results in syntax.
Allan M. Collins and M. Ross Quillian. Retrieval time from semantic memory. Journal of Verbal Learning & Verbal Behavior, 8(2):240-247, 1969. doi: 10. 00110066600002255377666699666 URL
https:// /tops:://do...o .wrooioogg/ii 0tt.b/o..r/11..111////111000
(69)
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory, John Wiley Sons, 2012.
Valentina Cuccio and Vittorio Gallese. A Peircean account of concepts: grounding abstraction in phylogeny through comparative neuroscientific perspective. Philosophical Transactions of the Royal Society B: Biological Sciences, 373 or June 2018. doi: 1098/sstb.2017. 0128. URL Publisher: Royal Society.
2017. 0128
Ian Cunnings and Patrick Sturt. Retrieval interference and semantic interpretation. Journal of Memory and Language, 102:16-27, 2018. doi: 10.10166j.mm..2118855u.u URL https
(/doi.org/// 10w...111....... 201 0.05.001

---

Published as a conference paper at ICLR 2025
Stanislas Dehaene, Fosca Al Roumi, Yair Lakretz, Samuel Planton, and Mathias Maylé-Meyer. Symbols and mental programs: a hypothesis about human singularity. Trends in Cognitive Sciences, 26(9):
751-766, 2022. ISSN 1364-6613, doi:
URL https:
// //www. scw.ssciee encedi rect. com/ http://ceeneearrceeeeeeeeee 31t.:///ee/////11116166332200000 322000011116006444 413
Jakub Dotlačil.
Parsing as a Cue-Based Retrieval Model.
Cognitive URL
Science,
1308), 2021. doi: http:://doii.rrg/1.1111111 13020.
https
//online ponniinelibbrry.yo wi wttp://wwccccoooo/////////////// 11100/1111111111111 13020
_eprint:
Karl Duncker. On problem-solvingg
580. 1945. ISSN 0096-9753.
doi: 10. 10.1037/00093599 URL
h0093599
Kevin Ellis, Maxwell Nye, Yewen Pu, Filipe de Avila Belbute-Peres Sosa, Joshua B. Tenenbaum, and Armando Solar-Lezama. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. Communications of the ACM, 64(4):116 125, 2021.
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2)119--211 1990. ISSN 0364-0213.
doi: http://ccieeeeeet.eeeee//////000 com/
URL https:: www. scwencecieectttttt
Leng-Lei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang. On interpretability of artificial neural networks: A survey. IEEE Transactions on Radiation and Plasma Medical Sciences, PP:1-1, 03 2021. doi: 10. 1011100PSSMSS000 2021336648.
Anne Fernald, Traute Taeschner, Judy Dunn, Mechthild Papousek, Bénédicte de Boysson- Bardies,, and Ikuko Fukui. A cross lansuala study of prosodic modifications in mothers' and fathers' speech to preverbal infants. Journal of Child Language, 1989.
Paolo Ferragina and Giovanni Manzini. Indexing compressed text. J. ACM, 52(4) July 2005. ISSN 5001-5411. doi: 00.1145/00820361182233 URL
1082036. 082039
Johann Gottlieb Fichte. The Science of Knowing: Fichte's 1804 Lectures on the Wissenschaftslehre. second series. State University of New York Press, Albany, 2005. Original work published 1804.
François Fleuret, Ting Li, Charles Dubout, Emma K. Wampler, Steven Yantis, and Donald Geman. Comparing machines and humans on visual categorization test. Proceedings of the National Academy of Sciences, 108(43) 2011. doi: 10. 10.1073/pnas. 110pppanns.111188881 1091661000 https: www. www.pppas... /ttp://////////01110113////1//// wwwp:/aaa...11111111111111001110
URL
Edward Fredkin. Trie memory. Commun. ACM, 31),499999 sep 1960. ISSN 0001-0782. doi: 10.1 10.114576779003674000.
367400
N. Friedmann, A. Belletti, and L. Rizzi. Growing trees: The acquisition of the left periphery. Glossa: a journal of general linguistics, 6(1):131, 2021. doi: 10. /6995//lossa.5877.
Dileep George, Rajeev V. Rikhye, Nishad Gothoskar, J. Swaroop Guntupalli, Antoine Dedieu, Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps. Nature Communications, 12(1):339,, April 2021. ISSN 2041-1723. doi: 10. 10.11088/444447-0022255555 axricLLes/441477-221222222
and Miguel Lázaro-Gredilnn
URL https: www nature.... com/
M. Gerlach and Font-Clos. A standardized project gutenberg corpus for statistical analysis of natural language and quantitative linguistics. Entropy, 22(1):126, 2020.
Fernand Gobet, Peter Lane, Steve Croker, Peter Cheng, Gary Jones, lain Oliver, and Julian Pine. Chunking mechanisms in human learning. Trends cognitive sciences, 2-236-243, 07 2001. doi: 10.1016/S1366--66 6613(6))660000666244
Vishwa Goudar, Barbara Peysakhovich. David J. Freedman, Elizabeth A. Buffalo, and Xiao- Jing Wang. Schema formation in a neural population subspace underlies Searning-to-lean in flexible sensorimotor problem-solving. Nature Nearoscience, 2023. doi: 10.10088/41593--223442 293-9.
12

---

Published as a conference paper at ICLR 2025
S. A. Greibach. A note undecidable properties of formal languages. Mathematical Systems Theory, 2:1-6, 1968. doi: 10.1007/BF01691341.
Geoffrey E. Hinton, Peter Dayan, Ava To, and Radford M. Neal. The helmholtz machine through time. F. Fogelman- -oulli and Gallinari (eds.), ICANN-95, pp. 483-490, Paris, France, 1995. International Conference on Artificial Neural Networks, Springer.
R. Hofstadter and M. Mitchell. The copycat project: A model of mental fluidity and analogy-making In K. J. Holyoak and J. A. Barnden (eds.), Advances in Connectionist and Neural Computation Theory, volume 2, pp. 31-112. Ablex Publishing Corporation, 1994.
Takuya Ito, Tim Klinger, Doug Schultz, John Murray, Michael Cole, and Mattia Rigotti. Compo- sitional generalization through abstract representations in human and artificial neural networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 32225-32239, Curran Associates, Inc., 2022. ttt ott.p//cee..ooo......oooooooooo. httpa///w////////eeeeeeeeeeeee/e Fi F222:/0000aaaaccceeeeaattaaaaaa2
Fred Jelinek, John Lafferty, and Robert Mercer. Basic methods of probabilistic context free grammars. Speech Recognition and Understanding, NATO ASI Series, 75, 01 1992. doi: 000007998--364--66600 _
Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. Adaptor grammars: a framework for specifying compositional nonparametric bayesian models. In Proceedings of the 19th International Conference on Neural Information Processing Systems, NIPS 06, pp. 641-648, Cambridge, MA, USA, 2006. MIT Press.
W. J. Johnston and S. Fusi. Abstract representations emerge naturally in neural networks trained to perform multiple tasks. Nar Commum, 14:1040, 2023. doi: 10. https:/ http:///ddooorgg//.o. 00010////-11477-2335355553
URL
Immanuel Kant. Critique of Pure Reason. The Cambridge Edition the Works of Immanuel Kant. Cambridge University Press, New York, NY, 1998. Translated by Paul Guyer and Allen W. Wood.
George Konidaris. On the necessity of abstraction. Current Opinion in Behavioral Sciences, 29:1-7, October 2019. ISSN 2542.1546. doi: 2.1016/).obeeha.0111111.11 www. www.scieence.rre:: com/ wttp:///c.i.eeeeeeeee/////////a1
URL https:
Robert Kozma, Roman Ilin, and Hava Siegelmann. Evolution of abstraction across layers in deep learning neural networks. Procedia Computer Science, 144:203-211 01 2018. doi:
P. Lison and J. Tiedemann. Opensubtitles216:: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), 2016.
Hongjing Lu, Nicholas Ichien, and Keith J. Holyoak. Probabilistic Analogical Mapping with Semantic Relation Networks. Technical Report arXiv:2103.16700 arXiv, October 2021. URL http:// /ttpvviivvv...////v//e.ae/aaaaev orgg abs /tb.////333.. 16704 1rXiv:2103.16704 [es] type: article.
Christopher W. Lynn, Ari F. Kahn, Nathaniel Nyema, and Danielle Bassett. Abstract of events arise from mental errors in learning and memory. Nature Communications, 11(1:2313, May 2020. ISSN 2721-1723, doi: 10. 10.18////41477
15146-7. URL htRpstttt:///w...r
Com/ htmm/articles/s41111 010777020-1100 1200-11166-111
B. MacWhinney. The CHILDES Project: Tools for analyzing talk. Lawrence Erlbaum Associates, Mahwah, NJ, third edition edition, 2000.
David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. MIT Press, Cambridge, MA, 1982. ISBN 978-0-262-63155..
Kieran Milan, Joel Veness, James Kirkpatrick, Michael Bowling, Anna Koop, and Demis Hassabis. The Forget-me-not Process.
13

---

Published as a conference paper at ICLR 2025
George A. Miller. The magical number seven, plus minus two: some limits capacity for processing information. Psychological Review, 1956. ISSN 0033295X. doi: 10.1037/h0043158.
Jack Miller, Charles O'Neill, and Thang Bui. Grokking beyond neural networks: An empirical exploration with model complexity, 2024.
Matthew R. Nassar, Julie C. Helmers, and Michael J. Frank. Chunking as rational strategy for lossy data compression visual working memory. Psychological Review, 125(4):486-51, Jul 2018. doi: 10. 10.10377rev0000101.
Gonzalo Navarro and Veli Mäkinen, Compressed full-text indexes. ACM Comput. Surv,, 39(1)-2-ss. apr 2007. ISSN 0360-0300. doi: 10.114512113770.216622. URL https:://aiioz///// 1145/1216370.121663
Victor Vikram Odouard and Melanie Mitchell. Evaluating understanding on conceptual abstraction
benchmarks, 2022. URL
14187
Stellan Ohlsson and Erno Lehtinen. Abstraction and the acquisition of complex ideas. International
Journal of Educational Research, 27(1):37--4, Jamuary 1997. ISSN 0883-0355. doi: 10.
1016/S0888-033517788842223 URL wttp:///www.iieeeeeeeet..c.c....
articcee/s1//0888055578888
Jean Piaget. Cognitive development in children: Development and learning. Journal of Reseurch
in Science Teaching, 2: 1:6--1186, 1964. doi: 10.1002/tea.3660000036 URL htth:t: éx doi
org 36600//203
E. Post. A variant of a recursively unsolvable problem. Bulletin of the American Mathematical Society, 52:264-268, 1946. doi: 10.10090/0002-99000 194---99999-11-9-55----55--
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting small algorithmic datasets. arXiv preprint 22010.2201.02177. 2022.
Alec
Radford, Jeff Wu, Rewon
Child, D. Luan, Dario Amodei, and models are unsupervised multitask learn-
Πly Sutskever. Language URL
ers. 2019.
org/paper/ -
Jenny R Saffran, Elizabeth K Johnson, Richard Aslin, and Elissa L Newport. Statistical learning
of tone sequences by human infants and adults. Cognition, 70(1):27-52, 1999. ISSN 0010-0277. doi: http:://w..ogg//..oooooo0022//79
com/ cttp://cc.aaeee//rr..eeeeee////2
Atilla Schreiber, Shuchen Wu, Chenxi Wu, Giacomo Indiveri, and Eric Schulz, Biologically-plausbbe hierarchical chunking mixed-signe signal neuromorphic hardware. In Machine Learning with New Compute Paradigms, 2023. URL http:httt///oeereeeeeeee
hwwp:/ee/eeeee net/ didtrummiid-iio IuN2WXtFSy
Claude E. Shannon. Coding theorems for discrete source with fidelity criterion. IRE National Convention Record, (1):1442163, 1959.
Nathaniel J. Smith and Roger Levy. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302-31) 319, 2013. ISSN 0010-0277. doi: wttp://cciinneeettcllle///////// 2013.02.013. URL https: www.w/wwww.mmmmmmmmm//./o/ sci www.scedeeeeiiccc rect. com/
cog.ittion.o.n.
S001 0101000771300011
Catherine E. Snow. Mothers' speech children learning language. Child Development, 43(2):
549-565, 1972.
D. Sportiche, H. Koopman, and E. Stabler. An Introduction Syntactic Analysis and Theory. Wiley, 2013. ISBN 9781118470473. URL https: //ttps://boos.....////
MXk JAQAAQRAJ
14

---

Published as a conference paper at ICLR 2025
CARL STERN. Kant's theory of empirical concept formation. The Southwestern Journal of Philosophy, 8(2)17-23,, 1977. ISSN 0038481X, 21541043. URL htthhttp//wwwt /wwp:///wwww...m jstor /rg/ stable/431555
Yee Whye Tch. A hierarchical Bayesian language model based on Pitman- Yor processes. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL- '06, pp. 985-992, Sydney, Australia, 2006, Association for Computational Linguistics. doi: 10.3115/1220175. 1220299. URL http:: acm. orq/citation
Yee Whye Teh, Michael Jordan, Matthew Beal, and David M. Blei Reviewed work(s): Hierarchical Dirichlet Processes. Journal of the American Statistical Association, 1016476):156-11333 2006.
URL /
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Bert Van Oers. Contextualisation for abstraction. Cognitive Science Quarterly, 2000, 1, 279 305. Cognitive Science Quarterly, 1:279-305, January 2001.
W. von Humboldt and M. Losonsky. Humboldr: 'On Language': On the Diversity of Human Language Construction and Its Influence on the Mental Development of the Human Species. Cambridge Texts in the History of Philosophy. Cambridge University Press, 1999. ISBN 9780521667722. URL
htttps://book: htwp.oogeeooe.e.eooooo.oe...ee..
de/ooeekkookkseeeooo wttp::1ddd__0__D___0000 UODDG1
Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. Call for papers the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus, 2023. URL httpppp://aaxl...oor//////////a/
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.
F.M.J. Willems, Y.M. Shtarkov, and T.J. Tjalkens. The context-tree weighting method: basic properties. IEEE Trunsactions Information Theory, 41(3):653-664 664, May 1995. ISSN 00189448. doi: 10.1109/18.8800
http:tttp////eeeplllle
feee. org www.eee/rr//cccoe//////e/e /tt///doowtc////22ce 3382012/
Shuchen Wu, Noemi Elteto, Ishita Dasgupta, and Eric Schulz. Learning Structure from the Ground up H--Hearrrchhall Representation Learning by Chunking. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 36706-36721. Curran Associates, Inc.,
2022. URL https::/pproceediiiiiiso wwwpinngg...nuuueeeee neur httpp/scccccpppeeeleeeeeeeeeeeee .
E1 wlle/eebbb2221 httaaa/220c222aaaaaooae2.6666661
Shuchen Wu, Mirko Thalmann, and Eric schulz. Motif lcarning facilitates sequeeteeeeesueeeeeeeeee sequence memorization and generalization, Dec 2023. URL http:f.ooppppppp./s/ppppaaa
Biling Yee. Abstraction and concepts: when, how, where, what and why? Language, Cognition and Neuroscience, 125-011) November 2019. ISSN 2327-3798. doi: 10. 10.1080/23273798.201
1660797. URL Routledge _eprint: http::/ddoioorg/10000000233333:1 0t1.2/77777799809999666699999999
orwpioorg/iii 1080 /22703/237779 2019.
Publisher:
J. Ziv and A. Lempel. Compression of individual sequences via variable-rate coding. IEEE Transactions on Information Theory, 2455:530-556, 1978. doi: 1999/1IT.9998.1555994
15

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
A.1
SET UP
An observational sequence S made up of discrete, integer valued, size-one elementary observational unit coming from an atomic alphabet set A. One example of such an observational sequence S is:
010021002112000...
An example belief set that contains only concrete chunks can be B = 1,1, 211, 12, 22, 21122
Using the belief set to parse the sequence results the following partition. 0 010021 0012100212222 Q.
Another example belief IB that contains chunks {21v, 0100, 0003 with embedded variables
V = {w). The denoting set of ((v) = {00, 12]. Then the sequence S is parsed as 0100 21 21
000.
Definition (Completeness)
We say that a belief set is complete if at any point during the sequence parsing process, the upcoming observations can be explained by least one chunk in the belief set.
In this work, the learning mechanism guarantees that the belief set complete.
Definition 2 (Parsing Length [W)
A parsing length [W| of sequence is the length of the sequence measured in chunks.
A.2 GENERATIVE MODEL
Pseudocode to generate sequences with nested abstract hierarchies. Input: of atomic elements A; the number of combinations d; sequence length Output: seq, sequence made of concrete observational units
Algorithm 1:
eg < initialize representation graph:
for i 4- 1 to d do
RAND <- random number between and 1;
if RAND > 0.5 then
// Ob ject Creation
B eg obbjectssnddCattggriis:
Recomb + random (aaaloom.hhoice(2.. 3, 4,4,
samples candom.samples B, k=ncombo);
while any of the first and last element in samples are categories do
samples end
mandom.samleed k=Tlommha);
newobject + concatenate(sammles))) ggaddChhnknneuobjee)) end
else
// Category Creation
B t cg.obbecct
&- aundom.choice(2 5))
Reombnn
samples f- nombb pandom.sampled B, k=mcombo);
newentegory +- create new category denoting somples;
ggaadVaaribb(reevvtteegyyy
end end
eg. ggaasiignProbbbiliisSSSooeceeet
sampledseg + gg.samplcohheettAnSpeeeyy variadppeeeiiiiaaeleessss Variables(I);
seq f- convert sampledseg to sequence;
16

---

Published as a conference paper at ICLR 2025
A new object is created by concatenating random selection of pre-existing objects categories from the existing inventory. After the inventory has been expanded up to the d-th iteration, objects assigned with independent occurrence probability sampled from flat Dirichlet distribution [p((c), ple a)); ai, a/a)
| P(a))a11, (aa))a-1,,; 04 Ivi. Where the beta function when expressed using gamma function is: B(a) 1. T(a.) and a = (a1..., (aa)).))))). The parameters (a1..., ((A)) are identically set to one.
B(a)
r o.)
Similarly, a probability is sampled from flat Dirichlet assign the independent occur- rence probability of the set of object E(u) that each category v denotes. Ve E V, (p((c)..... ple al, done for each created category.
Bia B(v) P(q)a-1, a Ivi. This procedure is
Intuitively, objects represent recurring observations of specific entities, such molecules composed of chemical elements. Categories, on the other hand, represent broader categories of entities that share common properties, such chemical elements belonging the same class (e.g., noble gases alkali metals). The observation sequence forms a nested hierarchy, where entities and their categories frequently interact and combine with others.
A.3 APPROXIMATE RECOGNITION INVERSE
Theorem 1. If the left entity and the right en (which can be either a chunk or variable) of ground truth variable o addition its denoting chunks 01, e,.c2.... Cm has been learned, and every parsing of er. CR, and C1, C2,..., Cm is consistent with the constitution of the sequence by the its way of generation, then C1 C2,..., Cm will be necessarily proposed by common udjacency and common preadjacency criterion into novel variable ef and the true denoting entities will be subset the denoting set Evv)). {c1, C2 Cm E E(v').
Proef. By definition, {ei, C2, Cm is in the adjacency Adj (cr.) entries of ct. and the preadjacency entries of CR. Preadj(cap) (cR). And therefore, {c1, 02.... Cm E Adj(cc) Preadj(cR) and hence will be included the set of denoting entities for new variable.
Proposing variables based on transition matrix
High
adjacency
preadjacency
Low Frequency
Figure 7: Common adjacency and preadjacency structure to identify a variable.
Theorem In an infinitely long sequence. a chunk c in the generative model is made up by concatinating the entities {C1, C2, ..., ). and these entities {c1, ..., Cn) are parsed in an identical way as how the generative model samples e, then HVM will eventually learn to chunk e' including all the entities in fcr, C22 02,,... )
Proof. By contradiction. If any of the entities not included in the learned chunks, correlation will still exist in the transition entries between subparts that compose e, which will be resolved by the next chunk proposal iteration.
17

---

Published as a conference paper at ICLR 2025
In practice, identifying the ground truth representation graph is much more complicated, as the representations learned by HVM can be mapped to context-sensitive grammar [A.6] which is more complex and undecidable than context-free grammar. Learning the correct chunks that exactly match the generative model relates to the non-identifiability problem that multiple grammars can generate the same set of observed data (Post 1946] Greibach]] (968) inferring the exact grammar from positive examples is undecidable. The sequence length comparable to the ground truth is possible to obtain. Instead of comparing the model with the specific chunks of the ground truth, use of evaluation measures to evaluate the model's 's performance.
Algorithm 2: HVM (online version, for learning sequences from human experiments)
Input: Learning sequences seg; representation graph ey; boolean flag for chunk learning threshold chunk; boolean flag for variable learning abstraction
Output: eg, @eunk_recore
eg t initialize representation graph;
// Initialize chunk record
chank_record t {};
t + 0:
while not seg_over do
current chunks, cg, seq, chunk record chwnk_recordd
chunks(cg, seq):
eg t- t-earnning_an and appate(ccureett chunk_record cg, threshold_chuee =
Truc):
if abstraction then
cg t (bttraciiooupppattteuuree end
eg);
cg. corgott::)):
multiply all frequency record by
end
A.4 PARSING SEARCH STEPS
HCM retrieves learned chunks match them with the upcoming sequence, parsing the largest chunk that aligns with the sequence. However, as the dictionary size |C] grows, the number of search steps increases, leading inefficiency. To address this, HVM organizes the chunks in C within parsing graph, harnessing shared nodes across multiple chunks to speed up the chunk identification process.
Definition 3 (Parsing Graph (PG))
Parsing graph is graph for chunk identification. The nodes inside such graph are the chunks C. Each parent node is the overlap of all of its children.
To parse a chunk, the model traverses the memory tree structure to find the path up to the deepest node inside the tree consistent with the upcoming sequence. Pseudocode A.4 describes this process.
18

---

end end end Published as a conference paper at ICLR 2025
Algorithm 3: Pseudocode for traversing a tree to find a path consistent with upcoming sequence
Input: rootNode, the root node of the tree; sequence, the sequence to matched
Output: path, the path to the leaf consistent with the sequence, if found path an empty list; if not root ode then
return null;
end else end
return TRAVERSETTREE((OoooNN sequence, palh);
Function TraverseTree (node, sequence, palh):
path.append(node))
IsCoNsistENTtode,, sequence) then
if eede. then
return path; end
forall child chieechildren do
result TRAVERSETRREI(hhin sequence, path.copy()) result then return result;
return null;
Definition 4 (Parsing Search Steps)
Parsing search step refer the number of search comparison made in the parsing graph to find the deepest chunk the tree consistent with the upcoming sequence.
Example Shown in Figure 8 when the sequence upcoming is 123, one starts from the ancestors of the graph and check whether or 2 at the beginning of the sequence, identifies 1, and proceeds into the subtree which stems from 1. The leaves are contains 12 and 13. Then HVM checks whether 12 is in the sequence or 13, thereby identifying 12.
The mechanism identify the biggest concrete chunk that can be used parse the sequence involves the following steps:
Search from the top of the tree (nodes with parents)
Find the node in the next layer consistent with sequence
Go to the children the node until the node contains no more children (leaf node) or none of the node's children is consistent with the sequence
As illustrated in Figure & the parsing search steps of chunk 12: PSS = 2 + For chunk 221: PSS( = 2 + For 21: PSS(21) = 2 +
The advantage of memory abstraction
Theorem 3 (Guarantee fewer search steps). Identifying chunk in tree-structured memory takes fewer search steps than without.
Proof. The hierarchically structured memory system reduces the searching step to the number of branches on the path that leads to the identified chunk, which is a subset of all nodes in the tree. C
Expected parsing search steps steps is denoted as PSS(cc.
For a chunk e that occur with probability P(c). the parsing search
As an example, three chunks c1 C22 and c3 composes a sequence and they are parsed with probability P(c)), P(c2), and P(cs) in sequence.
19

---

Published as a conference paper at ICLR 2025
Parsing chunk
Sequence: [1,
Start in ancestor of the graph
2
Search in chunks [1, 2). if
they are sequence O(1)
Search chunks [(122), (1[3), they are sequence
21
22
Identify 12
221
223
225
Figure 8: Example process when parsing a chunk using parsing graph. Gray region denotes chunk overlaps, which are the non-leaf nodes inside the parsing graph being the common prefix of their children.
Without the hierarchical memory structure, the average parsing search steps to identify chunks in the sequence is:
PSS(c))P(a) PSS(c)))()) PSS(s))) PSS(cc)
With the prefix Trie, the number times needed to check whether a chunk matches certain abstraction branch requires number of parsing steps:
PSS(c) Nohuldren (Pa(c))
Pa(c) denotes the parent of chunk in PG, Nehitdren (Pa(c)) is the number of children that chunk c's parent contain. An abstract chunk in such parsing graph can be the prefix intersection of three chunks C1, C2, and a = C2 N Cg. Parsing abstraction summarizes all the subordinate chunks that underlies such abstraction. Therefore abstraction node is more likely to be parsed compared to individual chunks.
P(a) = (P(e) + P(c2) + P(ca))
Additionally, abstraction can be used anchor to find the subsequent denoting chunks. P(cila), P(ez a) and P(cyla). The expected parsing search steps to identify chunks for such sequence becomes:
no Es(PSS) (e2 P(e2) + P(ea))
PL(c a)P(ai)a) + PL(e2 P(c2a)) + PL(c3 ((cala)
Generalizing this formula any parsing graph, the expected parsing search steps given the parsing graph's ' internal abstraction and concrete chunks would be
E.SS((G) = PPO Σ
len(Nehildren (P (PA(c))
CEC
path(c)
path(c) the path starting with the root node of the graph that leads to the identification of the chunk node Since P(c),, spath(() P(a) Ile, Eeathht)())) P(c))pa(c:)), one can also give lower and upper bound on the number of parsing search steps needed to arrive at random chunk in the parsing graph:
EPSS(PG) = Σ P(u)PA(t))PSS(( PA(v))
DEC
PSS(( - b) denotes the subgraph that starts from b which reaches a. If o is inside the ancestor nodes, then ancestor (v) = Ø, and therefore P(vlancestor(()) = P(v).
The expected parsing search steps for representation graph can be calculated by averaging out the expected parsing search steps for each sub graph inside parsing graph. The lower bound the number of steps that leads to the shallowest node, and the upper bound is the number of steps that leads to the deepest leaf node.
arg min cEC
Σ
len(nchildren (PA(c))) S ECec(C) & arg max
Σ
len(Pchhtdren (PA(c)))
GEC
wEpath(e)
uEpath(e)
(4)
20

---

Published as a conference paper at ICLR 2025
slogtes)
+lggt(2)
ologt//)
World
World
World
0.5
0.5
0.5
0.5
0.5
0.5
Animals
Plants
Animals
Plants
Animal
Plants
Variable
Mammals
Reptiles
Mammals
Reptiles
Fungi
Flowers
Eniiliing shunks
Coarser representation generally consumes less complexity.
Parent
Figure 9: An increasing level of abstractions
A.5 ENCODING COST
A representation graph RG specifies distribution of observation instances encoded in compositional manner. One can the minimal encoding cost to distinguish all the variables.
RC(G) Σ
Σ - P(ul)) + Σ
log P(w) + Σ log P(c)
(5)
DEG nEE()
eEG
cEG
The more ambiguous variable is, the less resources needed to encode the variable; the smaller the variable graph is, the less edges it has, the bigger the probabilities parsing and variable identification, the smaller the encoding cost. Every time when new edge points from pre-existing variable to new variable, the encoding resource expands by an amount determined by the conditional probability.
Example Representation graphs with different nested structure translates to different encoding cost. In Figure 9 three graphs contains an increasing abstraction specificity. In graph the variable 'world' is split into two variables: animals and plants. Each of these variables do not denote a specific observation, but serves as a placeholder for any instances of specific observation that fits into that particular category. The encoding cost for such a graph, given the specified observational probability on the edges would be:
EC(gl) log
(ogg
orld)) = - log(0.5)
(6)
Since once the variable World has been identified, one only need to distinguish the subcategories inside the variable W orld, and the minimal code length distinguish one subcategory (animals) from another (plants) would be the conditional probability: P Porla)
Whereas if gl does contain nested structure, one would need encode the variable world separately from the variable animal and variable plants, in this way, the alternative encoding length without an edge connecting the sub categories with the main category would be:
EC(g1) log (Pg(P(Worilen orll)) log( ( (((lants)) log( (AAnimals)) = - log(0.5)
For g2, the case is slightly different: the variable category that denotes animals further split into the category of mammals and reptiles. Thereby, extra encoding costs are needed.
EC(g2) = log( ((W ((Worlld) - -og(P(Paanss/Worid) log( P (Animals|| World) log( ( ((Mammals)) Animals)) log(P (Repiles)) Animals))
(8)
-4 (og(0.5)
If each of these categories encoded separately and each variable is pointed to other variables, the information content encode variable graph without edges in the case of 92 would be:
21

---

Published as a conference paper at ICLR 2025
EC(52) = log( P(World)) - (og(P((Planta))
log( (( ((Animals)) log( P (Marmals)) log(P( (((epplile))
(9)
6 log((..5)
Since P(Mammals) ((maamaassjanimaall P(animals).
This difference is more pronounced going from g2 to g3:
EC(g3) log( (P(Worl)))) togg(P(PPannts) - log (og((((nni(dddsssoorrttt
log (al.aaammmmmssAA/naaaaa - logeP(Repillls (atg((((((eetteeeeeaaaamssss) Animals))
(10)
log (( (Panuglllaaanss)) loo( (PPllPollrurrslllalna)
-66 l0gg(0055
Whereas for g3. not encoding variables in nested structure would result in an encoding cost of
EC(33) EC((2) + log log(P(Fungi)) log( (og((((Flowers))
(11)
lo0
These examples illustrate organizing variables into a nested recursive structure saves encoding cost.
A.6 ALTERNATIVE FORMATION AS LEARNING A CONTEXT SENSITIVE GRAMMAR
HVM learns a 5-tuple from the sequences G = (A, C, R. set of atomic units A {ap} set of chunks C= }. h = 1,2, ...,, each chunk e a sequence of atomic units and variables. The model uses chunks from C to parse the observation sequence. Assume random variable C as the chunk being parsed, taking a value e from C, c E C, c P. P is the parsing probability.
Exec P(C'=) c)
A set of variables V { 1,2,
set of rules = {F(m)) F(r2), F(u,)) that each specifies the set of chunks F(u) = (cz)
denoting each variable E V. Probabilities variable denoting Vi, E3 P( - g2) =
A.7 RATE-DISTORTION THBORY
Rate-distortion theory specifies that the best possible compression of signal X contains lower bound on quality loss specified by the Rate-Distortion Function (R(D)): R(D) infq Rqs.t.E(((X, X)] <
D (Shannon] 1959: Cover & Thomas 2012). According to the RD theory, the minimum R (the be transmitted over communication channel for
amount of compression) at which information
given level of information loss D is specified by the Rate-Distortion Function (R(D)): R(D) = min E(dd( SD I(X; X). The mutual information quantifies how much information needs to be preserved during compression achieve this fidelity, and the function minimizes this quantity while R D) info RQ s.t.E[d(X. ((,XX)) X)] S D. Q represents the encoding function that maps from x to á. RD agnostic to the choice of the distortion the representation complexity, which measures the nestedness of the
satisfying the distortion constraint. Written in another way.
function, and in this case variables learned by HVM.
A.8 PARSING PROBABILITY
A sequence S is parsed by the chunks in C to obtain distribution of chunk parsing probability Pc- This distribution on the support set chunks has shapes dependent on the parsing mechanism. HVM employs greedy strategy and chooses the deepest consistent chunk in the memory tree to parse segment of the upcoming sequence. Other parsing strategies will result in alternative distributions for Pc, in addition to the transition probability Po(c:lc))
Define the set HS(S) as the set of all distributions results in any parsing method using the chunks in C to parse sequence S, and any parsing probability P E He(S). Hence each evaluation measure
22

---

Published as a conference paper at ICLR 2025
the representation is bounded by the optimal and the most unfortumate parsing occasions. Taking the predictive power measure as an example:
arg min (S)
Ep cl PNVS /c| S arg max Eplc
(12)
Pe PEHc(S)
The measured value is bounded by the expected value from the best parsing distribution and the worst parsing distribution.
A.9 UNCERTAINTY
Variables introduce uncertainty. Without variables, the chunks inside E,, needs be encoded as distinct chunks, consuming encoding cost of EEEE, - log P(c).
We have the ground truth set of sequential observational units gt. Let's say in a recall task, the agent learns variable chunk to describe the ground truth, the variable denotes to three concrete chunks e₁ 22 and Cg. The accuracy when 61 is being sampled will be Digt, c1), when D is a distance function evaluating the agreement between the ground truth chunk gt and c1. Then expected accuracy within a variable:
evaluate the
E(gt, 8)
Σ P(gt)) eER
(13)
P(clu) is the probability of sampling chunk given that the variable v is identified.
Compared chunks without variables, recalling chunks with embedded variables introduces variability, and brings down recall accuracy. But this strategy is especially suited when encoding resources limited. Then, better when the resources are assigned to the more probable chunks that are predictive of bigger sequence, while the varying entities that occur with lower probability can be denoted as a variable that serves as a placeholder.
Uncertainty(C)
EP(c)
P(c) Σ H(v)
(14)
GEC
VEC
And (vv) = EcEE(N)
P(c) -
log P(c).
A.10 OTHER EVALUATION MEASURES
Explanatory Volume: IS] The length of the original sequence (in atomic units) divided by the W
length of the parsed sequence (in units of chunks), i.e., The average size of the sequence that the current representation graph can explain at each parsing step.
Sequence Negative Log Likelihood: The lower bound of information content needed encode the sequence S, as S is parsed into chunks (C1, C2,... Cx),, log P(S) = log P(c:)).
Representation Entropy: The uncertainty contained within each chunk parse
Eeec P(c) Erev()) E)er()) -P(ulv) P(x
A. REGRESSING LLM ON HUMAN DATA
We also regressed the negative log likelihood (NLL) of all LLMs against human sequence recall time and presented the resulting R-squared values in Figure TO During the training block, the NLL of LLMs shows stronger correlation with human recall times compared to alternative models. However, when it comes to human transfer, the cognitive models demonstrate much stronger correlation than the LLMs, with HVM aligning most closely to human transfer performance.
23

---

Published as a conference paper at ICLR 2025
Tsinigg
Transfer
0.5
8.Square
0.4
0.0
A6
HDM
1WM
Lamo133 Model
Liama7
LianaTOB
GPT2
Usma 13B Liaman0B Liamana
GPT2
HCM
AL
RVM
Figure 10: Regressing all models' sequence likelihood on human sequence recall time.
24


Reviews:
[{"review_id": "qCMUdPUNfd", "reviewer": "Reviewer_RUZK", "summary": "**Update after rebuttal:** Thanks to the very detailed response by the authors, my questions and potential misunderstandings have been clarified and I have raised my score from a 5 to a 6. I still think the results on a data source specifically designed to highlight the advantages of the model must be taken with a small grain of salt (recognising that there are other good arguments for designing the data source), and the comparison against LZ78 is okay, but it should not be too surprising that LZ78 is not an ideal compressor for the synthetic data source.\n\n---\n\nThe paper tackles the problem of discovering useful abstract representations in sequence prediction, that allow for better compression and transfer (generalization) with the aim of “[capturing] the learning and transfer of abstract representation in human cognition”. To this end, the paper extends a previously proposed model for probabilistic hierarchical chunking, HCM, with the ability to include learnable abstract variables into the chunk dictionary. Learning the model and performing inference with it is somewhat involved (as is generally the case for non-parametric probabilistic models), but explained well in the paper. On synthetic data that is designed to embody the assumptions underlying the model, the model performs very well and outperforms HCM and Lempel-Ziv (LZ78) as a baseline. The model also performs favorably on four natural language datasets (cut into sequences of 1000 characters), and correlates better with human recall times in a color memorization task. The paper finds that in-context learning with LLMs (GPT2 and Llama-2) behaves qualitatively differently on a similar task. Finally, the paper draws a connection between learning abstractions and lossy compression.", "questions": "**Questions and improvements:**\n\n1. The work and claims could be strengthened by evaluating on more datasets that focus on abstraction, but have not been generated by the authors. This is only relevant for a major revision.\n2. Topic models and various forms of hierarchical latent variable models have been used and discussed extensively in linguistics, machine learning, and cognitive science. How does the HVM relate to commonly used topic models (LDA, and more modern ones)? Ideally this is discussed on a technical level in detail, but at the least it needs to be included with more detail in the related work discussion.\n3. How does the generative model relate to the Hierarchical Dirichlet Process (Teh et al. 2006)? \n4. Why was LZ78 chosen as a baseline? It is a lossless general purpose compressor, and has not been designed specifically for natural language data or data with hierarchical structure. Personally I think a much more interesting comparison would be against Context-Tree Weighting (Willems et al. 1995), or maybe the forget-me-not process (Milan et al. 2016), though the latter is perhaps a bit of an overkill (and quite involved to implement).\n5. Table 1: “we took random snippets of 1000 characters and calculated evaluation metrics”, that seems like fairly short sequences. Why 1000? Is there a scalability issue with longer sequences? How sensitive are the results, particularly the comparison against LZ78 / CTW to the sequence length?\n6. The paper mentions lossy compression multiple times, but as far as I understand all evaluation metrics are lossless in the end (more “lossy” models simply require longer coding lengths / have less coding efficiency)? I am struggling to follow section 4.5 (despite having spent half of my PhD on rate distortion theory). For sure LZ78 is not a lossy compressor - it is lossless. Typically, the distinctive feature of lossy compression is that not all prediction/reconstruction errors matter equally, i.e. the distortion function is typically not the log loss (lossy compression requires a quantitative definition of which information is relevant and to which degree, relative to some task/goal; this is what the distortion function does; the log loss treats all information equally). What is the distortion function in the paper? If one is willing to go lossy, there is a famous trade-off between fidelity and “complexity” (really, the information rate): the rate-distortion curve. I have a hard time relating Fig. 6b to a rate-distortion curve - the “Representation Complexity” seems to be more related to the rate than the distortion, but the figure legend says exactly the opposite. And how is “Abstraction Iterations” (Fig. 6c) related to the abstraction level and representation complexity and thus ultimately the distortion (which also applies to L467-477)? I do agree that lossy compression can be used to formalize a particular kind of abstraction, but it seems to me that what is happening in the paper is more similar to a minimum-description-length argument for *lossless compression* (the more complex the model, i.e. the deeper the tree of abstractions, the better it can compress a sequence, but the price to pay for it is by having a more complex model). The mistake may be fully on my side, but please clarify.\n7. Some of the discussion / conclusion is a bit strong. I would not reject the paper based on this, but I have listed concrete issues in the minor comments.\n8. Why was the generative model introduced? Were there no suitable generators or datasets in the literature (that are more widely used)? Which shortcomings of previously used data (generators) does the current paper tackle? (I am leaning towards listing this as a minor point, but I also think that any paper that introduces a new data set or data generation procedure should justify it over using what’s been published and used previously by others).\n\n**Minor comments:**\n1. L 164 - line break within inline equation.\n2. Discussion in L279-284 leaves out that HCM achieves better coding efficiency than HVM if I understand correctly.\n3. L 315: “LZ78 performs well in terms of compression efficiency, which is expected given its design purpose”. I don’t fully agree, LZ78 is a general purpose lossless compressor, it has not been specifically developed to compress natural language.\n4. L 499: “our work provides a reductionist cognitive model that explicitly specifies the minimal components needed for a model to learn interpretable abstract structure from sequences.” - what makes the model particularly “cognitive”? I also mildly disagree that the model “specifies the minimal components”, rather, it is one solution with few components, but it is unclear that this is the minimum needed (and also minimal in which sense?).\n5. L 503: “Our generative mechanism offers a probabilistic, hierarchical sequence generation model relying on chunk-based recursive generation and inventory growth rather than formal grammar rules.” - is this an advantage; does this address some shortcoming in the literature?\n6. L 520: “Previously, grammar learning, chunk learning, and statistical/associative learning were studied in isolation as distinct aspects of sequence learning.” - it should be pointed out that this sentence refers to the cognitive science(?) literature (in other fields, like algorithmic information theory, which deals primarily with sequential learning, this distinction does not play a big role).\n7. L 523: “Our work suggests a normative origin of concrete and abstract chunk learning” - I think the normative claim is a bit overstated in light of the results and no discussion that rules out all other possibilities.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The model is an interesting, sensible, and original improvement over HCM.\n* Empirical results show that the model learns well and works well on synthetic data and some natural language datasets.\n* The work is very well put into wider perspective in the introduction.", "weaknesses": "* Many of the main experiments were conducted on data from a generative model that fits exactly the modeling assumptions for HVM (or tasks inspired by these assumptions). As is often the case when a paper proposes a novel model and a novel data set / generator, the fact that the model works well on data that was specifically designed for the model to work well is not a very strong argument. Luckily the paper also shows good results on “natural” data, and qualitatively matches some aspects of human behavior (recall times) that are not trivially predicted from the synthetic data.\n* Hierarchical Dirichlet processes and related models (perhaps most famously many variants of topic models) have been used and discussed exhaustively in the ML, linguistics, and cognitive science literature. But an in-depth discussion and technical comparison of HVM against these is currently missing.\n* From a cognitive perspective, a severe limitation is that learning in HVM strongly depends on initial learned representations and the order in which learning experiences are represented. While some human learning mechanisms have these traits, I am not sure whether they apply to the learning of abstract concepts in natural language to the extent that the model would predict.\n\n\n**Verdict:**\nThe model as an improvement over HCM is sensible, sound, and results clearly show the benefits. Additionally, the cognitive plausibility (at least on an abstract computational level) of the model is decent (at least to me as a non-linguist), and supported by the experiment with human participants. The paper is well written and presents a nice set of experiments. On the other hand, I think many (but not all!) of the experiments must be taken with a grain of salt, since the data has been either synthetically generated to match the model, or has been designed with the same qualities in mind (like in the color memorization task). The comparison against in-context learning in previous-generation LLMs on the same task (translated into text) is ok, but I am not sure whether there is a big take-away other than saying LLMs in-context learning on this task is different from the model and different from human learning on this task. It is unclear whether LLMs should even be designed to mimic HVM in context (the discussion seems to mildly hint at this by claiming normativity of the model). The work is very interesting to a comp. neurosci. audience and a comp. linguistics audience, but its impact in the ML and AI community is likely to be quite limited (nonetheless, a part of the ICLR audience has a background in the aforementioned fields). Some technical discussion around hierarchical Dirichlet processes (or related models) and topic models is missing. Finally, some of the writing and some claims are perhaps a bit overstated (see concrete points under ‘Questions’). In its current state, for an ML conference, I think the relevance and significance of the current work is fairly limited. I think the paper would benefit from a major revision and could be significantly strengthened to be more impactful. I am therefore currently leaning towards rejection (at a top-tier ML conference).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jvT11Fo1Tl", "reviewer": "Reviewer_NkSJ", "summary": "This paper introduces a hierarchical variable learning model (HVM) that learns abstract patterns from sequences by combining chunking and variable discovery mechanisms. HVM builds on previous chunking models by adding the ability to identify abstract variables that represent groups of chunks that appear in similar contexts. The authors demonstrate that HVM achieves better compression and parsing efficiency compared to baseline models, correlates well with human sequence learning behavior, and provides insights into how abstraction enables better generalization. They evaluate HVM on both synthetic data and real-world language datasets, comparing it against traditional compression algorithms and large language models (LLMs).", "questions": "Please answer the two points under weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Impressively, this paper presents some novel theoretical contribution, a clear theoretical framework for combining chunking and abstraction in sequence learning, with formal proofs and guarantees.\n\nThey also evaluated their model through multiple angles: computational efficiency, correlation with human behavior, comparison with LLMs, a good set of comparisons. \n\nAnd I enjoyed their connection to cognitive science: The work bridges computational and cognitive approaches, providing insights into human learning mechanisms.", "weaknesses": "1. The paper's comparison to LLMs is relatively narrow and focuses primarily on a specific sequence recall task and limited to short sequences, and therefore seems slightly contrived situation. The paper would benefit from explorations of slightly more complex abstraction tasks to study the general applicability of their method. \n\n2. The comparisons in the paper are quite limited and don't adequately address the rich literature on sequence compression and pattern detection. A single example I have in mind of a similar cogntiively-inspired latent variable learning model is CSCGs (https://pmc.ncbi.nlm.nih.gov/articles/PMC8062558/), but there are more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O55C31Dzk6", "reviewer": "Reviewer_dGcC", "summary": "In this work, the authors build on a previous Hierarchical Chunking Model (HCM - a probabilistic model that learns to produce hierarchical chunks from sequential non-iid data) to create Hierarchical Variable learning Model (HVM), which groups chunks into high-level variable categories that have similar interaction properties. This model aims to compress sequences in a structured manner similar to humans. To test this out, the authors used sequence data from a variety of language datasets (childes, bnc, gutenberg, open subtitles) using HCM and LZW, a classical compression algorithm as baselines. The authors then used the model to account for human behavior in a sequence memory task that requires humans to re-use specific variables (against a control where there isn't a reusable variable). They also compared popular LLMs (GPT2, LLama2) on this task. HVM showed the biggest difference between the control and variable groups, which is the main effect that humans had.", "questions": "* What is the relationship between this model and other Bayesian Wake-Sleep class of algorithms, like the older Helmholtz machine or the newer generation such as DreamCoder? There are definitely similarities in having a generative and recognition model, but I didn't see a discussion on this in the paper? I think this would be quite relevant to put in the related works section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* I think this is a really innovative algorithm that builds on the recent HCM in a pretty novel and cool way. It's clearly motivated by the human ability to abstract. \n\n* The evaluation is quite rigorous and showing performance on real world data as well as accounting for human behavior in a relevant task is a very nice touch. \n\n* There are a lot of rigorous proofs in the appendix. The authors have clearly thought a lot about the theoretical foundations of this algorithm as well as shown good empirical proof of its use.", "weaknesses": "* I think the paper can mainly be improved in clarity. For example, when getting to Figure 3, it's kind of hard to figure out which exact datasets these results are from. In general, most of the text in the work is dedicated to describing the algorithm and results. I think adding some more information on what datasets are being used would be useful. For example, line 310: \" BabyLM language dataset, which contain text snippets from a collection of data domains\" what data domains are there? It was hard for me to understand what kind of data this was. \n\n* For Figure 5, I think the authors should plot the difference between the conditions rather than the conditions themselves. Currently, it's hard to understand why HVM shines here more than the LLMs. As I understand, the individual likelihoods don't matter as much as showing a significant decrease from control -> variable conditions. You can put the individual likelihood plots in the supplement. The plot as it is, in my opinion, undersells and obscures the result. \n\n* (minor weakness) I get the feeling that this algorithm isn't quite as scalable as other powerful sequence learning technologies we have today such as LLMs. This is not to say the algorithm is not useful because of that, because of course you do get more interesting structure and interpretability out of it (and it's also a better model of how humans do sequence learning). But I think this is at least worth mentioning in the discussion. If this model does actually have almost as good or better scalability than tools we have today such as SSMs or transformers, I think that would be a huge bonus and definitely needs mentioning. \n\n* (minor weakness) I think a potential missed opportunity for this algorithm is interpreting the discovered variables on specific datasets that we know has rich hierarchical structure. For example, can you use this algorithm on musical notes to recover leitmotifs? But there is enough work here that this can count as future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNjCOHvIXL", "reviewer": "Reviewer_z6tL", "summary": "The authors introduce a hierarchical variable learning model (HVM) that learns abstractions on sequence data via chunking and variable learning. Sequences are first parsed according to a parsing graph containing existing chunks hierarchically. If one chunk follows another chunk in a statistically significant manner, their concatenation is added to the list of chunks alongside hierarchical information. Variables are also identified as a set of chunks that consistently occur before/after another chunk. Variables can also be used in chunk formation.\n\nThe model is tested on both a synthetic dataset and real world sequential datasets (language modeling) and compared against HCM and LZ78 on parsing search steps, sequence length, sequence negative log-likelihood, and encoding efficiency. It is shown that the design of HVM does indeed provide benefits with respect to these metrics.\n\nThe authors also HVM reflects human memorization and transfer performance on a memorization and transfer task.\n\nIn addition, the authors compare HVM performance to LLMs and associative learning models on the same cognitive task.\n\nIt is also shown that higher levels of abstraction in HVM leads to relatively higher likelihood when parsing unseen sequences compared to HCM, suggesting improved generalization.", "questions": "- Would it be possible to use these learned representations for a task that requires abstraction (e.g. abstract reasoning)?\n- When humans perform abstractions, semantic content also manners. Can this be integrated into the approach proposed in the paper?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is organized and well written\n- The paper includes evaluations of the model from many different perspectives, including evaluation on synthetic and real world language datasets, comparison with human performance on a cognitive task, etc\n- The paper has a good level of technical rigour with the addition of definitions, theorems, and algorithms in the appendix.\n- The topic of abstraction is of great significance to the field of AI, and the paper proposes a novel approach to tackling this issue", "weaknesses": "- Some small presentation issues in the appendix: equation in line 1020 is too long, figure 7 has a red line from a word processor, figure 8 has a red line that seems to be an error\n- While I did praise the comprehensive evaluation, it is almost _too_ much content; it was difficult for me to understand the model without referring to the appendix. I suggest including information about the set up in the main text to improve clarity.\n- Since the model builds up chunks by considering frequencies of adjacent chunks, there might be a limit to the expressivity of the formed abstraction. For example, while it might be able to capture the pattern of two chunks consistently being n chunks apart (if the intermediate chunks are all members of the set representing a variable), it cannot capture a pattern of two chunks being a variable number of chunks apart, where the number of chunks is determined by a symbol being present in the sequence.\n- The evaluation only considers coding efficiency, compression efficiency, etc as a proxy for abstraction, which I do not believe to be sufficient for the purpose of demonstrating the effectiveness of the learned abstract representations. The demonstration of generalization does provide some evidence (Q1 below related to this point).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)