Learning Structure from the Ground up---Hierarchical Representation Learning by Chunking

From learning to play the piano to speaking a new language , reusing and recombining previously acquired representations enables us to master complex skills and easily adapt to new environments . Inspired by the Gestalt principle of grouping by proximity and theories of chunking in cognitive science , we propose a hierarchical chunking model ( HCM ) . HCM learns representations from non-i.i.d sequential data from the ground up by first discovering the minimal atomic sequential units as chunks . As learning progresses , a hierarchy of chunk representation is acquired by chunking previously learned representations into more complex representations guided by sequential dependence . We provide learning guarantees on an idealized version of HCM , and demonstrate that HCM learns meaningful and interpretable representations in visual , temporal , visual-temporal domains and language data . Furthermore , the interpretability of the learned chunks enables flexible transfer between environments that share partial representational structure . Taken together , our results show how cognitive science in general and theories of chunking in particular could inform novel and more interpretable approaches to representation learning . 1 INTRODUCTION . The last decade has witnessed a meteoric rise in the abilities of artificial systems , in particular deep learning models ( LeCun et al. , 2015 ) . From beating the world champion of Go ( Silver et al. , 2016 ) to predicting the structure of the human proteome ( Jumper et al. , 2021 ) , deep learning models have accomplished impressive end-to-end learning achievements . Yet , researchers were quick to point out shortcomings ( Marcus , 2018 ; Lake et al. , 2017 ) . Two such shortcomings concern the hierarchical structure and interpretability of the representations learned by neural network architectures . Specifically , deep learning models suffer from a lack of interpretability since ANNs are sub-symbolic , nested , non-linear structures . This means they the way predictions are generated can be difficult to understand ( Samek et al. , 2017 ; Ribeiro et al. , 2016 ; Doshi-Velez & Kim , 2017 ) . Furthermore , deep learning models can also struggle to learn hierarchical representations altogether ( Lake & Baroni , 2018 ; Fodor & Pylyshyn , 1988 ) . To address these shortcomings , it has been suggested that machine learning researchers could seek inspiration from cognitive science and construct models that resemble the hierarchical and interpretable representations observed in human learners ( Chollet , 2017 ; Lake et al. , 2015 ) . We take inspiration from the cognitive phenomenon of chunking and the Gestalt principle of grouping by proximity . To get an intuition for chunking , try to read through the following sequence of letters : “ DFJKJKJKDFDFJKJKDFDF ” . Upon reaching the end , if you were tasked to repeat the letters from memory , you might recall fragments of the sequence such as “ DF ” or “ JK ” . By parsing the sequence of letters only once , one detects frequently occurring patterns within and memorize of them together as units , i.e . chunks . Chunking has been observed in a range of sensory modalities . We perceive units and structures when learning a language ( Perruchet et al. , 2014 ; McCauley & Christiansen , 2017 ) , in action sequences ( Penhune & Steele , 2012 ; Rosenbaum et al. , 1983 ) , and in visual structures Hinton ( 1979 ) ; Brady et al . ( 2009 ) ; Egan & Schwartz ( 1979 ) . The extracted chunks have been argued to facilitate memory compression ( Gobet et al. , 2001 ; Miller , 1956 ) , compositional generalization ( Schulz et al. , 2017 ) , predictive processing ( Koch & Hoffmann , 2000 ; Müssgens & Ullén , 2015 ) , and communication ( Schulz et al. , 2020 ) . In the current work , we propose a hierarchical chunking model ( HCM ) that learns chunks from non-i.i.d sequential data with a hierarchical structure . HCM starts out learning an atomic set of chunks to explain the sequence and gradually combines them into increasingly larger and more complex chunks , thereby learning interpretable hierarchical structures . The output of the model is a dynamical graph that is a trace of the evolving representation . The resulting representations are therefore easy to interpret , and flexibly reusable ( e.g . we can choose to re-use specific parts ) . We derive learning guarantees on an idealized generative model and demonstrate convergence on sequential data coming from this generative model . Furthermore , we show that our model can benefit from having learned previous structures with shared components , leading to flexible transfer and generalization . Finally , we demonstrate that HCM can return interpretable representation in discrete sequential , visual , and language domains , in several small-scale experiments . Taken together , our results show how cognitive science in general and theories of chunking and grouping by proximity in particular can inform novel and more interpretable approaches to representation learning . 2 HIERARCHICAL CHUNKING MODEL . When talking about chunks in a sequence , a chunk is defined as a unit created by concatenating several atomic sequential units together . Take the training sequence shown in Figure 1a as an example . The sequence consists of discrete , size-one atomic units from an atomic alphabet A0 : in this case A0 = { 0 , 1 , 2 } . A chunk c is made up of a combination of one or more atomic units in A0\ { 0 } . 0 denotes an empty observation in the sequence . Intuitively , if the training sequence contains inherent hierarchical structure , then there are patterns which span several sequential units sharing these internal structures , like repeated melodies and submelodies in a piece of music . If the pattern occurs in the sequence , observations between sequential units within the pattern will be correlated . In this case , chunking patterns within a sequence as units simplifies perceptual processing in the sense that the sequence can be perceived one chunk at a time , instead of one sequential unit at a time . Furthermore , the acquired “ primary ” chunks serve as building blocks to discover larger chunks that are embedded within the intricate hierarchy of the sequential structure . More formally , HCM acquires a belief set B of chunks , and uses chunks from the belief set to parse the sequence one chunk at a time . HCM assumes that a sequence is generated from samples of independently occurring chunks with probability of PB ( c ) evaluated on the belief set B . The probability of observing a sequence of parsed chunks c1 , c2 , ... , cN can be denoted as P ( c1 , c2 , ... , cN ) = ∏ ci∈B PB ( ci ) . Chunks as perceiving units serve as independent factors that disentangle observations in the sequence . From the beginning of the sequence , HCM ‘ perceives ’ sequential units one chunk at a time , in other words , the training sequence is parsed by HCM into chunks . At every parsing step , the longest chunk in the belief set that is consistent with the upcoming sequence is chosen to explain the up-coming sequential observations . The end of the previous chunk parse initiates the next parse . Observing a hierarchically structured sequence as illustrated in Figure 1a , HCM gradually builds up a hierarchy of chunks starting from an empty belief set . It first identifies a set of atomic chunks to construct its initial belief set B . Initially , these will be chunks of length one , yielding one-by-one processing of the primitive elements . For one belief set B , HCM keeps track of the marginal parsing frequency M ( ci ) for each chunk ci in B , a vector with size |B| and the transition frequency T between chunk ci followed by chunk cj , as illustrated in Figure 1b . Entries in M and T are used to test the hypothesis that consecutive chunk parses have a correlated consecutive occurrence within the sequence . If two chunks cL and cR have a significant adjacency dependence based on their entries in M and T , they are chunked together to become cL ⊕ cR , which augments the belief set B by one . One example of chunk merging is shown in Figure 1c . There are two different versions of HCM . The Rational Chunk Learning HCM produces chunks in an idealized way which we use to study learning guarantees . The online version of HCM is an approximation to the rational HCM that can be adapted to different environments and processes sequences online . Pseudo-code for both algorithms can be found in Algorithm 1 and 2 in the SI . Rational Chunk Learning : HCM as an Ideal Observer The model is initiated with an empty belief set , and it first finds a minimally complete belief set after the first sequence parse . In each iteration , the entire sequence is parsed to evaluate M and T , which are used to find consecutive chunk parses in the existing belief set fulfilling the dependence testing criterion . From these dependent chunk pairs , the pair with the largest estimated joint probability is combined into a new chunk . The new chunk enlarges the belief set by one . The chunks in the new belief set are used to parse the sequence in the next iteration . This process repeats until all of the chunks in the belief set pass an independence testing criterion . Online Chunk Learning The online chunk learning HCM approximates the ideal observer HCM by learning new chunks when the training sequence is processed online . To have a feature that encourages adaptation to new environmental statistics , entries in M and T can be subject to memory decay . We use the ideal observer model to demonstrate learning guarantees , but use the online model to learn representations in realistic and more complex set-ups . 2.1 HCM BUILDS REPRESENTATIONS FROM THE GROUND UP . As HCM learns from a sequence , it starts with no representation and gradually builds up the representations which can be readily interpreted at any learning stage . The representation can be described by a chunk hierarchy graph Ĝ with the vertex set being the chunks and edges pointing from chunk constituents to the constructed chunks . Shown in Figure 1d is the gradual build-up of one such chunk hierarchy graph as the model learns from a training sequence coming from the generative hierarchy displayed in Figure 1a . At t = 10 , HCM learns only the atomic chunks , at t = 60 , HCM has already constructed two additional chunks ; when t = 100 , two more additional chunks are constructed based on the previously acquired chunks , and at t = 150 , HCM arrives at the final chunk hierarchy . 3 HIERARCHICAL GENERATIVE MODEL . To study learning guarantee of the HCM , a generative model with a random hierarchy of chunks is constructed . The relation between chunks and their constructive components in the generative model is described by a chunk hierarchy graph Gd with vertex set VAd and edge set EAd . One example is illustrated in Figure 1a . Ad is the set of chunks used to construct the sequence . The depth d specifies the number of chunks created in the generative process . Starting with an initial set of atomic chunks A0 , at the i-th iteration , two chunks cL , cR are randomly chosen from the current set of chunks Ai and are concatenated into a new chunk cL⊕cR , augmenting Ai by one to Ai+1 . Meanwhile , an independent occurrence probability is assigned to each chunk under the condition that the probability of occurrence for every new chunk ci in the construction process evaluated on the support set Ai carries the largest probability mass . To generate a sequence from a constructed hierarchical graph , chunks are sampled independently from the set of all chunks and appended after each other , under the constraint that no two chunks with a child chunk should be sampled consecutively .

Summary: This paper presents HCM, an approach for chunking a sequence of data into a hierarchical representation. More specifically, HCM learns a tree with atomic units (ie the low-level inputs, in this case integers representing things like text characters or quantized pixel values) as the leaves and increasingly complex groupings of them higher up the tree.   HCM learns by iteratively parsing the provided data (ie stream of tokens), in each pass computing marginals for the current set of chunks as well as transition frequencies between them. After updating its marginals and transition frequencies, the two chunks with highest joint probability are combined into one. The process continues until all pairs of chunks pass an independence test.   I believe the main contribution of this paper is in that it presents an idea for interpretable grouping based on the principle of grouping by proximity from cognitive science, and a largely qualitative proof of concept for it.