Unsupervised Hierarchical Concept Learning

1 INTRODUCTION . Consider a video ( Figure 1 ) that demonstrates how to cook an egg . Humans subconsciously learn concepts ( such as boiling water ) that describe different concepts ( or skills ) in such demonstrations Pammi et al . ( 2004 ) . These learned skills can be composed and reused in different ways to learn new concepts . Discovering such concepts automatically from demonstration data is a non-trivial problem . Shankar et al . ( 2019 ) introduces a sequence-to-sequence architecture that clusters long-horizon action trajectories into shorter temporal skills . However , their approach treats skills as independent concepts . In contrast , humans organize these concepts in hierarchies where lower-level concepts can be grouped to define higher-level concepts Naim et al . ( 2019 ) . We extend the architecture in Shankar et al . ( 2019 ) to simultaneously discover concepts along with their hierarchical organization without any supervision . We propose an end-to-end trainable architecture UNHCLE for hierarchical representation learning from demonstrations . UNHCLE takes as input a long horizon trajectory of high-dimensional images demonstrating a complex task ( in our case , chess and cooking ) and the associated textual commentary and isolates semantically meaningful subsequences in input trajectories . We emphasize that it does not require temporal annotations which link subsequences in the trajectories of images to the freeflowing commentary , but instead , autonomously discovers this mapping . Therefore , this work takes a step towards unsupervised video understanding of high-dimensional data . Our contributions can be summarized as follows : • We introduce a transformer-based architecture to learn a multi-modal hierarchical latent embedding space to encode the various concepts in long-horizon demonstration trajectories . UNHCLE abstracts these concepts ( shown through visual qualitative analysis ) without requiring any temporal supervision , i.e. , it divides long-horizon trajectories into semantically meaningful subsequences , without access to any temporal annotations that split these trajectories optimally . • We show the quantitative effectiveness of learning high-level concepts in a hierarchical manner compared to learning them in isolation while outperforming several baselines on YouCook2 ( Zhou et al . ( 2017 ) ) and Chess Opening dataset1 . • We further introduce a mechanism to incorporate commentary accompanying demonstrations in UNHCLE and show improvements in hierarchical concepts discovered . 1https : //www.kaggle.com/residentmario/recommending-chess-openings • We introduce TimeWarped IoU ( TW-IoU ) , an evaluation metric that we use to compare the alignment of our discovered concepts and ground-truth events . Existing approaches to representation learning for demonstrations or videos typically require significant supervision . Typically , sequence-to-sequence architectures are trained on datasets segmented by humans . During inference , these architectures generate proposals for timestamps that segment the input trajectory into semantically meaningful sequences . These complex sequence-to-sequence models require significant amounts of annotated data , making them costly to train them . More generally , video and scene understanding is an important research area with wide-ranging applications . Most recently , Chen et al . ( 2019 ) utilize semantic awareness to perform complex depth estimation tasks to acquire the geometric properties of 3-dimensional space from 2-dimensional images . Tosi et al . ( 2020 ) utilize similar semantic information for depth estimation , optical flow and motion segmentation . Boggust et al . ( 2019 ) attempt to ground words in the video , but apply significant supervision to synchronize them , requiring human intervention . We attempt to learn similar embeddings but do so in a completely unsupervised manner , not utilizing any of the temporal labels available . The field of learning from demonstrations ( Nicolescu & Mataric ( 2003 ) ) seeks to learn to perform tasks from a set of demonstrated behaviors . Behavioral Cloning is one popular scheme ( Esmaili et al . ( 1995 ) ) . Atkeson & Schaal ( 1997 ) and Schaal ( 1997 ) show how agents can learn simple tasks like cartpole simply from demonstrations . Pastor et al . ( 2009 ) also study how robots can learn from human demonstrations of tasks . Peters et al . ( 2013 ) and Kober & Peters ( 2009 ) fit a parametric model to the demonstrations . Niekum et al . ( 2012 ) , Murali et al . ( 2016 ) , and Meier et al . ( 2011 ) first segment trajectories into subsequence and then apply a parametric models to each subsequence . More recently , Schmeckpeper et al . ( 2019 ) shows that agents can learn to maximize external reward using a large corpus of observation data , i.e. , trajectories of states on a relatively smaller corpus of interaction data , i.e. , trajectories of state-action pairs . Hierarchical task representations have been studied as well . Instead of treating demonstrations in a flat manner , one may also infer the hierarchical structure . A few recent works attempt to do so ( Xu et al . ( 2018 ) ; Sun et al . ( 2018 ) ) , or as task graphs ( Huang et al. , 2019 ) . Both Xu et al . ( 2018 ) and Huang et al . ( 2019 ) address generalizing to new instances of manipulation tasks in the few-shot regime by abstracting away low-level controls . However , all of these approaches require an environment i.e. , a transition and reward function to learn from . On the contrary , humans show an ability to learn by watching demonstrations , which we attempt to replicate . Temporal abstractions of action sequences or skill/primitive learning is also a related field . Eysenbach et al . ( 2018 ) , learn a large number of low-level sequences of actions by forcing the agent to produce skills that are different from those previously acquired . However , due to the diversity bias , the agent results in learning many useless skills that can not be used for any semantically meaningful task . Similarly , Sharma et al . ( 2019 ) attempt to learn skills such that their transitions are almost deterministic in a given environment . These approaches also require access to an environment , whereas we try to learn without an environment . 2 APPROACH . 2.1 UNHCLE : UNSUPERVISED HIERARCHICAL CONCEPT LEARNING . Intuitively , we define a concept as a short sequence of states which repeatedly occur across several demonstration trajectories . Concepts have an upper limit on their length in time-steps . These concepts can be obtained from the images of demonstrations , denoted by z , and from the associated textual description , represented by c. We also refer to them as Observational Concept ( OC ) and Instructional Concept ( IC ) respectively and the module that encodes these concepts is referred to as Observational Concept Abstraction ( OCA ) and Instructional Concept Abstraction ( ICA ) modules . Additionally , concepts are hierarchical in nature - thus , lower-level and higher-level observational or image concepts are denoted by zL and zH , respectively . Analogously , lower-level , and higher-level textual concepts are represented by cL and cH , respectively . Similarly , the higher-level and lower-level modules are denoted by low or high for the corresponding level . Once we obtain these concept abstractions across levels , we can also transform them across the levels using a Concept Regeneration ( CR ) Module to transform low-level concepts to high-level concepts and vice-versa . Instead of just traversing in the concept level hierarchy , we can also use the Concept Instruction Regeneration Module or CIR to obtain the original instructions that map to that concept in its respective concept modality . Subsequently , we provide details about the different stages in our proposed technique . Encoding Observation : Given a long horizon trajectory of demonstration images along with its associated textual description , UNHCLE is able to abstract a hierarchy of concepts from demonstration images . We first pass these input images through ResNet-32 ( He et al . ( 2016 ) ) to get a sequence of image vectors as S = s1 : m , and the associated text is converted into word vectors W = w1 : n using BERT-base ( Devlin et al . ( 2018 ) ) . Observations combine to produce lower-level concepts whereas higher-level concepts are simply aggregations of such lower-level concepts . Thus , the Lower-level Observation Concept Abstraction module ( OCAlow ) is trained to embed a sequence of image vectors of a video ( s1 , s2 .... sm ) into a sequence of concept vectors ( zL1 , z L 2 , .. , z L U ) where u < < m such that zL1 : u = OCA low ( s1 : m ) . Subsequently , lower-level concepts combine together to form a higher level of concepts using the Higher-level Observation Concept Abstraction module ( OCAhigh ) such that zH1 : v = OCA high ( zL1 : u ) . Encoding Instructions : We also endeavour to discover these higher-level and lower-level concepts through natural language instructions . The Lower-level Instruction Concept Abstraction module ( ICAlow ) and Higher-level Instruction Concept Abstraction module ( ICAhigh ) are responsible for this functionality . From a corpus of words , ( w1 , w2 .... wn ) , the ICAlow module generates concept ( cL1 , c L 2 , .. , c L u ) , u < < n : c L 1 : u = ICA low ( w1 : n ) . Subsequently the ICAhigh encodes the lower-level language concepts cL1 : u into higher level concepts as c H 1 : v = ICA high ( cL1 : u ) . Traversing across the concept hierarchy : Learning concepts at different levels has an added advantage of traversing these concepts in the concept hierarchy . This additionally allows us to utilize these hierarchical traversals to obtain coarse or fine-grained concepts at any level . We can thus regenerate the lower concepts from higher-level instruction concepts using the Lower-level Concept Regeneration Module ( CRlow ) such that z′L1 : u′ = CR low ( cH1 : v ) . We can then later utilize this to obtain lower-level concepts from the higher-level concepts to regenerate the demonstration images SL = sL1 : mL in a cross-modal fashion . Observation and Instruction Regeneration : Under a concept , the sequence of frames is nearly deterministic i.e . the knowledge of a concept uniquely identifies the accompanying sequence of images in the demonstration trajectory . Subsequently , we regenerate the demonstration image vectors SL = sL1 : mL from lower-level concepts using Lower-level Concept Observation Regeneration Module ( CORlow ) such that sL1 : mL = COR low ( z′L1 : u′ ) . We also regenerate the demonstration image vectors SU = sU1 : mU from higher-level concepts abstracted from language using Higher-level Concept Observation Regeneration Module ( CORhigh ) such that sU1 : mU = COR high ( cH1 : v ) in a similar cross-modal manner . Finally , inspired by humans who can easily describe concept representations using free-flowing natural language , we first regenerate lower-level concepts from higher-level observation concepts using the Lower-level Concept Regeneration Module ( CRlow ) such that c′L1 : u′ = CR low ( zH1 : v ) , and subsequently , regenerate the word vectors WL = wL1 : nL from lower-level concepts using Lower-level Concept Instruction Regeneration Module ( CIRlow ) such that wL1 : nL = CIR low ( c′L1 : u′ ) . Additionally , the higher-level concepts identified by the OCAhigh module from demonstration frames are also described using a meaningful free-flowing commentary by the Higher-level Concept Instruction Regeneration module or CIRhigh . Thus , we regenerate the word vectors WU = wU1 : nU from higherlevel concepts using CIRhigh such that wU1 : mU = CIR high ( zH1 : v ) .

Summary: This paper addresses the problem of extracting a hierarchy of concepts in an unsupervised way from demonstration data. The authors present a Transformer-based concept abstraction architecture called UNHCLE and show how it discovers meaningful hierarchies using datasets from Chess and Cooking domains. In particular, the model is designed to function without specific temporal supervision, which makes it potentially practical for real-world applications.