Unsupervised Hierarchical Concept Learning

1 INTRODUCTION . Consider a video ( Figure 1 ) that demonstrates how to cook an egg . Humans subconsciously learn concepts ( such as boiling water ) that describe different concepts ( or skills ) in such demonstrations Pammi et al . ( 2004 ) . These learned skills can be composed and reused in different ways to learn new concepts . Discovering such concepts automatically from demonstration data is a non-trivial problem . Shankar et al . ( 2019 ) introduces a sequence-to-sequence architecture that clusters long-horizon action trajectories into shorter temporal skills . However , their approach treats skills as independent concepts . In contrast , humans organize these concepts in hierarchies where lower-level concepts can be grouped to define higher-level concepts Naim et al . ( 2019 ) . We extend the architecture in Shankar et al . ( 2019 ) to simultaneously discover concepts along with their hierarchical organization without any supervision . We propose an end-to-end trainable architecture UNHCLE for hierarchical representation learning from demonstrations . UNHCLE takes as input a long horizon trajectory of high-dimensional images demonstrating a complex task ( in our case , chess and cooking ) and the associated textual commentary and isolates semantically meaningful subsequences in input trajectories . We emphasize that it does not require temporal annotations which link subsequences in the trajectories of images to the freeflowing commentary , but instead , autonomously discovers this mapping . Therefore , this work takes a step towards unsupervised video understanding of high-dimensional data . Our contributions can be summarized as follows : • We introduce a transformer-based architecture to learn a multi-modal hierarchical latent embedding space to encode the various concepts in long-horizon demonstration trajectories . UNHCLE abstracts these concepts ( shown through visual qualitative analysis ) without requiring any temporal supervision , i.e. , it divides long-horizon trajectories into semantically meaningful subsequences , without access to any temporal annotations that split these trajectories optimally . • We show the quantitative effectiveness of learning high-level concepts in a hierarchical manner compared to learning them in isolation while outperforming several baselines on YouCook2 ( Zhou et al . ( 2017 ) ) and Chess Opening dataset1 . • We further introduce a mechanism to incorporate commentary accompanying demonstrations in UNHCLE and show improvements in hierarchical concepts discovered . 1https : //www.kaggle.com/residentmario/recommending-chess-openings • We introduce TimeWarped IoU ( TW-IoU ) , an evaluation metric that we use to compare the alignment of our discovered concepts and ground-truth events . Existing approaches to representation learning for demonstrations or videos typically require significant supervision . Typically , sequence-to-sequence architectures are trained on datasets segmented by humans . During inference , these architectures generate proposals for timestamps that segment the input trajectory into semantically meaningful sequences . These complex sequence-to-sequence models require significant amounts of annotated data , making them costly to train them . More generally , video and scene understanding is an important research area with wide-ranging applications . Most recently , Chen et al . ( 2019 ) utilize semantic awareness to perform complex depth estimation tasks to acquire the geometric properties of 3-dimensional space from 2-dimensional images . Tosi et al . ( 2020 ) utilize similar semantic information for depth estimation , optical flow and motion segmentation . Boggust et al . ( 2019 ) attempt to ground words in the video , but apply significant supervision to synchronize them , requiring human intervention . We attempt to learn similar embeddings but do so in a completely unsupervised manner , not utilizing any of the temporal labels available . The field of learning from demonstrations ( Nicolescu & Mataric ( 2003 ) ) seeks to learn to perform tasks from a set of demonstrated behaviors . Behavioral Cloning is one popular scheme ( Esmaili et al . ( 1995 ) ) . Atkeson & Schaal ( 1997 ) and Schaal ( 1997 ) show how agents can learn simple tasks like cartpole simply from demonstrations . Pastor et al . ( 2009 ) also study how robots can learn from human demonstrations of tasks . Peters et al . ( 2013 ) and Kober & Peters ( 2009 ) fit a parametric model to the demonstrations . Niekum et al . ( 2012 ) , Murali et al . ( 2016 ) , and Meier et al . ( 2011 ) first segment trajectories into subsequence and then apply a parametric models to each subsequence . More recently , Schmeckpeper et al . ( 2019 ) shows that agents can learn to maximize external reward using a large corpus of observation data , i.e. , trajectories of states on a relatively smaller corpus of interaction data , i.e. , trajectories of state-action pairs . Hierarchical task representations have been studied as well . Instead of treating demonstrations in a flat manner , one may also infer the hierarchical structure . A few recent works attempt to do so ( Xu et al . ( 2018 ) ; Sun et al . ( 2018 ) ) , or as task graphs ( Huang et al. , 2019 ) . Both Xu et al . ( 2018 ) and Huang et al . ( 2019 ) address generalizing to new instances of manipulation tasks in the few-shot regime by abstracting away low-level controls . However , all of these approaches require an environment i.e. , a transition and reward function to learn from . On the contrary , humans show an ability to learn by watching demonstrations , which we attempt to replicate . Temporal abstractions of action sequences or skill/primitive learning is also a related field . Eysenbach et al . ( 2018 ) , learn a large number of low-level sequences of actions by forcing the agent to produce skills that are different from those previously acquired . However , due to the diversity bias , the agent results in learning many useless skills that can not be used for any semantically meaningful task . Similarly , Sharma et al . ( 2019 ) attempt to learn skills such that their transitions are almost deterministic in a given environment . These approaches also require access to an environment , whereas we try to learn without an environment . 2 APPROACH . 2.1 UNHCLE : UNSUPERVISED HIERARCHICAL CONCEPT LEARNING . Intuitively , we define a concept as a short sequence of states which repeatedly occur across several demonstration trajectories . Concepts have an upper limit on their length in time-steps . These concepts can be obtained from the images of demonstrations , denoted by z , and from the associated textual description , represented by c. We also refer to them as Observational Concept ( OC ) and Instructional Concept ( IC ) respectively and the module that encodes these concepts is referred to as Observational Concept Abstraction ( OCA ) and Instructional Concept Abstraction ( ICA ) modules . Additionally , concepts are hierarchical in nature - thus , lower-level and higher-level observational or image concepts are denoted by zL and zH , respectively . Analogously , lower-level , and higher-level textual concepts are represented by cL and cH , respectively . Similarly , the higher-level and lower-level modules are denoted by low or high for the corresponding level . Once we obtain these concept abstractions across levels , we can also transform them across the levels using a Concept Regeneration ( CR ) Module to transform low-level concepts to high-level concepts and vice-versa . Instead of just traversing in the concept level hierarchy , we can also use the Concept Instruction Regeneration Module or CIR to obtain the original instructions that map to that concept in its respective concept modality . Subsequently , we provide details about the different stages in our proposed technique . Encoding Observation : Given a long horizon trajectory of demonstration images along with its associated textual description , UNHCLE is able to abstract a hierarchy of concepts from demonstration images . We first pass these input images through ResNet-32 ( He et al . ( 2016 ) ) to get a sequence of image vectors as S = s1 : m , and the associated text is converted into word vectors W = w1 : n using BERT-base ( Devlin et al . ( 2018 ) ) . Observations combine to produce lower-level concepts whereas higher-level concepts are simply aggregations of such lower-level concepts . Thus , the Lower-level Observation Concept Abstraction module ( OCAlow ) is trained to embed a sequence of image vectors of a video ( s1 , s2 .... sm ) into a sequence of concept vectors ( zL1 , z L 2 , .. , z L U ) where u < < m such that zL1 : u = OCA low ( s1 : m ) . Subsequently , lower-level concepts combine together to form a higher level of concepts using the Higher-level Observation Concept Abstraction module ( OCAhigh ) such that zH1 : v = OCA high ( zL1 : u ) . Encoding Instructions : We also endeavour to discover these higher-level and lower-level concepts through natural language instructions . The Lower-level Instruction Concept Abstraction module ( ICAlow ) and Higher-level Instruction Concept Abstraction module ( ICAhigh ) are responsible for this functionality . From a corpus of words , ( w1 , w2 .... wn ) , the ICAlow module generates concept ( cL1 , c L 2 , .. , c L u ) , u < < n : c L 1 : u = ICA low ( w1 : n ) . Subsequently the ICAhigh encodes the lower-level language concepts cL1 : u into higher level concepts as c H 1 : v = ICA high ( cL1 : u ) . Traversing across the concept hierarchy : Learning concepts at different levels has an added advantage of traversing these concepts in the concept hierarchy . This additionally allows us to utilize these hierarchical traversals to obtain coarse or fine-grained concepts at any level . We can thus regenerate the lower concepts from higher-level instruction concepts using the Lower-level Concept Regeneration Module ( CRlow ) such that z′L1 : u′ = CR low ( cH1 : v ) . We can then later utilize this to obtain lower-level concepts from the higher-level concepts to regenerate the demonstration images SL = sL1 : mL in a cross-modal fashion . Observation and Instruction Regeneration : Under a concept , the sequence of frames is nearly deterministic i.e . the knowledge of a concept uniquely identifies the accompanying sequence of images in the demonstration trajectory . Subsequently , we regenerate the demonstration image vectors SL = sL1 : mL from lower-level concepts using Lower-level Concept Observation Regeneration Module ( CORlow ) such that sL1 : mL = COR low ( z′L1 : u′ ) . We also regenerate the demonstration image vectors SU = sU1 : mU from higher-level concepts abstracted from language using Higher-level Concept Observation Regeneration Module ( CORhigh ) such that sU1 : mU = COR high ( cH1 : v ) in a similar cross-modal manner . Finally , inspired by humans who can easily describe concept representations using free-flowing natural language , we first regenerate lower-level concepts from higher-level observation concepts using the Lower-level Concept Regeneration Module ( CRlow ) such that c′L1 : u′ = CR low ( zH1 : v ) , and subsequently , regenerate the word vectors WL = wL1 : nL from lower-level concepts using Lower-level Concept Instruction Regeneration Module ( CIRlow ) such that wL1 : nL = CIR low ( c′L1 : u′ ) . Additionally , the higher-level concepts identified by the OCAhigh module from demonstration frames are also described using a meaningful free-flowing commentary by the Higher-level Concept Instruction Regeneration module or CIRhigh . Thus , we regenerate the word vectors WU = wU1 : nU from higherlevel concepts using CIRhigh such that wU1 : mU = CIR high ( zH1 : v ) .

Summary: The paper introduces the solution of an important task: hierarchical concept learning(or temporal abstractions) from demonstration data. Specifically, this paper considers 1) unsupervised setting; 2) the hierarchy of concepts, and conducts experiments in two datasets. However, there are some points in the experiment section to be discussed.