Learning Procedural Abstractions and Evaluating Discrete Latent Temporal Structure

1 INTRODUCTION . A fundamental problem in machine learning is the discovery of structure in unsupervised data . In this work we are particularly interested in uncovering the latent structure in procedural data – potentially high-dimensional observational time-series resulting from some latent sequence of events . For example , a video showing how to change a tire might involve jacking up the car , removing one nut from the wheel , removing a second nut , etc . There exists an enormous wealth of videos of procedures available and inferring the latent structure of such videos could be useful in a huge range of applications , from supporting search queries ( ” find all segments where someone jacks up the car ” ) to adding to nascent work on learning from observation ( Borsa et al. , 2017 ; Stadie et al. , 2017 ; Torabi et al. , 2018 ) in which robots may be taught merely by observing a procedure performed in a video . Procedural learning is related to activity recognition and the broader field of latent temporal dynamical learning , but focuses on the simpler but still ubiquitous setting where the latent activity sequence is a fixed procedure . As we started to develop methods for performing latent procedure inference from temporal data , we considered how to evaluate our resulting methods . One important evaluation protocol for unsupervised learning methods is external evaluation , where predicted labelings of the discrete variables are compared to ground-truth labels ( Rosenberg & Hirschberg , 2007 ) . But the precise criteria for this evaluation is critical . Though there are an enormous number of methods for modeling and inferring latent structure in dynamical systems , including Hidden Markov models , dynamic Bayesian networks , and temporal clustering for time-series data e.g . Fox et al . ( 2008b ; 2014 ) ; Krishnan et al . ( 2015 ) ; Linderman et al . ( 2017 ) ; Zhou et al . ( 2008 ; 2013 ) ; Vidal & Favaro ( 2014 ) , the external evaluation for such approaches is typically done using the same clustering metrics that are used for non-temporal data : metrics like the normalized mutual information ( NMI ) ( Strehl & Ghosh , 2002 ) or by computing cluster correspondences using the Munkres algorithm ( Munkres , 1957 ) . Unfortunately such metrics disregard temporal information and can therefore make it hard to assess important considerations that arise in latent temporal structure extraction , including in procedural inference . To address the limitations of prior evaluation criteria for the external evaluation of discrete latent temporal structure , our first contribution is the creation of new criteria . We identify at least two key aspects of temporal latent structure discovery that are not captured well with existing non-temporal metrics : segment structure , the temporal durations between transitions from one discrete latent state to another , and repeated structure , where a particular latent variable may appear in multiple segments . Figure1 shows an example latent sequence associated with a person performing cardiopulmonary resuscitation ( cpr ) – the colors denote the activity type ( such as blowing air into the lungs ) and the temporal duration of a fixed activity type is shown as a continuous block of color . Our new metrics can evaluate both segment structure and repeated structure , and we later demonstrate how these metrics may be of broader interest in latent temporal dynamical learning , as they can illustrate and illuminate different strengths and weaknesses of existing work on common benchmark datasets . Equipped with new measures for evaluating success , we return to our key goal , inferring the latent procedure observed in a set of high dimensional demonstrations . Past expressive models like Hidden Markov models can struggle in this setting , since the resulting procedure may not be Markov : while the setting can be forced to appear Markov by employing the history as part of the state , this can reduce sample efficiency ( by artificially increasing the latent space size ) . Additionally assuming the system is Markov when it is not can mean inferring stochastic latent dynamical structure when it is in fact deterministic . Recent work ( Sener & Yao , 2018 ) tackles a closely related setting , but does not allow repeated primitives , which is a key aspect of many procedures . Instead we introduce a new generative Bayesian model ( PRISM ) for identifying procedural abstractions without supervision . Results on datasets for procedural tasks – including surgical demonstrations and how-to videos – show that our method can learn procedural abstractions effectively compared with state-of-the-art methods and popular baselines , and outperform prior work on both our new metrics and classic metrics when repeated structure is present . 2 DEFINITIONS AND SETTING . Temporal Clustering . A time-series Xi = ( xi1 , xi2 , . . . , ximi ) , i ∈ [ n ] is a sequence of feature vectors ( or items ) of length mi with xit ∈ Rd . A temporal clustering C = ( c1 , . . . , cm ) is a sequence of cluster labels with ct ∈ ΓC where ΓC is a set of cluster labels . Temporal clusterings map each item in the feature trajectory X to a cluster label in ΓC . We use Ci as shorthand to denote the temporal clustering generated by some method for Xi . The ground truth temporal clustering for Xi will be denoted by Gi , with corresponding ground truth label set ΓG . For simplicity , we define Ca : b = ( ca , ca+1 , . . . , cb ) where 1 ≤ a ≤ b ≤ m are time indices . For α ∈ ΓC , let C [ α ] = { t|ct = α , 1 ≤ t ≤ m } be the set of time indices in C whose cluster labels equal α . C [ α ] is the cluster or partition ( in the standard sense ) corresponding to the label α. Temporal clusterings are distinct from time-series segmentation ( Chung et al. , 2004 ) and changepoint detection ( Killick et al. , 2012 ) , which consider only identifying boundaries of different dynamic regimes in the time-series . Segment . A segment is a contiguous sequence of identical cluster labels in C. Each cluster in C may be split across multiple segments . We represent each segment as a pair of time indices ( a , b ) , where a represents the start time-index of the segment while b the end time-index ( both included ) . Formally , we let S ( C , α ) = { ( a , b ) |Ca : b = ( ca = α , ca+1 = α , . . . , cb = α ) , 1 ≤ a ≤ b ≤ m } be a function that maps a temporal clustering C and cluster label α ∈ Γ to the set of segments associated with α in C. We let SC = ⋃ α∈ΓC S ( C , α ) be the set of all segments in C. Procedure . Let P be a function that removes running duplicates from a sequence of labels , yielding a sequence of tokens . For instance , P ( A , A , A , B , B , B , C , C , C , A , A , B ) = ( A , B , C , A , B ) yielding 5 tokens . For a temporal clustering C , we define its corresponding procedure to be P ( C ) . The procedure captures the primitives ( as tokens ) present in the temporal clustering and the sequence in which they occur . We say that C exhibits repeated structure if atleast one primitive is reused in C. This corresponds to having multiple tokens of the same cluster label in P ( C ) . The procedure ( A , B , C , A , B ) contains repeated structure since both A and B are reused . We also define a weight function W that counts the number of running duplicates in a sequence e.g . W ( A , A , A , B , B , B , C , C , C , A , A , B ) = ( 3 , 3 , 3 , 2 , 1 ) . Notice that W computes the length of each segment in the sequence . Weights in W ( C ) thus have a one-to-one correspondence with tokens in P ( C ) , equaling the length of each token in the procedure . Taken together W ( C ) , P ( C ) can be used to recover C. Lastly , we letH ( P1 , P2 , W1 , W2 ) be a similarity measure between 2 temporal clusterings represented by P1 , W1 and P2 , W2 respectively . Fig 2 illustrates these ideas for a temporal clustering defined over the label set Γ = { A , B , C , D } . The procedure exhibits repeated structure since B has ( atleast ) 2 tokens/segments corresponding to it . 3 EVALUATING LATENT TEMPORAL STRUCTURE . External clustering evaluation is the process of comparing a clustering to a ground-truth labeling in order to evaluate it . Unfortunately , as we discuss below , using traditional clustering evaluation criteria is problematic for evaluating temporal clusterings . Existing evaluation criteria . The use of standard clustering metrics for evaluation is prevalent in prior work that attempts to evaluate latent temporal structure ( Zhou et al. , 2008 ; 2013 ; Fox et al. , 2008b ; 2009 ; 2014 ; Krishnan et al. , 2015 ; Sener & Yao , 2018 ; Hoai & De la Torre , 2012 ) . For a temporal clustering C the widely used purity ( Rosenberg & Hirschberg , 2007 ) metric penalizes the presence of items from different ground truth labels in the same cluster . This is a desirable property that a clustering evaluation criterion should have . Purity is defined as purity = 1m ∑ α∈ΓC maxβ∈ΓG |G [ β ] ∩ C [ α ] | , A related metric is homogeneity ( Rosenberg & Hirschberg , 2007 ) , which captures a similar idea in a different way : homogeneity = 1− H ( ΓG |ΓC ) H ( ΓG ) where H ( ΓG ) is the entropy of the clustering G when partitioned by the labels ΓG , H ( ΓG |ΓC ) is the conditional entropy of G given C. Intuitively the conditional entropy term looks inside every cluster in C , and checks the entropy of the items inside in terms of the ground truth labels . The conditional entropy will be low ( and homogeneity high ) if every cluster in C contains items of only a single ground truth label . Another important metric is completeness ( Rosenberg & Hirschberg , 2007 ) , which prefers clusterings where all items from a ground-truth label lie in the same cluster i.e . the ground truth label is not split across clusters . Completeness is defined identically to homogeneity , but with the clusterings swapped G ←→ C. Both completeness and homogeneity are considered important criteria for clustering . Maximizing either at the expense of the other is bad – high homogeneity and low completeness imply a clustering that is too fine-grained ( e.g . every item in its own cluster ) , while low homogeneity and high completeness imply a clustering that is too coarse ( e.g . a single cluster containing all items ) . Therefore , metrics that balance these 2 criterion often perform best ( Rosenberg & Hirschberg , 2007 ) – normalized mutual information ( NMI ) ( Strehl & Ghosh , 2002 ) and V-measure ( Rosenberg & Hirschberg , 2007 ) are examples of such metrics . Another popular suite of metrics uses the Munkres algorithm ( Zhou et al. , 2008 ) – a one-to-one correspondence between clusters and ground-truth labels is computed so that any classification criterion ( commonly accuracy ) under this correspondence is maximized . Characterizing latent temporal structure . Two major characteristics distinguish the evaluation of temporal clusterings from the standard clustering case . • Repeated structure . The presence of repeated structure in ground-truth – ground-truth labels that are split across several segments in the trajectory . Ideally , we want that this repeated structure is identified correctly e.g . in Fig 3 identifying that the segments A0:2 , A4:6 are repetitions of the same primitive . • Segment structure . The locations and sizes of the segments in ground-truth e.g . in Fig 3 we would like temporal clusterings that switch between primitives at exactly the time points t = 2 , 4 with the same 3 segment structure . Neither repeated or segment structure are seen in standard clusterings , since they treat data as a bag of items . Limitations of existing criteria . Existing metrics can not score repeated or segment structure because they convert temporal clusterings to standard clusterings – a contingency matrix ( Rosenberg & Hirschberg , 2007 ) – before evaluation . This erases the temporal information important to score these ideas . In fact , qualitatively distinct temporal clusterings can be mapped to the same contingency matrix , and receive the same score , which is clearly undesirable . We use Fig 3 as an illustrative example to highlight these shortcomings . In Fig 3 , C1 perfectly matches G , while both C2 and C3 split the cluster G [ A ] – C2 into C2 [ X ] and C2 [ Z ] and C3 into C3 [ X ] and C3 [ Z ] . In addition , C2 also misses the repeated structure in A while C3 does not . C3 captures this repeated structure as XZ , with X0:1Z1:2 corresponding to A0:2 and X4:5Z5:6 corresponding to A4:6 . However , it does so at the cost of creating more segments than C2 – ignoring labels , C2 has a segmentation exactly like G. Intuitively , we prefer C1 > C3 > C2 if we care more about repeated structure , and C1 > C2 > C3 , if we care more about the segmentation quality . However , all 3 temporal clusterings C1 , C2 , C3 in Fig 3 are assigned a perfect purity or homogeneity , since as seen in Fig 3b , no cluster in any of the 3 contains items from more than one ground-truth label . Moreover , no traditional clustering criteria can distinguish C2 from C3 since their cluster composition is identical , as seen in Fig 3b . We now discuss new evaluation criteria that can systematically evaluate and score both repeated and segment structure .

Summary: In "Learning procedural abstractions and evaluating discrete latent temporal structure" the authors develop a hierarchical Bayesian model for patterns across time in video data. They also introduce new metrics for understanding structure in time series (completeness and homogeneity). This work is appropriate for ICLR. They provide some applications to robotics, suggesting that this could be used to teach robots to act in environments by learning from videos.