Learning from Demonstration with Weakly Supervised Disentanglement

1 INTRODUCTION . Learning from Demonstration ( LfD ) ( Argall et al. , 2009 ) is a commonly used paradigm where a potentially imperfect demonstrator desires to teach a robot how to perform a particular task in its environment . Most often this is achieved through a combination of kinaesthetic teaching and supervised learning—imitation learning ( Ross et al. , 2011 ) . However , such approaches do not allow for elaborations and corrections from the demonstrator to be seamlessly incorporated . As a result , new demonstrations are required when either the demonstrator changes the task specification or the agent changes its context—typical scenarios in the context of interactive task learning ( Laird et al. , 2017 ) . Such problems mainly arise because the demonstrator and the agent reason about the world by using notions and mechanisms at different levels of abstraction . An LfD setup , which can accommodate abstract user specifications , requires establishing a mapping from the high-level notions humans use—e.g . spatial concepts , different ways of applying force—to the low-level perceptive and control signals robot agents utilise—e.g . joint angles , efforts and camera images . With this in place , any constraints or elaborations from the human operator must be mapped to behaviour on the agent ’ s side that is consistent with the semantics of the operator ’ s desires . Concretely , we need to be able to ground ( Vogt , 2002 ; Harnad , 1990 ) the specifications and symbols used by the operator in the actions and observations of the agent . Often the actions and the observations of a robot agent can be high-dimensional—high DoF kinematic chains , high image resolution , etc.—making the symbol grounding problem non-trivial . However , the concepts we need to be able to ground lie on a much-lower-dimensional manifold , embedded in the high-dimensional data space ( Fefferman et al. , 2016 ) . For example , the concept of pressing softly against a surface manifests itself in a data stream associated with the 7 DoF real-valued space of joint efforts , spread across multiple time steps . However , the essence of what differentiates one type of soft press from another nearby concept can be summarised conceptually using a lower-dimensional abstract space . The focus of this work is finding a nonlinear mapping ( represented as a high-capacity neural model ) between such a low-dimensional manifold and the high-dimensional ambient space of cross-modal data . Moreover , we show that , apart from finding such a mapping , we can also shape and refine the low-dimensional manifold by imposing specific biases and structures on the neural model ’ s architecture and training regime . In this paper , we propose a framework that allows human operators to teach a PR2 robot about different spatial , temporal and force-related aspects of a robotic manipulation task , using tabletop dabbing and pouring as our main examples . They serve as concrete representative tasks that incorporate key issues specific to robotics ( e.g . continuous actions , conditional switching dynamics , forceful interactions , and discrete categorizations of these ) . Numerous other applications require the very same capability . Our main contributions are : • A learning method which incorporates information from multiple high-dimensional modalities—vision , joint angles , joint efforts—to instill a disentangled low-dimensional manifold ( Locatello et al. , 2019 ) . By using weak expert labels during the optimisation process , the manifold eventually aligns with the human demonstrators ’ ‘ common sense ’ notions in a natural and controlled way without the need for separate post-hoc interpretation . • We release a dataset of subjective concepts grounded in multi-modal demonstrations . Us- ing this , we evaluate whether discrete latent variables or continuous latent variables , both shaped by the discrete user labels , better capture the demonstrated continuous notions . 2 IMAGE-CONDITIONED TRAJECTORY & LABEL MODEL . In the task we use to demonstrate our ideas , which is representative of a broad class of robotic manipulation tasks , we want to control where and how a robot performs an action through the use of a user specification defined by a set of coarse labels—e.g . “ press softly and slowly behind the cube in the image ” . In this context , let , x denote a K × T dimensional trajectory for K robot joints and a fixed time length T , y denote a set of discrete labels semantically grouped in N label groups G = { g1 , . . . , gN } ( equivalent to multi-label classification problem ) and i denote an RGB image1 . The labels y describe qualitative properties of x and x with respect to i—e.g . left dab , right dab , hard dab , soft dab , etc . We aim to model the distribution of demonstrated robot-arm trajectories x and corresponding user labels y , conditioned on a visual environment context i . This problem is equivalent to that of structured output representation ( Walker et al. , 2016 ; Sohn et al. , 2015 ; Bhattacharyya et al. , 2018 ) —finding a one-to-many mapping from i to { x , y } ( one image can be part of the generation of many robot trajectories and labels ) . For this we use a conditional generative model , whose latent variables c = { cs , ce , cu } can accommodate the aforementioned mapping—see Figure 2 . The meaning behind the different types of latent variables—cs , ce and cu—is elaborated in section 3 . 1What i actually represents is a lower-dimensional version of the original RGB image I . The parameters of the image encoder are jointly optimised with the parameters of the recognition and decoder networks We want the variability in the trajectory and label data to be concisely captured by c. We choose by design the dimensionality of the latent variables to be much lower than the dimensionality of the data . Therefore , c is forced to represent abstract properties of the robot behaviour which still carry enough information for the behaviour to be usefully generated—absolute notions like speed , effort , length , and relative spatial notions which are grounded with respect to the visual context . Another way to think of c is as a continuous version of y which can therefore incorporate nuances of the same label . In the existing literature discrete labels are usually represented as discrete latent variables—e.g . digit classes ( Kingma et al. , 2014 ) . However , the discrete labels humans use are a rather crude approximation to the underlying continuous concepts—e.g . we may have the notion for a soft and a softer dab even though both would be alternatively labelled as soft . For this reason , we use continuous latent variables , shaped by discrete labels , to represent these subjective concepts . The joint distribution over x and y , conditioned on i , is modelled according to Eq . 1 . A prior of choice—isotropic Gaussian , Uniform , Categorical—is placed over the latent variables c which are independent of the image . p ( x , y|i ) = ∫ p ( x , y|i , c ) p ( c ) dc = ∫ p ( x|c , i ) p ( y|c ) p ( c ) dc ( 1 ) We choose to represent the distribution over x and y as Gaussian and Multinomial distributions respectively . The parameters of these distributions are defined as nonlinear functions ( of the image context and latent variables ) which are represented as the weights θ of a neural network pθ—µθ ( · ) is a vector of means , σθ ( · ) is a vector of log variances and πθ ( · ) is a vector of probabilities . pθ ( x|c , i ) = N ( x|µθ ( c , i ) , σθ ( c , i ) ) ( 2 ) pθ ( y|c ) = Cat ( y|πθ ( c ) ) ( 3 ) The parameters of pθ are optimised by maximising the variational lower bound ( VLB ) of the data distribution—see Eq . 4 . We additionally optimise a recognition network qφ , parametrised by φ , which acts as an approximation to the posterior of c , also modelled as a Gaussian . The recognition network qφ is used to find good candidates of c , for given x and i , thus making the optimisation of θ , tractable ( Kingma & Welling , 2013 ) . The posterior is conditioned on the environment context as well , since i and c are conditionally dependent , once x is observed ( Husmeier , 2005 ) . Intuitively , we want some of the latent variables to represent relative spatial concepts—e.g . left of the cube in the image . For such concepts , under the same robot trajectory x we should get different latent codes , given different contexts i . The omission of y from qφ means that we might not be able to fully close the gap between the data distribution and its VLB , leading to a less efficient optimisation procedure ( due to higher variance ) . That is mainly owing to the fact that we use weak supervision—only for some demonstrations do we have labels y . The derivation of and commentary on the optimised VLB are provided in the Supplementary Materials . log p ( x , y|i ) ≥ Eqφ ( c|x , i ) ( log pθ ( x , y|c , i ) ) − DKL ( qφ ( c|x , i ) ||p ( c ) ) ( 4 ) qφ ( c|x , i ) = N ( c|µφ ( x , i ) , σφ ( x , i ) ) ( 5 ) 3 WEAK LABELLING AND USER SPECIFICATION . Interpretability through Weak Labels Even though the latent variables are constrained to be useful both for the tasks of trajectory and label generation , nothing forces them to carry the intuitive absolute and relative notions described in Section 2 in a factorised fashion—e.g . for dabbing , c1 to correspond to the effort applied when pressing against the table , c2 to correspond to how quickly the robot presses against the table , etc . Under an optimised model , those notions can be captured in c but with no guarantee that they will be represented in an interpretable way— i.e . c needs to be disentangled . We achieve disentanglement in the latent space by establishing a one-to-one correspondence between a subset of the latent axes—cs , ce—and the concept groups in G ( Hristov et al. , 2018 ) . cs is optimised to represent spatial notions , ce—effort-related and temporal notions . The rest of the latent dimensions—cu—encode features in the data which don ’ t align with the semantics of the labels and their concept groups but are still necessary for good trajectory reconstruction ( Figure 2 ) . Under these assumptions the label likelihood from Eq . 3 becomes : pθ ( y| { cs , ce } ) = |G|∏ j 1 { yj 6=∅ } Cat ( yj |πj ( cj ) ) ( 6 ) Labelling is weak/restricted since each demonstration is given just a single label from a single concept group . This is meant to represent a more natural LfD scenario , where the main task of the expert is still to perform a number of demonstrations , rather than to exhaustively annotate each demonstration with relevant labels from all groups . For example , a demonstrator might say “ this is how you press softly ” and proceed to perform the demonstration ( as is common , say , in an instructional video ) . The demonstrated behaviour might also be slow and short but the demonstrator may not have been explicitly thinking about these notions , hence may not have labelled accordingly . The missing label values are incorporated with an indicator function in Eq . 6 . Condition on User Specification Apart from being used in the optimisation objective , the weak labels are also used in an additional conditioning step at test time . The generative process we have consists of first sampling values for c from its prior . These are then passed together with i through pθ in order to get distributions over x and y , from which we can sample . However , we are really interested in being able to incorporate the provided information about the labels y into the sampling of semantically aligned parts of c. Specifically , we are interested in being able to generate robot behaviours which are consistent with chosen values for y ( what we call a user specification ) . Postoptimising the θ and φ parameters , we choose to approximate the posterior over c for each label l in each concept group j with a set of Gaussian distributions N ( µjl , Σjl ) . We use Maximum Likelihood Estimation for µjl and Σjl over a set of samples from c. The samples are weighted with the corresponding label likelihood pθ ( yj = l|πj ( cj ) ) . As a result , the process of generating a trajectory x ∼ pθ ( x|i , c ) can be additionally conditioned on an optional user specification y = { y1 , . . . , y|G| } , through c , in a single-label fashion ( Eq . 7 ) or a compositional fashion ( Eq . 8 ) . Pseudo-code for the generation of trajectories for both types of conditioning is provided in the Supplementary Materials . c ∼ p ( c|yj = l ) = N ( c|µjl , Σjl ) ( 7 ) ∀j ∈ { 1 , . . . , |c| } : cj ∼ p ( cj |yj = l ) = { prior , if yj = ∅ or j > |G| N ( c|µjl , Σjl ) ( j ) , otherwise ( 8 )

Summary: This is a well-written paper that discusses how to learn disentangled representations for the learning from demonstrations (LfD) task in robotics. It is shown that using weak-supervision on top of unsupervised learning frameworks (that use the variational autoencoder for instance) can work well in this case. These disentangled factors of variation in the data are shown to correspond well to the 'abstract concepts' of the human demonstrations. This is shown in the example of the PR2 robot dabbing demonstrations, including visual data as well robot trajectories.