Learning from Demonstration with Weakly Supervised Disentanglement
Authors: Yordan Hristov
Subramanian Ramamoorthy
Venue: ICLR 2021 Poster (2021)
Keywords: representation learning for robotics
physical symbol grounding
semi-supervised learning

Abstract: Robotic manipulation tasks, such as wiping with a soft sponge, require control from multiple rich sensory modalities. Human-robot interaction, aimed at teach- ing robots, is difficult in this setting as there is potential for mismatch between human and machine comprehension of the rich data streams. We treat the task of interpretable learning from demonstration as an optimisation problem over a probabilistic generative model. To account for the high-dimensionality of the data, a high-capacity neural network is chosen to represent the model. The latent variables in this model are explicitly aligned with high-level notions and concepts that are manifested in a set of demonstrations. We show that such alignment is best achieved through the use of labels from the end user, in an appropriately restricted vocabulary, in contrast to the conventional approach of the designer picking a prior over the latent variables. Our approach is evaluated in the context of two table-top robot manipulation tasks performed by a PR2 robot – that of dabbing liquids with a sponge (forcefully pressing a sponge and moving it along a surface) and pouring between different containers. The robot provides visual information, arm joint positions and arm joint efforts. We have made videos of the tasks and data available - see supplementary materials at: https://sites.google.com/view/weak-label-lfd.

Published as a conference paper at ICLR 2021
LEARNING FROM DEMONSTRATION WITH WEAKLY SUPERVISED DISENTANGLEMENT
Yordan Hristov
Subramanian Ramamoorthy
School of Informatics
School of Informatics
University of Edinburgh
University of Edinburgh
yordan. www.whi.stttooe.mm uc
s wamamoorthhy@dd uc.
ABSTRACT
Robotic manipulation tasks, such as wiping with soft sponge, require control from multiple rich sensory modalities. Human- interaction, aimed at teach- ing robots, difficult in this setting as there is potential for mismatch between human and machine comprehension of the rich data streams. We treat the task of interpretable learning from demonstration as an optimisation problem over prob- abilistic generative model. To account for the high- high-dimensionality of the data, a high-capacity neural network is chosen to represent the model. The latent vari- ables in this model explicitly aligned with high-level notions and concepts that are manifested in a set of demonstrations. We show that such aligmment best achieved through the use of labels from the end user, in an appropriately restricted vocabulary, in contrast to the conventional approach of the designer picking a prior over the latent variables. Our approach evaluated the context of two table-top robot manipulation tasks performed by PR2 robot that of dabbing liquids with a sponge (forcefully pressing a sponge and moving along a surface) and pouring between different containers. The robot provides visual information, arm joint po- sitions and arm joint efforts. We have made videos of the tasks and data available see supplementary materials at: https://sites.goot wwwgle.coo//viiw///////ooo/o/oow wwtp://ww//wwwww....../. wwt.d/iewwww--labl-lllee... -laeel....
INTRODUCTION
Learning from Demonstration (LiD) Argall et al. 2009) is a commonly used paradigm where a potentially imperfect demonstrator desires to teach a robot how to perform particular task in its environment. Most often this is achieved through combination of kinaesthetic teaching and su- pervised learning-imitationn imitation learning (Ross et al., 201T). However, such approaches do not allow for elaborations and corrections from the demonstrator to be seamlessly incorporated. As result, new demonstrations are required when either the demonstrator changes the task specification or the agent changes its -tyicxt-typical scenarios in the context of interactive task learning (Laird al. 2017). Such problems mainly arise because the demonstrator and the agent reason about the world by using notions and mechanisms at different levels of abstraction. An Lff) setup, which can commodate abstract user specifications, requires establishing a mapping from the high-level notions humans use-e.g. spatial concepts, different ways of applying force-to to the low-level perceptive and control signals robot agents utilise-c.g. joint angles, efforts and camera images. With this in place, any constraints or elaborations from the human operator must be mapped to behaviour on the agent's 's side that is consistent with the semantics of the operator's 's desires.
Concretely, we need be able ground (Vogt 2002 Harnad 1990) the specifications and symbols used by the operator in the actions and observations of the agent. Often the actions and the observa- tions of robot agent can be nigh-dimensional- high DoF kinematic chains, high image resolution, etc. etc.-making the symbol grounding problem non-trivial. However, the concepts we need to be able manifold, embedded in the high-dimensional data space (Fefferman et al. 2016). For example, the concept of pressing softly against surface manifests itself in data stream associated with the 7 DoF real-valued space of joint efforts, spread across multiple time steps. However, the essence of what differentiates type of soft press from another nearby concept can be summarised conceptually using a dwweediimeniionall abstract space. The focus of this work is finding nonlinear mapping (represented high-capacity neural model) between such
to ground lie on a much-lower--imensionn

---

Published as a conference paper at ICLR 2021
left
image
hard
fast
slow
soft
trajectories
right
label specification
latent manifold
new trajectories X
User demos through teleoperation and variety of modalities (left) are used to fit com- mon low-level disentangled manifold (middle) which contributes in an interpretable way to the generative process for new robot behaviour (right).
Figure I:
low-dimensional manifold and the high- nigh-dimensional ambient space of cross-mode data. Moreover, show that, apart from finding such mapping, we can also shape and refine the low-dimensional manifold by imposing specific biases and structures on the neural model's 's architecture and training regime.
In this paper, propose framework that allows human operators to teach PR2 robot about differ- ent spatial, temporal and force-related aspects of a robotic manipulation task, using tabletop dabbing and pouring as our main examples. They serve as concrete representative tasks that incorporate key issues specific to robotics (c.g. continuous actions, conditional switching dynamics, forceful inter- actions, and discrete categorizations of these). Numerous other applications require the very same capability. Our main contributions are:
A learning method which incorporates information from multiple high-dimensional modalities-vision,, joint angles, joint efforts-to instill disentangled low-dimensional manifold (Locatello al., 2019). By using weak expert labels during the optimisation process, the manifold eventually aligns with the human demonstrators' 'common sense' notions in natural and controlled way without the need for separate post-hoc interpreta- tion.
We release a dataset of subjective concepts grounded in multi-modal demonstrations. Us- ing this, evaluate whether discrete latent variables or continuous latent variables, both shaped by the discrete user labels, better capture the demonstrated continuous notions.
2 IMAGE-CONDITIONES TRAJECTORY & LABEL MODEL
In the task we use to demonstrate our ideas, which is representative of broad class of robotic manipulation tasks, want control where and how robot performs action through the of a user specification defined by set of coarse labels-e.g. "press softly and slowly behind the cube in the image". In this context, let, denote joints and a fixed time length T. denote a set of discrete labels semantically grouped in N label groups G (.1,..., IN (equivalent multi-label classification problem) and denote an RGB imagd" The labels y describe qualitative properties of x and x with respect to i-e.g. left dab, right dab, hard dab, soft dab, etc. We aim model the distribution of demonstrated robot-arm trajectories and corresponding user labels y, conditioned on visual environment context i. This problem equivalent to that of structured output representation (Walker et al. 2016 [Shnn et al.) 2015 Bhattacharyya et al. can be part of the generation of many robot trajectories and labels). For this we use a conditional generative model, whose latent variables c {c,, Cz, C.) can accommodate the aforementioned mapping- see Figure _ The meaning behind the different types of latent variables Cs, Ce and C, is elaborated in section
T'' dimensional trajectory for K robot
one-to-many mapping from ( y (one image
-ee
What actually represents is a lower-dimensional version of the original RGB image I. The parameters of the image encoder are jointly optimised with the parameters of the recognition and decoder networks
2

---

Published as a conference paper at ICLR 2021
We want the variability in the trajectory and label data to be concisely captured by We choose by design the dimensionality of the latent variables to be much lower than the dimen- sionality of the data. Therefore, is forced represent abstract properties of the robot be- haviour which still carry enough information for the hehaviour to be usefully generated-absolle notions like speed, effort, length, and relative spatial notions which spect to the visual context. Another way to think of is as a continuous version of y which can therefore incorporate nuances of the same label. In the existing literature discrete labels are usually represented as discrete latent variables e.g. digit classes (Kingma et al. 2014 However, the discrete labels humans use are rather crude approximation to the underlying continuous concepts- e.g. we may have the no- tion for soft and a so softer dab even though both would be alternatively labelled as soft. For this reason, ables, shaped by discrete labels, to represent these subjective concepts.
grounded with
continuous latent vari-
y
Figure The lower-dimensional encoding i of the environment context I is observed. Conditioned on i and the latent variables c, sampled from the prior over c, we get a distribution over possible robot trajectories x and user labels
The joint distribution over x and y, conditioned i, is modelled according to Eq. [ A prior of choice- isstropicc Gaussian, Uniform, Categorical-is placed over the latent variables which are independent of the image.
y(x, = f ylxx
p(x|c, (yp(y)ly(ylpppp)))
We choose to represent the distribution over and y as Gaussian and Multinomial distributions respectively. The parameters of these distributions are defined as nonlinear functions (of the image context and latent variables) which are represented as the weights 0 of a neural network po po-po(-) vector of means, ox(-) (.) is vector of log variances and TT(-)) is vector of probabilities.
Pe(x c, = [(x]p
(2)
po(y c) = Cat(y|/r(c)))
(3)
The parameters of po optimised by maximising the variational lower bound (VLB) of the data distribution-sse Eq. We additionally optimise a recognition network Jg. parametrised by ø. the posterior of c, also modelled network qs is used to find good candidates of c, for given x and i, thus making the optimisation of tractable (Kingma & Welling 2013). The posterior is conditioned on the environment context as well, since i and с are conditionally dependent, once x is observed (Husmeier (005). Intuitively, we want some of the latent variables represent relative spatial concepts e.g. left of the cube in the image. For such concepts, under the same robot trajectory x we should get different latent codes, given different contexts i. The omission of y from ds means that we might not be able to fully close the gap between the data distribution and its VLB, leading to a less efficient optimisation procedure (due to higher variance). That is mainly owing to the fact that we use weak supervision- only for some demonstrations do we have labels y. The derivation of and commentary on the optimised VLB are provided in the Supplementary Materials.
which acts an approximation
Gaussian. The recognition
log yxxxyyii 2 Bys(e po ylc,
(4) qa(c|x, -X)))1((x))) (x(xx)) (5)
DxL ())lp())))
WEAK LABELLING AND USER SPECIFICATION
Interpretability through Weak Labels Even though the latent variables are constrained to be useful both for the tasks of trajectory and label generation, nothing forces them to carry the intuitive absolute and relative notions described in Section 2 in a factorised fashion fashion-e.g. for dabbing, e₁ to correspond to the effort applied when pressing against the table, C2 to correspond to how quickly the robot presses against the table, etc. Under an optimised model, those notions can be captured in but with guarantee that they will be represented in an interpretable way- i.e. needs to be
3

---

Published as a conference paper at ICLR 2021
disentangled. We achieve disentanglement in the latent space by establishing a one-to-one corre- spondence between a subset of the latent axes Cz. C. and the concept groups in g (Hristov et al. 2018). C, is optimised to represent spatial notions, - -ffort-related and temporal notions. The rest of the latent dimensions--C. cn-eesido encode features in the data which don't t align with the semantics of the labels and their concept groups but are still necessary for good trajectory reconstruction (Figure 2). Under these assumptions the label likelihood from Eq. becomes:
10]
ce() II (yy (c,))
(6)
Labelling is weak/restricted since each demonstration is given just single label from a single concept group. This is meant to represent more natural LfD scenario, where the main task of the expert is still to perform a number of demonstrations, rather than to exhaustively annotate each demonstration with relevant labels from all groups. For example, a demonstrator might say "this is how you press softly" and proceed to perform the demonstration (as common, say, in an instruc- tional video). The demonstrated behaviour might also be slow and short but the demonstrator may not have been explicitly thinking about these notions, hence may not have labelled accordingly. The missing label values are incorporated with an indicator function in Eq. 6
Condition on User Specification Apart from being used in the optimisation objective, the weak labels are also used in an additional conditioning step at test time. The generative process we have consists of first sampling values for e from its prior. These are then passed together with i through Po in order to get distributions over x and y. from which we can sample. However, we are really interested in being able to incorporate the provided information about the labels y into the sampling of semantically aligned parts of c. Specifically, we are interested in being able to generate robot behaviours which are consistent with chosen values for y (what we call user specification). Post- optimising the 0 and & parameters, we choose to approximate the posterior over с for each label I in each concept group j with set of Gaussian distributions N (pztt Ej²) We use Maximum Likelihood Estimation for μy and Σjt over a set of samples from c. The samples are weighted with the corresponding label likelihood po (y;; (cy)) As result, the process of generating a trajectory x ~ po [x]i, c) can be additionally conditioned on an optional user specification through c, in single-label fashion (Eq. ? or compositional fashion (Eq. 88 Pseudo-code for the generation of trajectories for both types of conditioning is provided in the Supplementary Materials.
fy1.-.- ) U[c))
c~ p(cly, = 1) N(clug Eni)
[prior,
y; = = or 191
Vj e (l,....//c/:: p(c,ll, =)
[N(clum En)e),
otherwise
(8)
4 METHODOLOGY
In terms of concrete model architecture, our model is a Conditional Variational Autoencoder (CVAE) (Sohn et al. 2015) with an encoding network da and a decoding network Pa- The pa- rameters of both networks optimised jointly using methods for amortised variational inference and a stochastic gradient variational bayes estimation (Kingma & Welling 2013
Encoding & Decoding Networks Due to the diversity of modalities that we want use, de is implemented as a combination of 2D convolutions (for the image input), ID convolutions (for the trajectory input) and an MLP that brings the output of the previous two modules to the common concept manifold c. For the decoding network, we implement a Temporal Convolutional Network (TCN) (Bai et 2018 which operates over the concatenation h of single concept embedding and single image encoding i. However, the TCN takes sequence of length T and transforms it another sequence of length T, by passing it through series of dilated convolutions, while is equivalent to a sequence of length 1. Thus, we tile the concatenated vector T times and to each
Thhe models are implemented in PyTorch Adam al. 2017 and optimised using the Adam optimiser
Kingma& & Ba 2014)
4

---

Published as a conference paper at ICLR 2021
This broadcasting technique has previously shown to achieve better disentanglement in the latent space both on visual (Watters et al. 2019) and time-dependent data (Noseworthy et al., 2019). We only set up the mean H4g of the predicted distribution over x to be non-linear function of h: t| while o is fixed, for better numerical stability during training. This allows to the 1.2 distance- Meaan Squared Error betwenn the inferred trajectories μl, acting as reconstruction, and the ground truth ones.
instance of that vector h,, {I...., T), attach time dimension t,
Label Predictors For each concept group 9 we take values from the corresponding latent axis oj and pass it through single linear layer, with a softmax activation function, to predict a probability vector æ. Maximising the label likelihood is realised as Softmax--Cross Entropy (SCE) term in the loss function. Optimising this term gives us a better guarantee that some of the latent axes will be semantically aligned with the notions in the label groups. The fact that some labels might be missing is accounted for with an indicator function which calculates V8, Vg with respect to the SCE, only By is present.
Weighted Loss The full loss function that we optimise is presented in Eq. It is composed of three main terms-the trajectory MSE, equivalent to the negative trajectory log-likelihood the label SCE, equivalent to the negative weak label log-likelihood and the KL divergence between the fit amortised posterior over and its prior. The concrete values of the three coefficients a, 3, are discussed in the Supplementary Materials.
10
min C(x. y. I) (gss(x(( 1) 0.0.w
p (clx.)) (log pe (x I))+7 Σ
1 (1)z)) H (o.w y1)
amortised
prior
MSE
SCE
postetior
EXPERIMENTS
The models we experiment with include Variational Autoencoder (Kingma & Welling 2013) with Gaussian prior (VAE), an Adversarial Autoencoder (Makhzani et al. 2015) with Uniform prior (AAE) and a VAE with mixed continuous and categorical latent variables (GS). For the last model, the continuous variables have Gaussian prior and the categorical variables utilise the Gumbel- Softmax reparametrisation (Jang et al., 2017). Each of the three models, is trained while utilising the weak lahels-VAF. VAF-weak AAE-weak, GS- GGS-weak-and with no label information VAF, AAE, GS. Whether or not labels are used during training is controlled by setting
in Eq. 9
The setup used in the experiments consists of a PR2 robot, HTC Vive controller and headset, and a tabletop with a single object on it-a red cube. Multiple pressing motions on the surface of the table are demonstrated by teleoperating the end effector of the robot. The pose of the end effector, with respect to the robot's torso, is mapped to be the same as the pose of the controller (held by the demonstrator) with respect to the headset (behind the demonstrator in Figure
Data A total of 100 demonstrations
captured from the PR2, consisting of:
Initial frame of each demonstration: 128x128-pixel Kinect2 image (RGB);
7-DoF joint efforts;
5 groups of discrete weak labels;
7-DoF joint angle positions;
Each demonstration has a single label attached to it. In total, we have 4 spatial symbols-ve where in the image, with respect to the red cube, do we dab-and and force-related symbols-how do we dab (Table IT. Figure 3 provides graphical depiction demonstrat- ing the differences between the different force-relate groupings qualitatively.
spatial (where)
effort (how) Joint Efforts soft & hard short & long fast slow
Image + Joint Angles
left & right front & behind
Table I: All labels and modalities used in the demonstrations.
All trajectories are standardised to be of fixed length-the the length of the longest trajectory (240 timesteps) by padding them with the last value for each channel/joint. Additionally, both the joint posi-
5

---

Published as a conference paper at ICLR 2021
soft
hard
short
long
slow
fast
4.0
300 (b)
259 203
300
260
100 230
(a)
(c)
(d)
(e)
(f)
Figure 3: Example user-given labels across different demonstrations. Depicted are effort tra-
jectories from the training data for single robot joint- sie-sshollde derulderr lift joint- (a) and (b) designate the maximal exerted effort, (c) and (d) designate the length for which the maximal effort maintained, (e) and (f) designate the time reach the maximal effort (slope).
tions and efforts are augmented with Gaussian noise or by randomly sliding them leftwards (sim- ulating an earlier start) and padding accordingly. The size of the total dataset after augmentation 1000 demonstrations which are split according to a 90-10 training-validation split. The size of the latent space is chosen to be |cl 8. In the context of the problem formulation in Section c, = C, ce = {c2, c3, C and Cu = {cs, Co, 07
Conditioning on a Single Label We the Gaussian distributions for each label in each concept group j- Nu,,, - sample and generate trajectories that are consistent with the meaning of particular label. For set of 5 test images, which have not been seen during training, draw 1000 samples of per image for each the 10 labels. For each label, then judge whether the generated trajectories match the semantics of the corresponding label using series of manually-designe heuristics, described in the Sup- plementary Materials. Average accuracy across all images and all samples is reported for each label. For the labels of spatial nature, for each 7-dimensional (7 DoFs) robot state J;; x {a1,.... .TT), we compute the corresponding end-effector pose p; through the forward kinematies K of the robot. Using the camera model of the Kinect2 sensor, each pose p; is projected in the image condition on. report the Mean Absolute Error (MAE) in normalised pixel coordinates between the dab location and the landmark location (in pixel coordinates) for the misclassified trajectories. This gives us a sense of how far from the desired ones the generated samples are. The dab location is inferred as the point in the end-effector pose trajectory with the lowest 2 coordinate. For example, we have conditioned on the left label but some of the sampled trajectories result in a dab to the ri git of the eube in the image, report how far from the true classification of le was the robot when it touched the table.
Figure 4: Physical setup for teleoperating a PR2 end-effector through an HTV Vive controller.
Conditioning on Composition of Labels Since the concepts we encode in the latent space of the model should be independent of each other they can composed together-e.g. trajectory can be simultaneously hard, long and fast. We examine how well is this conceptual independence preserved in the latent space by generating trajectories conditioned on possible combinations of labels regarding effort, speed and length of press-8 possible combinations in total. We still use the Gaussian distributions for each label in each concept group but only take the values along the relevant latent axis Eq. [ To fully close the loop, the trajectories which sample from the model could further be executed on the physical robot through a hybrid position/force controller (Reibert, 1981). However, such an evaluation beyond the scope of the paper.

---

Published as a conference paper at ICLR 2021
6 RESULTS & DISCUSSION
Our experiments demonstrate that the models which utilise weak user labels within our proposed setup can more reliably generate behaviours consistent with a given spatial or force-related user specification, in contrast to baseline models which do not benefit from such information. Moreover, we find that continuous latent variables better capture the independent continuous notions which underlie the discrete user labels in comparison to discrete latent variables. Below we present quan- titative results from the experiments with further qualitative results in the Supplementary Materials.
When conditioning on single force Tabel-Table 3 the best-performing models are VAE-weak and AeE-weak. In the worse-nerforning cases VAE, GS-it would appear that the models suffer from partial mode collapse. For instance, in Table penultimate row, the VAE can not represent soft trajectories, most samples are hard. Interestingly AAE and AAE- perform similarly well. We attribute that to the fact the Uniform prior used does not have discrete number of modes to which the optimised posterior might potentially collapse. Simultaneously, regardless of their prior, none of the weak models suffers from mode collapse, since the label information helps for the corresponding notions to be more distinguishable in the latent space. All models have high accuracy, when conditioned on a spatial label-Table 2 However, GS-weak, VAE-weak and AAE- weak appear to be closer to the true conditioned-on concept when a bad trajectory is generated, as measured by the lower Mean Absolute Error of the image-projected end-effector positions.
left
right
front
back
Model
Acc
MAE
Acc
MAE
Acc
MAE
Acc
MAE
GS
0.83 (0.09)
0.052
0.76 (0.22)
0.040
0.85 (0.09)
0.039
0.90 (0.10)
0.060
GS-weak
0.78 (0.16)
0.034
0.92 (0.13)
0.013
0.92 (0.09)
0.030
0.94 (0.06)
0.028
AAE
0.84 (0.12)
0.043
0.88 (0.09)
0.043
0.81 (0.15)
0.039
0.97 (0.03)
0.024
AAB-weak
0.84 (0.24)
0.018
0.89 (0.12)
0.039
0.83 (0.19)
0.030
0.98 (0.04)
0.022
VAE
0,80 (0.18)
0.026
0.82 (0.19)
0.031
0.86 (0.08)
0.050
0.97 (0.05)
0.029
VAE-weak
0.95 (0.07)
0.031
0.87 (0.14)
0.026
0.93 (0.10)
0.027
0.99 (0.02)
0.018
Table 2: Accuracy (mean and std deviation; higher mean is better) and MAE (lower is better) for sampled trajectories under all models, conditioned on a fixed spatial label. Bold numbers designate when the weak version of the model (utilising the weak user labels) outperforms the non-weak one.
Model
soft
hard
short
long
slow
fast
GS
0.26 (0.01)
0.96 (0.01)
0.95 (0.00)
0.76 (0.01)
0.92 (0.01)
0.77 (0.01)
GS-weak
0.84 (0.01)
0.98 (0.00)
0.98 (0.00)
0.69 (0.01)
0.88 (0.01)
0.74 (0.02)
AAE
0.98 (0.00)
1.00 (0.00)
1.00 (0.00)
0.75 (0.01)
0.93 (0.01)
0.64 (0.01)
AAE-weak
0.98 (0.00)
0.99 (0.00)
0.99 (0.00)
0.97 (0.01)
0.84 (0.01)
0.84 (0.01)
VAE
0.04 (0.01)
0.95 (0.00)
0.96 (0.01)
0.59 (0.01)
0.93 (0.01)
0.52 (0.01)
VAE-weak
0.92 (0.00)
1.00 (0.00)
1.00 (0.00)
0.90 (0.01)
0.95 (0.00)
0.68 (0.01)
Table 3: Accuracy (mean and std deviation) for sampled trajectories under all models, conditioned on fixed effort label. Bold numbers designate when the weak version of the model (utilising the weak user labels) outperforms the non-weak one.
While single-label conditioning is possible under all models, it is only VAE-weak, AAE-weak and GS-weat which provide a reliable mechanism for composing labels from different independent con- cept groups. That is possible due to the explicit alignment of each concept and its labels with a single latent axis. Trajectory generation done after conditioning on compositions of labels from the three independent concept groups designating effort- see Table [ (right column). Specifically, the difference between the sampled latent values for hard long slow and hard long fast, for example, are only along the axis used to predict s slow from fast-ca in this case. If the gener- ated trajectories match their specification (high measured accuracy) then indeed the perturbed axis is aligned with the notion behind the perturbed labels. Results from this experiment are presented in Figure 5 The wetter-performing models are the ones with continuous latent variables. While the VAE-weak and AAE-weak perform equally well for most label combinations and do manage to
7

---

Published as a conference paper at ICLR 2021
GS-weak AAE-weak
VAE-weak
0.75
accuracy
0.50
0.00
soft, short, slow soft. short, fast sott. long, slow
soft, long, fast
hard, shart, slow hard, short, fast hard, lona, slow
hard, long, fast
combinations
Figure 5: Accuracy comparison for generated trajectories conditioned on all compositions of binary labels from the three concept groups for effort, length and speed. The models with continuous latent spaces outperform the discrete one as the underlying concepts are continuous themselves.
encode independent concepts as such (along specific latent axes), the model with the mix of con- tinuous and categorical latent variables (GS-weak) does not match their results. Taking the second row in Table 3 into account, this means that the categorical model (which has more constrained capacity than the VAE-weak and AAE-weak) can capture the data distribution (shows good single- label-conditioning reconstructions) but in entangled fashion. Namely, going from corresponds to perturbing more than single latent axis simultaneously. This further supports our initial hypothesis that while discrete, the weak labels only provide mechanism for shaping the latent representation of what are otherwise continuous phenomena.
fast
7 POURING EXPERIMENT
An additional pouring task is demonstrated where the manner of execution varies spatially and in
terms of how it is executed see Table 4 and Figure 6 20 demonstrations are captured for each label. The size of the latent space is kept to be |cl = 8. In the context of the problem formulation in Section ? c, {co}, Ca {c1, 02) and Cu {c3; Ca, C5; CB; 07).
cup
cup
partially
sideways
behind
Figure 6: Example user-given labels across different demonstrations for pouring.
When compared to the dabbing setup, this task presents more significant challenge in identi- fying the switching latent dynamics for pouring each cup (compared to the single red cube in the previous task, acting reference point for resolving spatial prepositions).
spatial (where)
manner (how)
Image + Joint Angles red cup & blue cup
Joint Angles
behind & sideways
fully & partially
Table 4: All labels and modalities used in the ad- ditional pouring demonstrations.
Interpolation experiments for the VAE-weak model are presented Figure 7 for test cup configurations. The first latent dimension of the model-en. optimised to predict in which cup pour-is linearly interpolated in the range [-2, 2]. In all four rows, we can see the end effector projections in the image plane, through the robot kinematics and camera model, showing a gradual shift from initially pouring in the red cup to eventually pouring in the blue cup. The color evolution of each end-effector trajectory from cyan magenta signifies time progression. Moreover, given the disentangled nature of the VAE-weak model, we also generate robot trajectories conditioned on specific values for the 3 latent axes (Co, e1, O2), consistent with example label compositions- see Figure For picking the right value for each axis we use the post-training trraininn--stimatedd estimated Normal distributions for each label (consult Section 3). Additional qualitative results and sensitivity analysis of the model to the heterogeneous input modalities are presented in Appendix B and the supplementary website
8

---

Published as a conference paper at ICLR 2021
Luaba
Clil
609
$1.00
Figure 7: Linear interpolation from pouring in the red cup to pouring in the blue cup.
red bahind fut
Mie Vilaass
downays partel
Figure 8: Qualitative comparison of demonstrated robot trajectories and sampled trajectories, gen- erated after conditioning on composition of labels. Each pair consists of a curated ground-truth demo, characterised by particular label composition (red boxes) and a respective sample from the optimised model. Each sample plot consists of in-image projected end effector trajectories (top row) together with x/y/z (middle row) and roll/pitch/yaw (bottom row) end effector trajectories.
RELATED WORK
neural models to learn disentangled di-ddimnsionnal represen- tation achieve that by either tuning the optimised loss function (Higgins al.) 2017 Chen et al.] 2018), imposing crafted priors over the latent space (Chen et al. 2016) or using weak supervision (Hristov al. 2019 Rasmus al.) 2015). While being able produce interpretable manifolds, most of these approaches focus modelling visual data and respectively visuall--manifesee con- cepts, with minor exceptions-e.g modelling motion capture data (Ainsworth et al., 2018). In contrast, we solve problem involving multiple heterogeneous modalities-vision forces, joint angles. Moreover, addition to separating factors of variation in the data, to achieve latent space interpretability, also deal with the problem of entanglement across modalities -ie. which factors relate to which data modalities.
Methods that utilise
Dynamic Movement Primitives (Schaal 2006) and Probabilistic Movement Primitives (Paraschos) et al. 2013) are commonly used to model dynamical systems as 'point-mass attractors with a non- linear forcing term', However, the resulting control parameters can be nigh-dimensional and un- intuitive for end-user manipulation. Performing dimensionality reduction (Colomé et al. 2014) or imposing hierarchical priors (Rueckert al. 2015) both ideas seen in the literature means of making the dingh-iimnsional parameter space more meaningful to the human demonstrator. These have the advantage of yielding a clearer idea of how changes in the optimised parameters result in deterministic changes in the generated behaviours. However, most of these approaches limit them- selves to end-effector trajectories, or at most making use of visual input (Kober et al.) 2008).
Noseworthy et al. 2019) explore disentangling task parameters from the manner of execution pa- rameters, in the context of pouring. They utilise an adversarial training regime, which facilitates better separation of the two types of parameters in the latent space. However, it is assumed that the task parameters are known a priori. Interpretation of the learned codes is achieved post-training by perturbing latent axis values and qualitatively inspecting generated trajectories (standard evalua- tion technique for models which are trained unsupervised). We argue that through the use of weak supervision through discrete labels we have finer control over the 'meaning' of latent dimensions.

---

Published as a conference paper at ICLR 2021
9 CONCLUSION
We recontextualise the problem of interpretable multi-modal LfD as a problem of formulating a conditional probabilistic model. In the example task of table-top dabbing, we utilise weak discrete labels from the demonstrator to represent abstract notions, manifested in a captured multi-modal robot dataset. Through the use of high-capacity neural models and methods from deep variational inference, we show how to align some of the latent variables of the formulated probabilistic model with high-level notions implicit in the demonstrations.
ACKNOWLEDGMENTS
This work is partly supported by funding from the Turing Institute, part of the Safe Al for surgical assistance project. Additionally, we thank the members of the Edinburgh Robust Autonomy and Decisions Group for their feedback and support.
REFERENCES
Paszke Adam, Gross Sam, Chintala Soumith, Chanan Gregory, Yang Edward. D Zachary, Lin Zem- ing, Desmaison Alban, Antiga Luca, and Lerer Adam. Automatic differentiation in pytorch. In Proceedings of Neural Information Processing Systems, 2017.
Samuel K. Ainsworth, Nicholas J. Foti, Adrian K. C. Lee, and Emily B. Fox. oi-VAE Output inter- pretable VAEs for nonlinear group factor analysis. In Jennifer Dy and Andreas Krause (eds.), Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 119 128, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. htt::tt:
mir.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning, A survey of robot learning from demonstration. Robotics and autonomous systems, 4765):469-483, 2009.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional
and recurrent networks for sequence modeling. arXiv preprint arXiv: 100.0087,,, 2018.
Apratim Bhattacharyya, Bernt Schiele, and Mario Fritz. Accurate and diverse sampling of sequences based on a "best of many" sample objective. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8485-8493, 2018.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-2180, 2016.
Adrià Colomé, Gerhard Neumann, Jan Peters, and Carme Torras. Dimensionality reduction for probabilistic movement primitives. In 2014 IEEE-RAS International Conference on Humanoid Robots, pp. 794-800. 800. IEEE, 2014.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049 2016.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335- 346. 1990.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5)66, 2017.
Yordan Hristov, Alex Lascarides, and Subramanian Ramamoorthy. Interpretable latent spaces for learning from demonstration. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference Robot Learning, volume 87 of Proceedings of Machine
10

---

Published as a conference paper at ICLR 2021
Learning Research, pp. 957-968. PMLR, 29-31 Oct 2018. URL http://proceedinga. mlr. /02es//////hissttolBBaa..
Yordan Hristov, Daniel Angelov, Michael Burke, Alex Lascarides, and Subramanian Ramamoor- thy. Disentangled relational representations for explaining and learning from demonstration. In Conference on Robot Learning (CoRL., 2019.
Dirk Husmeier. Introduction to learning bayesian networks from data. In Probabilistic modeling in bioinformatics and medical informatics, pp. 17-57. Springer, 2005.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmax. In International Conference Learning Reprexentations (ICLR 2017). OpenReview. net, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.698 69800, 2014.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervee learning with deep generative models. In Advances neural information processing systems, pp- 3581-3589, 2014.
Jens Kober, Betty Mohler, and Jan Peters. Learning perceptual coupling for motor primitives. In 2008 JEEE/RS International Conference Intelligent Robots and Systems, pp. 834 839. IEEE, 2008.
John E Laird, Kevin Gluck, John Anderson, Kenneth D Forbus, Odest Chadwicke Jenkins, Christian Lebiere, Dario Salvucci, Matthias Scheutz, Andrea Thomaz, Greg Trafton, et al. Interactive task learning. IEEE Intelligent Systems, 32(4):6-21, 2017.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. International Conference on Machine Learning, pp. 4114 4124, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprint 1rXiv:1571.0564. 2015.
Michael D. Noseworthy, Rohan Paul, Subhro Roy, Dachyung Park, and Nicholas Roy. Task- conditioned variational autoencoders for learning movement primitives. 2019.
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic move- ment primitives. In Advances neural information processing systems, pp. 2616 2624, 2013.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In Advances in Neural Information Processing Systems, pp. 3546-3554, 2015.
MH Reibert. Hybrid position/force control of manipulators. ASME, of Dynamic Systems, Mea- surement, and Control, 103:2-12, 1981.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, 2011.
Elmar Rueckert, Jan Mundo, Alexandros Paraschos, Jan Peters, and Gerhard Neumann. Extracting low-dimensional control variables for movement primitives. 2015 IEEE International Confer- on Robotics and Automation (ICRA), pp. 1511-1518. IEEF, 2015.
Stefan Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid roboties. In Adaptive motion of animals and machines, pp. 261-280. Springer, 2006.

---

Published as a conference paper at ICLR 2021
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, pp. 3483-3491, 2015.
Paul Vogt. The physical symbol grounding problem. Cognitive Systems Research, 3(3):429-477, 2002.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference Computer Vision, pp. 835-851. Springer, 2016.
Nick Watters, Loic Matthey, Chris P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for disentangled representations in vaes. 2019.
A VARIATIONAL LOWER BOUND DERIVATION
For completeness we include derivation of the Variational Lower Bound of the data likelihood. The VLB is used when the amortised posterior q(]]x,i, i) matches exactly the true posterior p(c,x, y.). The posterior optimise is not conditioned on y, potentially meaning that might never close the gap between the log of the data distribution and the VLB, measured by DxL. (g(c)x. i) []]p(c|x,i,, y)) However, maximising the VLB still goes in the direction maximising the data distribution.
optimisation objective-Eii Eq. 9 Jensen's inequality becomes equality
p(xp(xy10 y/i) log
p(x)yli
log
p(xli, p(xli,cllpsieepp(edde
q(c/x, i)
= log
n(c|xx i)
p(x x c)p(yc)pp(c)
= =log Ey
(c||x, i)
p(x (x c)pxlcc)p())
Eg(elx.)) log
[Jensen]
q(c[x,i)
Ex((/),)) log p(ppp((xxppp + () logp(y/c) - 1 log g(c]x,
p(c)
By(c(x.1) logr(x/i, + Ey()) logr(y/c) - i)llp(c))
-Dkk(((()x, illp(c) - 1 p(xp( c) - 1g(e(x.1) logp(y/c))
-()
(10)
B LABEL CONDITIONING & EVALUATION HEURISTICS
Algorithms and 2 provide pseudo-code for the trajectory generation procedures described in Sec- tion The main difference can be summarised as following- the generative process conditioned on single label 1, all 8 dimensions of the latent samples are drawn from single 8-dimensional Gaussian distribution associated with that label and concept group-line 2 in Al- gorithm D However, when the process is conditioned on composition of labels from different groups a user specification the sampling procedure differs as c is iteratively built. For each la- tent axis cy. if label is specified for the concept group aligned with Cj. only the values along the j-th axis are taken of a sample from the corresponding 8-dimensional Gaussian. If no label is given, Cy is sampled from Standard Normal distribution (0, 1) (or the respective prior of choice.
12

---

Published as a conference paper at ICLR 2021
Same applies for latent dimensions which have not been aligned with any concept groups, since [c]
lines 2 to Algorithm
Algorithm 1: Trajectory generation conditioned on a single label
Input:
Visual Scene image I
Input:
Single user label from concept group g
Input: Image encoder, part of pe
Input: Decoder network 4e
Input: set of Gaussian distribution for each label in each concept group:
K (H J E f1......199 191....,11)):3 I E {1,....Igj])
Output: Generated trajectory of time length T
encode image input i <- PI(
sample latent values с ~ (c) Ej², such that j = g,
concatenate с and i: 4- [c: il:
tile h T times;
attach last time dimension h [h; t), (t1...... ......... ftt...,,,,,tt, 7
generate trajectory & (x(x;));
return x;
Algorithm 2: Trajectory generation conditioned on composition of labels
Input: Visual Scene image I
Input:
User label specification y (un,...,
Mg . potentially one label for each concept group
Input: Image encoder, part of po
Input: Decoder network 4s
Input: set of Gaussian distribution for each label in each concept group:
K = [C(c) Myy. E,)), j (I...., [G), E 1951]
Output: Generated trajectory x of time length
encode image input i < (T);
init I];
for each latent dimension E c do
if yJ # 0 and j <= [9] then sample w ~ ((w pyss E₁t), such that, 1 My: only take values along the j-th dimension: append wlj] to c;
else
sample w ~ ~N(w00, 1); append to c;
concatenate and i: h +- [c;
11 tile times;
12 attach last time dimension h = [b; t), t (4....,,77 -;
13 generate trajectory & < (x)x
return S;;
The deninnes heuristics used for reporting accuracy on the the generated trajectories have concrete semantics and values which have been chosen after closely inspecting the captured dataset, the given user labels and the robot configuration during the demonstrations.
spatial labels-left, right, front, behind- the location of the red object in the scene is extracted using standard computer vision techniques based color segmentation. The position of the cube inferred as the center of mass all red pixels detected. From the overall end-effector trajectory, derived through the robot forward kinematics, the pressing location is chosen as the end-effector position with the lowest coordinate. That position is then projected into the image plane in order determine where with respect to the detected object position did the robot touch the table.
soft vs hard trajectories the generated trajectories have normalised values in the range [-1,1]. A soft trajectory considered as such specific joint efforts are below partic- ular threshold. More specifically:
13

---

Published as a conference paper at ICLR 2021
shoulder 1ift joint efforts are below 0.5.
upper arm roll joint effort above -0.5.
short vs long trajectories the generated trajectories have length 7 = 240. A short trajectory considered as such if efforts in the range [max_effort-0..2, max_effort] are maintained for more than 50 time steps. Otherwise the trajectory is deemed long. The heuristic operates over the following joints:
shoulder lift joint efforts upper arm roll joint efforts
s slow vs fast trajectories slow trajectory is considered as such if the time it takes to reach maximal joint effort is more than 80 time steps. Otherwise the trajectory is deemed fast. The heuristic operates over the following joints:
shoulder lift joint efforts upper arm roll joint efforts
For resolving robot joint names, please consult Figure below:
shoulder panning joint
shoulder lift joint
upper arm roll joint
elbow flex joint
forearm roll joint
Figure 9: Color-coded robot arm joints and their corresponding names.
C ARCHITECTURE DETAILS
The model architectures are implemented in the PyTorch framework" and is visually presented in Figure 10 The image encoder network takes as input single RGB 128x128 pixel image. The trajectory encoder takes a single 14-dimensional, 240 time-step-long trajectory. Their outputs both feed into an MLP network which, after series of nonlinear transformations, outputs parameters of a distribution over the latent space c. Through the reparametrisation trick Kingma & Welling (013)) we sample values for Through residual connection, the output of the image encoder concatenated to the sampled latent values. The resultant vector is tiled 240 times, extended with time dimension, and fed into TCN decoder network to produce reconstructions for the original 14-dimensional input trajectories.
Across all experiments, training is performed for fixed number of 100 epochs using batch size of 8. The dimensionality of the latent space |c| = 8 across all experiments. The Adam opti- mizer (Kingma & Bal (014) used through the learning process with the following values for its cesumeters-(lerrnnggeee = 0.001, 31 0.9, 32 = 0.999, eps le 08, weightdecayrrae = amsgrad False).
14

---

Published as a conference paper at ICLR 2021
Image Encoder
FC (4)
FC (256)
2D Conv (k=3, p=l, c=
2D Conv (k=3, p=l, c=64)
2D Conv (k=3, p=l. c=64)
2D Conv (k=3, p=1, c=32)
2D Conv (k=3, p=l, c=32)
Input Image [128 128 3]
(a) Image Encoder
Trajectory Encoder
FC (32) T
FC (256)
ID Conv (k=7, p=3); ID Conv (k=5, p=2); ID Conv (k=3, p=1) [c=20]
Input Trajectory x [240 14]
(b) Trajectory Encoder
TCN Decoder
MLP
Temporal Block (dilation=4, k=5, 14)
FC (2x8) μ., log(o) FC (32)
Temporal Block (dilation=2, k=5, c=20)
Temporal Block (dilation=1, k=5, c=20)
FC (32) Concatenated [i; r ][ 36]
append time channel t
tile (240, 12)
Concatenated [i; c][1 x 12]
(c) MLP
(d) TCN Decoder
Table 5:
Network architectures used for the reported models. (a) is a 2D convolutional network, (b)
a ID convolutional network, (c) is a fully-connected MLP network, (d) a Temporal Convolution Network, made of stacked temporal blocks and dilated convolutions, described in Bai et al. (2018)
For all experiments, the values (unless when set to 0) for the three coefficients from Equation are:
a = 1, = 10
The values are chosen empirically in a manner such that the loss terms have similar magnitude and thus none of them overwhelms the gradient updates while training the full model.
D QUALITATIVE EVALUATION FOR DABBING
Figures to 15 present the qualitative results from perturbing in a controlled way the latent space optimised under the VAE-weak model. For figure the first of plots represents the generated joint angle positions for each of the 7 joints, for each of the 5 drawn samples, the second row-the generated joint efforts-and the third row the corresponding end-effector positions, from the forward robot kinematics, projected in the image plane. As we can see, perturbing co and C₁ corresponds clear and consistent movement of the end effector (through the generated joint position profiles) while there is not much change in the generated effort profiles- profiles-Fiiures [ and I2 Simultancously. we observe the opposite effect in other Figures. Perturbations in 22 correspond solely changes in the output efforts in the shoulder lift and upper arm rolls joints joints-Figure (i) and (j). Perturbations in C3 correspond to changes length for which the joint efforts are exerted but does not change their magnitude or where they are applied on the table-Figure [4] And finally, perturbations in C4 correspond to changes on how quickly the maximum efforts are reached most noticeable in the rolls joints Figure 15 (i) and (j). As result,
changing slope of the shoulder lift and upper we can conclude that the latent dimensions, which we aligned with the semantically grouped weak labels, have indeed captured the notions and concepts which underlie the labels.
15

---

Published as a conference paper at ICLR 2021
Image!
Trajectories X
Spec Y
[gi] front/back
[9,] softhard
cone alfort
[9:] rightlett
[9.] slow/quick
[9.] long/short
2D Convolutions
Convolutions
0000
residual connection
[6 9J
O OO
[co-
Giem
Temporal Convolutions
[G 9cl
py(y|c)
per-axis classification
11-4p11 correspandence and label
joPt affort
Sma
Reconstructions
Figure 10: Visual schematic of the overall model. Multiple different modalities are encoded to common latent space which is used both for reconstructing trajectories and predicting weak labels y, coming from the demonstrator.
03-33
(50 Wouu pen offen
[I Bistesler afoet
Upper ame mi affe
122
25 eleen fic offert
23 Seeere ml otore
10d wrtt fice offert
Cd Mit rol affert
20
Figure II: Linearly interpolate co (= from dabbing the left to dabbing to the right of the visual landmark in the scene.)
16

---

Published as a conference paper at ICLR 2021
shauster pan a4ga4
241 chwokder ECI sngle
AOUAT
diam Angle
Namkire A4g
III angle
1e1
221
0120
614
[m33
01:12
Figure 12: Linearly interpolate ej (= from dab to the front to dab to the back of the visual landmark in the scene.)
atoutdee litt
flxc
1pl wratrro
013
322
330
522
812
Mhaziie pan aften
Shockder atlad
dagggr are rsl altuk
Millkon effect
Senarr v1ou
1e0 MeC flas effet
Cd aflart
332
Figure 13: Linearly interpolate 02 (= from dab hardly y softly)
E MODEL SENSITIVITY ANALYSIS TO DIFFERENT DATA MODALITIES
As an exploration step towards analysing to what extend the auxiliary loss terms guide the model to pay attention to the different data modalities it is exposed to, train the VAE and VAE-weak mod- els on the demonstrations for pouring in the red cup and blue cup. The first latent dimension of the VAE-weak model is optimised to represent the conceptual variation of pouring in of the two cups. Post-training, the values of the latent vectors с for both models are fixed and trajectory samples drawn the models are conditioned four different test scenes, representing different cup configurations. Figure T6 demonstrates that, averaged across the different configurations, the VAE-weat model has greater variations in the sampled trajectories for all joints of the robot arm when compared to the VAE one. This suggests that unless utilised, the input image modality might be partly ignored the VAE does not have any auxiliary tasks to foree it use the scene image input, which can result in it just auto-encoding the trajectory data through In contrast, predicting where the robot pours the scene an abstract level is both function of the joint angle trajectory and the image inpus--utilise through the classification loss of VAB-weak. Thus, for the same specification (fixed c), conditioning on different scenes leads to different joint trajectories.
17

---

Published as a conference paper at ICLR 2021
aiouider pan stgla
(1 Diatovide IPE angle
Salppparrnn ralanga
dbow Tww.wTTA--11 aogl
Soresie 092
II wiit Plo ongle
11 alaratrolaaggl
0W00
432
02
10
Figure 14: Linearly interpolate 63 (= from dabbing for a long period of time to dabbing for short period of time)
Uewsiber ofterd
Shettaee etird
uagger ml efts
109 Silekeen eftert
Semore aloe
1nd Med fire aftert
a0t effert
Figure 15: Linearly interpolate c4 (= from dabbing quickly dabbing s slowly)
VAE-weak
VAE
Normallsed Joint angle variation
shoudder. Jen
Headeer. lR
afbrr Roi
foream rel
wet Rox
wrte rall
uperr am,
Joint names
Figure 16: Four test cup configurations (left) and mean normalised joint angle variation for sampled trajectories conditioned on them (right).
18


Reviews:
[{"review_id": "lVIfTjz82a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary* \nUnder the context of learning from demonstrations, the paper studies the problem of leaning interpretable low dimensional representations from high dimensional multimodal inputs using weak supervision. Paper argues that since robots and humans have different levels of abstractions and mechanisms, observation+action spaces between them are greatly misaligned which complicates learning by directly observing humans. However, the underlying concepts essential for tasks lie in a much lower-dimensional manifold. Learning this manifold effectively and in an interpretable way, especially using weak supervision, can significantly change how robots can acquire skills from demonstrations and generalize them to new unseen scenarios. Towards this end, the paper proposes to learn probabilistic generative models capturing high-level notions from demonstrations using variational inference. The strength of the paper is in demonstrating that conditional latent variable models can learn disentangled low dimensional represented using weak supervision; which authors effectively demonstrated using real-world experiments. My main reservations are in terms of the technical novelty of the paper and the narrow scope of experimental evaluation.\n\n*Suggestions*\n1. Use of variational inference for organizing high dimensional multi-modal inputs to low dimensional useful representations is already a well-established idea. However, the idea of learning \"interpretable representations\" using \"weak supervision\" is interesting and probably the most significant bit in this work. This however warrants appropriate experimental section probing styles/ strengths/ quality/ modality of supervision used. The current experimental section starts strong on the former (generating behaviors using concepts) but overlooks the latter, arguably the more interesting questions. \n2. Authors motivate the paper using the idea that learning interpretable low dimensional concepts called \"common sense\" will enable generalization and adaptation. Experimental section fails to investigate and establish these claims.\n3. Lastly, the experimental setup and the problem formulation around dabbing is very cleverly defined to communicate the ideas. This however comes across as too narrow. Additionally, the label groups for this problem cleanly separates into interpretable disjoint concepts. It's unclear if such interpretability and clear separation (hence the possibility of weak supervision) is true in general for common real-world problems. Paper makes no attempt to establish the generality of the approach. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BkAZ4Ywf4p", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a way to learn from demonstrations with weak or no labels. The premise behind this paper is that even when humans provide labels during a demonstration, those labels often do not fully describe the data (e.g., the human may say \"soft\" when \"fast\" would also apply). This paper presents a technique that uses latent variables to model the uncertainty over a group of class labels that could describe the task (e.g., slow, soft, left-of-object). The variables are modeled such that the observation is conditionally independent of the human provided labels given the latent variables. This allows the human provided labels to be decoupled (or disentangled as the paper calls it) from the observations. By doing so, it is possible to have only partial labels (weak labels). This model was applied to a task where a human would teleoperate a robot arm and apply a dabbing motion in relation to an object in the scene. The operator would provide only one of several possible applicable labels for each demonstration. The results show that the models using the weak labeling out-performed models with no labeling.\n\nThis paper proposes and interesting and novel way to handle weak labels from human demonstrators. By separating them, not only can they handle weak labels, but also multiple non-conflicting labels, or even no labels. This is an interesting contribution. The paper does a good job of comparing the methodology with 3 different off-the-shelf models for implementing the inference networks (GS, AAE, and VAE), as well as comparing each with and without the weak labels (baseline). There are several instances in the results where the weak label models far outperform the baseline models without weak labels (e.g., Table 3 VAE for \"soft\").\n\nHowever, there are many instances where the baseline models outperform the weak-label models, although not as dramatically. Nonetheless, the results in Tables 2 and 3 seem inconsistent. While the weak-label models are usually better, there are several cases where the baseline outperforms. Furthermore the extremeness of some of the results are a bit concerning. It would be nice to see some further exploration of this. In particular, the choice of task and label groups could be significantly correlated with the performance of the models. Further experiments should be done with other, more standard tasks and potentially user-provided labels to better determine the performance of the weak-label models as compared to the baselines.\n\nOverall, though, this paper does present a novel solution to the problem of user-provided labels. They're often not fully descriptive, which can leave some models confused (e.g., a demonstration can be \"soft\" but be labelled \"fast\"). The proposed solution attempts to solve this while also being probabilistically complete. While this certainly necessitates further experimental evidence to prove the usefulness of the model, the idea itself and preliminary evidence provided here are sufficient for publication.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8YW2gLw8Jc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== SUMMARY ==\n\nThis is a well-written paper that discusses how to learn disentangled representations for the learning from demonstrations (LfD) task in robotics. It is shown that using weak-supervision on top of unsupervised learning frameworks (that use the variational autoencoder for instance) can work well in this case. These disentangled factors of variation in the data are shown to correspond well to the 'abstract concepts' of the human demonstrations. This is shown in the example of the PR2 robot dabbing demonstrations, including visual data as well robot trajectories. \n\n== QUALITY & CLARITY ==\n\nThe paper is written very well, in fact most papers contain a lot of spelling mistakes but this paper was a joy to read in this regard :) The concepts are also explained clearly. However I would have expected better and more detailed coverage of related & past work.\n\n== ORIGINALITY & SIGNIFICANCE ==\n\nUnfortunately, I think the paper suffers from lack of originality, at least with respect to ML. From a robotics point of view, I would have accepted it as a very good application paper, if the authors had also presented real-robot results that show the learned model in action (generating actual trajectories for the robot). The authors however show only the prediction performance for (held-out) test demonstrations.\n\n== VERDICT ==\n\nI would like to thank the authors for a very well written paper. Overall, I think the paper needs some improvements such that it can be accepted in a revised version or most likely, in another conference. The paper needs to present real robot results and cover related work in more detail. Comparing the method to related work (DMP, or ProMP-variants, or any other competitive method) in the real-robot experiments would also be crucial. \n\n=== NOTES & SOME MINOR COMMENTS ===\n\n* No need to mention the link in the abstract\n\n* \"For example, the concept of pressing softly against a surface\nmanifests itselfin a data stream associated with the 7 DoF real-valued\nspace of joint efforts, spread across multipletime steps.\" -> For the PR2 robot? Either remove 7 DoF or add for which robot.\n\n* \"However, the essence of what differentiates one type of soft press from another nearby concept can be summarised conceptually using a lower dimensional abstract space\" -> Nearby concept sounds vague, please be more explicit or give\nexamples.\n\n* In Figure 2, the image encoding 'i' does not affect y. Why?\n\n* Related Work: ProMPs are not represented as dynamical systems\n\n* \"To fully close the loop, the trajectories which we sample from the\nmodel could further be executed on the physical robot through a hybrid\nposition/force controller (Reibert,1981). However, such evaluation is\nbeyond the scope of the paper.\" -> I don't think so. The real proof of concept is the actual robot\nexperiments!  Without real robot experiments to show how the generated\nconditioned trajectories actually perform, in my opinion the paper is\nan application paper without any significant ML contributions. The\npaper as of now I fear only confirms the fact already acknowledged in\nprevious ML papers, that disentanglement can be achieved through\n(semi)supervised learning [see Locatello et al. and papers citing\nthis work)\n\n* equivallent -> equivalent\n\n* force-relate -> force-related\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of learning interpretable, low-dimensional representations from high-dimensional multimodal input via weak supervision in a learning from demonstration (LfD) context. To mitigate the disparity between the abstractions that humans reason over and the robot's low-level action and observation spaces, the paper argues for learning a low-dimensional embedding that captures the underlying concepts. The primary contribution of the paper is the ability to learn disentangled low-dimensional representations that are interpretable from weak supervision using conditional latent variable models.

The paper was reviewed by three knowledgeable referees, who read the author response and discussed the paper. The paper considers a challenging problem in learning from demonstration, namely dealing with the disparity that exists between the ways in which humans and robots model and observe the world, a problem that is exacerbated when reasoning over high-dimensional multimodal observations. As the reviewers note, the use of variational inference to learn low-dimensional interpretable representations from weak supervision is compelling. The primary concerns are that the contributions need to be more clearly scoped and that the experimental evaluation is a bit narrow. The authors make an effort to resolve some of these issues, in part through the inclusion of an additional experiment that considers pouring tasks. However, the extent to which this second task mitigates concerns about the narrow evaluation is not fully clear. The paper would be strengthened by the inclusion of experiments in a less contrived setting (and one for which the concepts are not necessarily disjoint) as well as a clearer discussion of the primary contributions.
Decision: Accept (Poster)