Weakly-Supervised Trajectory Segmentation for Learning Reusable Skills
Authors: Parsa Mahmoudieh
Trevor Darrell
Deepak Pathak
Venue:  (2020)
Keywords: skills
demonstration
agent
sub-task
primitives
robot learning
manipulation

Abstract: Learning useful and reusable skill, or sub-task primitives, is a long-standing problem in sensorimotor control. This is challenging because it's hard to define what constitutes a useful skill. Instead of direct manual supervision which is tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks. We propose a weakly-supervised approach for trajectory segmentation following the classic work on multiple instance learning. Our approach is end-to-end trainable, works directly from high-dimensional input (e.g., images) and only requires the knowledge of what skill primitives are present at training, without any need of segmentation or ordering of primitives. We evaluate our approach via rigorous experimentation across four environments ranging from simulation to real world robots, procedurally generated to human collected demonstrations and discrete to continuous action space. Finally, we leverage the generated skill segmentation to demonstrate preliminary evidence of zero-shot transfer to new combinations of skills. Result videos at https://sites.google.com/view/trajectory-segmentation/

Under review as a conference paper at ICLR 2020
WEAKLY-SUPERVISE TRAJECTORY SEGMENTATION FOR LEARNING REUSABLE SKILLS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning useful and reusable skill, sub-task primitives, is long-standing prob- lem in sensorimotor control. This is challenging because it's hard to define what constitutes a useful skill. Instead of direct manual supervision which tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks. We propose a approach for trajectory segmentation following the classic work on multiple instance learning. Our approach is end-to-end trainable, works directly from high-dimensional input (e.g.. images) and only requires the knowledge of what skill primitives are present at training, without any need of segmentation or or- dering of primitives. We evaluate approach via rigorous experimentation across four environments ranging from simulation to real world robots, procedurally gen- erated to human collected demonstrations and discrete continuous action space. Finally, we leverage the generated skill segmentation to demonstrate preliminary ev- idence of zero-shot transfer new combinations of skills, Result videos ht tps: segmentation
//sites google. com wwwm//vee/traaeeeeecooeoeeeeeeee
INTRODUCTION
Humans have an uncanny ability to generalize from one task to another using either few, or at times, no new examples. This wouldn't be possible if they were to learn each new task from scratch. Humans rather extract reusable skills from already learned tasks and compose them to generalize to new tasks seamlessly. However, learning such repeatable skills has been long standing challenge in sensorimotor control, partly because it is hard to define what constitutes usefuu skill in itself.
One way to layout the scope skill is either by designing a corresponding reward function, or collecting expert demonstrations. For instance, consider a skill of reaching for object. One can easily learn a policy for this skill by either using reinforcement learning with 12 distance as reward (Sutton & Barto, 1998), by imitation learning from kinesthetic demonstrations (Argall et al., 2009; Hussein et al., 2017). However, neither of these approaches provide a natural form of supervision because the way this skill is performed in isolation way it could be used as part of some end task. For instance, reaching for cup for pushing is very different from the way one would reach for cup pick it up for pouring. Therefore, defining skill directly in a supervised manner could casily lead to biased set of examples.
be drastically different from the
promising alternative is to learn skills that are already embedded in some useful end tasks. Previous works have explored this in the context of an agent's 'ss own exploration (Eysenbach et al., 2018; Nair et al., 2018; Pathak al., 2018), where, the agent learns goal conditioned skill policies using data collected during its exploration phase. These skills are then used to plan for novel tasks at inference. However, exploration itself an open research problem, and hence, such approaches have difficulty in scaling to complex skills.
In this work, we follow an alternative paradigm where we extract skills from a collection human demonstrations gathered to perform different end tasks. A straightforward way to extract reusable skills would be to get an expert to label each time-step of demonstrations with the corresponding skill label. However, this per-step segmentation supervision is tedious for the expert and too expensive to scale. Moreover, such labeling would be biased towards what expert thinks is the right segmentation than towards segmentation which helps learn the skills better. This leads the question: is it

---

Under review as a conference paper at ICLR 2020
WxHxT
Encoder
CxT
LSTM
Cx1
log
exp(.)
Trajectory Images
Segmentation Model
Timestep-level Prediction
Trajectory-level Prediction
Trajectory-level Tags
Figure 1: We propose a weakly-superrvised approach for segmenting demonstrations into skill primitives, aka, sub-tasks, Our approach is end-to-end trainable, works directly from sensory data (c.g., images) and only requires the knowledge of class of primitive sub-tasks performed during the demonstration accomplish the end task, without any need segmentation or ordering primitives. corresponding to each sub-task and accumulate them to generate trajectory-level predictions which are then trained match trajectory-level tags. Our segmentation model consists of an encoder followed by recurrent LSTM model to capture time series dependency. We use log-sum-cxp perform smooth accumulation of time-step predictions into trajectory-Iewwe predictions.
key idea to make per-lime step prediction
possible to use the expert knowledge to only know the types of skills present in demonstration and figure out the segmentation from the data itself?
Inspired by the classic work multiple instance learning (MIL) (Andrews et al., 2002), we propose weakly-supervisee approach for segmentation of human demonstration trajectories into primitive repeatable skills. Our approach assumes the access to only trajectory-leve labels, i.e, what primitive skills, aka sub-tasks. present in the demonstration. The key insight is to learn primitive skills classifier model conditioned on the input per-time step reasoning structure in this classifier. An overview of approach is shown in Figure 1. Our model generates per time-step primitive skill label prediction estimates which are then accumulated via differentiable function generate trajectory-level predictions. MIL, where only the most confident prediction across time-steps steps is trained, our full model is trained end-to-end using trajectory-level multi-class loss directly from raw sensory images.
sensory data (e.g. demonstration images), and incorporate
contrast to classic
However, we are training with only trajectory-level supervision, then why should our per-step predictions of segmentation model converge to meaningful skill primitives? Data comes to rescue! Since our model trains across variety of demonstrations, and hence, it would have seen plenty of demonstration trajectories that contain a certain skill primitive (positives) and plenty that do not (negatives). The classification loss would force the segmentation model to focus discriminative cues that are common across all positives, and absent from negatives. These discriminative cues corresponding each skill would encourage the per time-step predictions to gradually correspond to the correct ground truth time-step labels.
We evaluate approach in four different environments: (a) As proof of concept, start with sim- ple 2D navigation task in grid-world setup where the demonstrations are programatically generated. (b) We then discuss result in roboties setup with continuous control actions, in particular, Jaco robotic arm performing button pushing demonstrations in touch pad with procedurally generated demon- strations. (c) We our approach in a robotic setup with actual human demonstrations collected on the RoboSuite benchmark. (d) Finally, we evaluate on real robot dataset with actual human demonstrations collected kinesthetically. Across all these environments, our approach outperforms the other variants of MIL and achieves decent segmentation accuracy. We then show zero-shot generalization to tasks containing novel permutations of skills in Jaco environment.
METHOD: SEGMENTING DEMONSTRATIONS INTO SKILL PRIMITIVES
Given a collection human demonstration trajectories, our goal is to learn a labeling for skill primitives at each time-step of the sequence, i.c., per time-step skill segmentation. Let X be human demonstration trajectory denoted by X {ø 22, D3 ...... Tr where T is the length of the demonstration and x, denotes tpple of observation (which is raw sensory image in our case) at time
2

---

Under review as a conference paper at ICLR 2020
and action taken from it. Note that the action data is optional for the purpose of skill segmentation, but can be useful post segmentation for learning skill policies via imitation. Let Y = {1)1 U2 Va .....921 be the latent ground truth labeling of skill primitives in the sequence. Each label yr. belongs one of the k labels from a set of all skill classes C = (1..... k). i.e., Yt E C. These per time-step labels are not only tedious for expert annotate, but also difficult scale. In this work, we do not assume access to yt during training, and learn the per time-step segmentation in a weakly-supervised fashion by only using trajectory-level 1-bit label during training, i.e., whether skill class is present in the trajectory or not. After training, our model is able to directly segment demonstrations at inference, without requiring any labels of any kind. An overview of our method is shown in Figure 1.
..r.)
The marginal probability of skill primitive at each time-step of demonstration can be written as P(ve (It; #t- 21 (1)) where is the parameter vector the segmentation model represented by neural network in our case. If we had access to the true sequence labels Y, the network parameters 0 can be easily learned via maximum log- lieliikelioood by representing the probability as follows:
P(yelO, {Ie, Tt-1 ...1...,,21111 x11) = Zi
exp (f(ye; 0, f2,, I--1 .1-1.....11))
(1)
where Ze is the partition function at t, defined as Z, Exee exp (k; (k; 0, f #t-1 21 (1)). The output of the function ft corresponds to the logit score value generated by the neural network, In order model temporal dependency on across observation time-steps It, we represent f(.) via a recurrent neural network, in particular, LSTM (Hochreiter & Schmidhuber, 1997).
2.1 WEAKLY-SUPERVISE TRAJECTORY SEGMENTATION
We are given a dataset of demonstration trajectories during training. D = (X'... ........))) X"), where n is total number of demonstrations available for training. Each demonstration trajectory is weakly labelled with what skill primitives are contained within the trajectory. Neither do we have access which time-step densely correspond which skill primitive, nor the permutation in which the skills are executed in. Instead, we are only given a set of skill primitive labels Cx E C present in the demonstration trajectory X.
Although our supervision is only at trajectory-level, we do not directly predict output label- ing Cx from input demonstration X. Instead, we instill the structure of per-step prediction in our weakly supervised segmentation model by first computing the per-step classification score fyr; 0, {zte -t-1....... .21) and then accumulate it across all time-steps to compute the probability of a class in the whole trajectory. This
setup is captured by classical paradigm of multiple instance learning (MIL) (Andrews et al., 2002). At inference, we use this per time-step score to compute the probability of skill primitives at each time / as described in Equation (1). There are multiple ways one could accumulate these per time-step scores discussed as follows.
2.2 ACCUMULATION OF TIME-STER PREDICTIONS
Intuitively, we would like estimator that could generate an aggregated score for the class depending on how much each time-step votes for that class. We would ideally like the highest score value across time-steps to contribute most to the decision whether class label Yx is present in the trajectory X or not. One simple way to achieve that is employ element-wist max operator, which is also the de facto approach in MIL to accumulate element-level scores. However, this would amount to passing gradients only to the most confident time-step and will completely eradicate the role of other time-steps. This is especially problematic in case of sequential trajectories because no skill primitive will be of only time-step long. Hence, instead of max, we use a soft approximation to which can take into account the contribution of all time-steps steps. In particular, use boog-sm-exxp operator. Given the the logit score f(y;; 0, {x,, ...,.....aa x1 at each time-step, the trajectory-level logit score g for class E C is computed follows:
T
g(c,0, X) = log Exx exp (S(r c; (11., x11)))
We perform this operation for all c E C and use softmax over g(c; 9, to compute trajectory-Ievel probability distribution Q(c]X, 0). Finally, the parameters 0 are optimized to maximize Q(c(X,0)
3

---

Under review as a conference paper at ICLR 2020
(a) 2D Navigation
(b) Dial Env
(c) RoboSuite Env
(d) MIME Env
Figure 2: We evaluate across four environments with different properties: (a) Discrete 2D Navigation as proof of concept. (b) Dial continuous control: Jaco robotic arm based manipulation with procedurally generated demonstrations. (c) RoboSuite environment: Sawyer robot in simulation with human-collected demonstrations. (d) MIME environment: Baxter robot in real world with human demonstrations. Result videos at ht https: //sites. //ssttessooggl.......... com/ wiew/tt wwwtrrraa crajectory segmentatinn atmeentation..
for each class with respect to the ground truth trajectory level tags Cx. Note that this optimization is fully differentiable through Equation 2, and hence can be optimized via stochastic gradient descent in the end-to-end fashion. Pinheiro & Collobert (2015) has also showed the effectiveness a temperature- based variant of log-sum-exp operation for semantic segmentation in images.
However, these per-step scores f would be almost uniformly random in the beginning of the training process due to the absence of per-step supervision. Since are training with only trajectory- level supervision, why should these per-step predictions should ever converge to meaningful skill segmentation? It turns to be the case because we learning across large variety of demonstration examples. Hence, for each skill primitive we would have seen plenty of positive as well as negative trajectories. The loss suppresses the negative classes and encourages the positive ones, hence our segmentation model would be forced to focus on discriminative cues that are exclusively common among the trajectories containing the skill primitives and not common to the cues that help distinguish other skills. Since our trajectory-leve segmentation is based on a deterministic transformation of per-step predictions, each per-step score will then be forced to focus on those discriminative cues. The discriminative nature encourages the per time-step predictions to slowly drift towards the true latent ground truth segmentation which are not available directly.
IMPLEMENTATION DETAILS, SETUP AND BASELINES
Training and Evaluation Details Our segmentation model consists of convolutional neural network encoder of each image of trajectory and a one layer fully connected encoder of the action to a 32 dimensional feature that concatenated to the image feature before being fed into an LSTM with 100 hidden units. We train with batch size of 64 for the Dial, RoboSuite, and MIME environments and a batch size 128 for the Grid-world environment. All models were trained with Adam with learning rate of le-4. For training. we use 50000, 2000, 1000, and 1600 trajectories for 2D Navigation, Dial, RoboSuite, and MIME Environments respectively. We evaluate our method on the training set, validation set that consists of the same number of skill primitives (sub-tasks) per trajectory in training, and a test set that consists more skill primitives per trajectory than seen training. The segmentation quality is measured by classification accuracy of the learned model averaged across all time-step. The time-step ground truth is only used for evaluation and not for training.
Baselines We compare our approach different formulations of weakly-supervised classification proposed earlier. None of these methods have been applied to a temporal trajectory segmentation before. In our work, we adapt them for sub-task segmentation. In particular, we compare to: (a) MIL (Andrews et al., 2002): In this bascline, we train the deep segmentation model using Classic MIL. formulation as proposed by Andrews al. (2002). We penalize the most confident time-step in the output corresponding to each sub-class to correctly predict the trajectory-level classes.
(b) FCN-MIL (Pathak et al., 2015b): This approach is an adaptation MIL for deep-networks. The MIL objective is treated as the loss function for training the segmentation network, but instead of training the most confident output, we train a neighborhood time-step, where k is hyper- parammrreeterr chosen
k-elements near the most confident the validation set. We found k=3, i.e., extra time-step on each side of argmax, to work the best across all datasets.
(c) CCNN (Pathak et al., 2015a): This is alternative approach to tackle the weakly
4

---

Under review as a conference paper at ICLR 2020
setup. Instead of training the most confident output, pseudo ground-truth per time-step truth is generated in a way that ensures that it is the closest pseudo ground-truth possible which contains only the classes specified by the vajecttoyyleeer tags. One could additionally put threshold constraints on the lower-bound of time-steps devoted to the present classes in the pseudo ground truth. The segmentation is then pushed this per time-step pseudo labels, and new pseudo ground-truth is generated after each iteration. This approach provides per time-step supervision using only trajectory level-tags. The CCNN approach was originally applied for per-pixel segmentation of images in computer vision literature (Pathak et al., 2015a), and we adapt it for temporal trajectories. (d) Random: For sanity check, we randomly pick sub-task class at every time-step with uniform probability.
(e) Random-CCh Clo:: This a random baseline with privileged class information even at test time. In this case, we only sample random uniformly from the set of classes that are already present in the trajectory. This is not fair comparison but provides an estimate for the difficulty of the problem. Note that we use the same network architecture and training details across all bascline implementa- tions. However, we tune each baseline separately in a thorough manner on validation sets. This is crucial to ensure an aaplles-o--aplle comparison.
4 RESULTS
We evaluate our approach and other baselines on four different datasets with very diverse charac- teristics. The 2D Grid-world environment has discrete action space, while the Dial, RoboSuite, and MIME environments have a continuous action space. The Grid-world and Dial environments have demonstrations collected procedurally by hand-designed controllers, while the RoboSuite and MIME environments have demonstrations collected by humans. Human demonstration in RoboSuite are collected via teleoperation and kinesthetically in MIME. Grid world, Dial and Robosuite are in simulation while MIME is from real robot. A snapshot of these environments is shown in Figure 2.
4.1 PROOF OF CONCEPT: 2D NAVIGATION IN DISCRETE TOY GRID-WORLD
In the grid-world environment, the action space consists of moving up. moving down, moving left, moving right, and picking up object it is hovering over. There are 5 different types of objects uniquely identified by their color and an end task would consist of picking up some subset of all the objects in a particular order. The primitives are defined picking up specific type of object. We train our segmentation model on trajectories with 2-4 skill prim- itives and test with 5 skill trajectories. Each instantiation of the environment has a different starting position of the agent, different starting position of the objects, and differ- ent set of objects needed to be grabbed. The image inputs used are 33 by 30 resolution color images, and the max trajectory lengths are 50. This environment serves as toy scenario for proof of concept. At the bottom-left of the image, there is an indicator which suggests which skill being executed. Hence, an efficient approach should achieve 100% accuracy, as is the case with our method as shown in Table 1.
Method
Train
Val
Test
Random
20.00
20.00
20.00
Random- Cll
36.00
36.00
20.00
CCNN
65.27
64.63
60.24
FCN-MIL
91.28
91.88
91.58
MIL.
90.78
91.08
91.08
Ours
100.00
100.00
100.00
Table 1: This cuvironment serves as only proof of concept. Since the environment is simple 2D grid with skill primitive indica- tor cell the bottom, an efficient approach should he able to achieve almost full accu- racy. Our method is able to perfectly segment the data into ground truths, while other base- lines can not despite easily separable skills.
4.2 DIAL CONTROL. ENVIRONMENT: JACO ROBOTIC ARM BASED MANIPULATION
In the Dial environment, proposed in (Shiarlis et al., 2018), there is torque-controlled JACO 6 DoF arm and a dial pad that the arm can interact with which is simulated in MuJoCo. There are naturally 10 different types of primitives available in this environment corresponding to pressing numbers zero through nine. We train our segmentation model on trajectories with two to four sub-tasks and test with five sub-task trajectories. Each instantiation of the environment has different sequence of numbers it expects to be dialed in the correct order. The image inputs used are 112 by 1 112 resolution gray-scale images, and the max trajectory lengths are 100.
5

---

Under review as a conference paper at ICLR 2020
Salt-
Sah S11--bo
899
(a) Dial Env: Jaco arm
(b) RoboSuite Env: Sawyer arm
S-ab- Tall-Takkk
Siib Toek
(c) MIME Env: Baxter
Figure 3: Figure shows qualitative visualization of the skills (sub-sampled) discovered by our approach on held-out test set. Dial Env: learned primitives using Jaco manipulation shown. Skill # N corresponds the arm dialing number N touch pad. (b) RoboSuite Env: Three predicted primitives shown where the primitives are to pick and place an object into corresponding bin (top to bottom: cercal, milk, box) (c) MIME Env: Four discovered primitives are shown starting from top left clockwise: reach to object, wipe, stir inside object, and pour out object.
Our method performs similar FCN-MIL on the train/val, and better on test. It significantly outperforms the other baselines (Table 2). However, show that segmentation for zero-shot execution performance as explained in Section 4.5. Learning perfect segmentation in the Dial environment is very challenging because there is little signal in most of the trajectory for each skill to signify exactly which digit will be pressed until the arm reaches proximity of the digit.
more useful
4.3 ROBOSUITE ENVIRONMENT: SAWYER ROBOTIC ARM BASED OBJECT PICK AND PLACE
The RoboSuite environment (Mandlekar et al., 2018) has Sawyer DoF and four different objects (bread, can, cereal, and milk) that the arm can interact with simulated with MuloCo physics engine. There are four different types of primitives available in this environment corresponding to pick and place of bread, can, cereal, and milk to correct corresponding bin. We train for trajectories with two three skill primitives and test on trajectories with four skill primitives. Each instantiation of the environment has different sequence and set of objects that need to be picked up and placed into their corresponding bins. The image inputs used are 128 by 128 resolution color images and max trajectory lengths are 100.
Our method significantly outperforms baselines in full trajectory segmentation by a significant margin. Only MIL performs above random present class on validation and test datasets. Learning perfect segmentation signal in most of the trajectories of each subtask signify exactly which object will be picked up until near the end of the primitive where the object has been picked up. The "pick object" portion of human demonstrations is usually much longer than the "place object" part because with the tele-operation setup the human stumbles a little bit until fully gripping the object. After the object is in the gripper, placing object in bin is quick reach the correct bin for the object.
the RoboSuite environment is also very challenging because there very little
4.4
MIME ENVIRONMENT: BAXTER ROBOTIC ARM BASED MANIPULATION
dataset that contains 8260 human-robot video demonstrations of 20 different robotic tasks (Sharma et al., 2018). We defined following primitives for subset
MIME is a

---

Under review as a conference paper at ICLR 2020
Dial Jaco Manipualtion
RoboSuite Manipulation
Method
Train
Val
Test
Train
Val
Test
Random
10,00
10.0
10.0
25.00
25.00
25.00
Random-
36.00
36,00
20.00
33.00
33.00
25.00
CCNN
9.05
9.58
10.70
31.92
22.24
21.80
FCN-MIL
64.42
59.64
57.81
29.79
23.76
22.21
MIL
19.02
15.89
14.00
35.95
28.81
28.69
Ours
61.57
59.52
59.93
44.96
37.67
33.88
Table 2: We show the segmentation performance across different methods on train, validation, and test datasets for Dial Jaco and RoboSuite Object Manipulation. (a) Dial Jaco: we train our segmentation model on trajectories with three to four subtasks and test with five subtask trajectories. Our approach beats all baselines by significant margin except for FCN-MII which performs similar to outperformed on the test set. (b) Robosuite: we train our segmentation model on trajectories with three subtasks and test with four subtask trajectories. Our approach outperforms all baselines by significant margin.
training and validation, but
of this dataset: reach for object, pour out object, stir inside object, stack objects, place object in box, wipe with rag (6 primitives). All videos have two primitíves where one is to reach for object and the other is the action do with on the object. There is held out test dataset for each robotic task which we use for evaluation. The image inputs used are 120 by 320 resolution grayscale images and the max trajectory lengths are 100. Our method beats all other baselines in full trajectory segmentation by least 1.8x on the test set (Table 3).
Method
Test Accuracy
CCNN
11.17
FCN-MIL
19.10
MIL
24.37
Ours
44.22
Table 3: Quantitative results on MIME dataset. We test our segmentation model held out test trajectories seen during training.
4.5 ZERO-SHOT RESULTS: JACO MANIPULATION
We use our segmentation model to create sub-datasets for each of our primitives to train behavior cloned skill policy for each. We then test our skill policies on performing higher sequence length tasks not seen in training data. During the creation of the sub-datasets. we rejected all segments smaller than 5 consecutive timesteps of the same labelled primitive. We applied gaussian smoothing on the segmentation prior extraction filter out noisy predictions.
We demonstrate the zero-shot capability of our model and baselines on the Dial control environment in Table 4. Our model performs at least 1.25x better than all baselines. We also show that although FCN-MIL had the same segmentation accuracy our method, after rejecting smaller than 5 timestep segments our method has a significantly higher post rejection segmentation accuracy. We speculate this is due to model committing less wrong prediction than the basclines. Therefore wrong predictions are more casily rejected with our segmentation model.
5 RELATED WORK
Multiple Instance Learning is the sub-field that deals with learning with weak-labels (Dietterich et al., 1997), formulated mostly as max-mare max-margin. Classic formulations include MI-SVM (Andrews et al., 2002) or LSVM (Felzenszwalb et al., 2010). There have been boosting (Ali & Saenko, 2014; Zhang et al., 2005) and Noisy-OR models (Heckerman, 2013) formulations as well. This has been extensively explored image segmentation (Pinheiro & Collobert, 2015; Pathak et al., 2015a). There is also work learning segment vidcos into primitive components that take place in each of the videos using human narration of the videos (Alayrac et al., 2018), (Naim et al., 2014), (Zhukov al., 2019), (Richard et al., 2017). However collecting human narrations is expensive process that cannot scale to very large datasets. In our setup, we need to label each timestep a video with a further preferable constraint of having the labels contiguous as possible.
Training goal conditioned policies have become very popular recently since they learn policies that are reusable for new tasks. The goals be defined in state space (Schaul al., 2015), (Andrychowicz

---

Under review as a conference paper at ICLR 2020
Method
Seg Acc. without Reject
Seg Ace with Reject
Zeroshot Success Rate
CCNN
9.58
28.08
FCN-MIL
59.64
68.58
38.8
MIL
15.89
50
39.6
Ours
59.52
79.01
50.4
Table 4: Quantitative Zero-Shot results on Dial Jaco Manipulation: This table shows the comparison of performance across ditferent methods. The first two columns reports the segmentation accuracy on validation set without and with minimum of 5 time-step segment rejection. The last column shows the zero-shot subtask success rate on 5 subtask tasks. Our method has significantly better post time-step segment rejection segmentation accuracy and zero-shot subtask performance.
et al., 2017) or in image space (Pathak et al., 2018), (Nair et al., 2018). These methods however suffer from not being able reach goals that are out of the training distribution. They also tend to greedily reach goals which make them suffer for long horizon tasks. This leads to having to break the problem down to checkpoint states and this can still be expensive and cumbersome for a human to define each new task (Pathak et al., 2018). Similar way to define new task is one-shot imitation learning where a new task is defined with one demonstration of the task; this can also be expensive one need to learn new tasks that are cumbersome for the human to collect (Duan et al., 2017).
Hicrarchical policies have been used to solve long horizon tasks by having a meta meta-police policy or manager that takes in states sparser time scale and chooses which sub-policy or short term goal for worker to achieve at dense time scale (Dayan & Hinton, 1993), (Vezhnevets et al., 2017), (Sutton et al., 1999), (Bacon al., 2017). However, the subpolicies learned in this type of options framework are not intrepretable in their specialization and therefore tend to be hard to reuse for new tasks.
Learning reusable and interpretable primitives from scratch has been very challenging. One way to tackle this is for humans to define the high level primitives we care about beforehand. We can then decompose complex tasks into subtasks primitives and learn the primitives first (Oh et al., 2017). By learning the primitives first, an agent can then perform any new task that can be decomposed into those primitives without needing any new human demonstration data (Andreas al., 2017). The human only needs to specify the order in which the primitives need to be done. In the presence of only the order of high level tags of what primitives were performed at the trajectory level, the optimization is non-trivial. Shiarlis et al. (2018) provide one approach using dynamic programming to efficiently compute gradients (Graves et al., 2006) through all possible trajectories high level permutation of primitives performed in each trajectory. We however have presented an approach where we would not need the order in which the primitives were performed thereby making the labeling process of videos much cheaper. (Kipf et al., 2019) and (Niekum et al., 2012) do unsupervised primitive segmentation in environments using state space whereas we do weakly supervised primitive segmentation in image space in all of our environments ((Kipf et al., 2019) only use image input for their toy gridworld environment).
given
6 DISCUSSION
Obtaining primitives from existing experience and composing them perform novel tasks presents a viable paradigm for tackling generalization to novel tasks. Due to the combinatorial nature of composition, this approach allows generalization exponentially many scenarios with a linear number of primitives. This work provides an end-to-end trainable formulation for extracting primi- tives. Our results demonstrate strong segmentation accuracy, and show early evidence for zero-shot generalization via compositionality. Our primitive segmentations, obtained from demonstrations of end tasks, are more naturally extracted 'options' than those defined by an expert, which may be prone to being biased. Another alternative, besides zero-shot composition, is to treat these primitives as new atomic actions to develop a hierarchical framework for modeling long-horizon tasks. Deciding the threshold for segmenting actions into different levels of controller is one of the main bottlenecks in hierarchical control, and current approaches usually resort to using domain knowledge to solve the issue. These extracted primitives, a.k.a 'macro' actions, provide an alternative scaffolding which could bootstrap the hierarchy in a bottom-up manner. To promote these follow-up ideas, we will publicly release our code and environments.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Learning from narrated instruction videos. IEEE transactions on pattern analysis and machine intelligence, 40(9):2194-2208, 2018.
K. Ali and K. Saenko. Confidence-rated multiple instance boosting for object detection. In CVPR, 2014. 7
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning- Soarning-Volume 70, pp. 166-175. JMLR. org, 2017.
Stuart Andrews, loannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple- instance learning. In NIPS, 2002. 2, 3, 7.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NIPS, 2017. 7
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 2009. 1
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017. 8
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. In Advances in neural information processing systems, pp. 271-278, 1993. 8
Thomas G Dietterich, Richard H Lathrop. and Tomás Lozano-Pérez, Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 1997.
Yan Duan, Marcin Andrychowiez, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech 7aremba. One-shot imitation learning. In NIPS, 2017. 8
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint, 2018.
Pedro F Felzenszwalb, Ross Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained past- models. IEEE Tran. PAMI, 2010.
Alex Graves, Santiago Fernández. Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference Machine learning, pp. 369-376. ACM, 2006. 8
David Heckerman. A tractable inference algorithm for diagnosing multiple diseases. arXiv preprint 1rXiv:1304.133 1511 2013.
Sepp Hochreiter and Jtirgen Schmidhuber. Long short-term memory. Neural computation, 1997. 3
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 2017. 1
Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro
Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. International Conference on Machine Learning, pp. 1418-3428, 2019.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, 2018. 6
Iftekhar Naim, Young Chol Song, Qiguang Liu, Henry Kautz, Jiebo Luo. and Daniel Gildea. Unsu- pervised alignment of natural language instructions with video segments. In Twenty-Eige Eighth AAAI Conference Artificial Intelligence, 2014.7 7

---

Under review as a conference paper at ICLR 2020
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In NeurlPS, 2018. 1,
Scott Niekum, Sarah Osentoski, George Konidaris, and Andrew G Barto. Learning and generalization of complex tasks from unstructured demonstrations. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5239-5246. IEEE, 2012.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Searning-Volume Volume 70, pp. 2661-2670. JMLR. org, 2017. 8
Deepak Pathak, Philipp Krähenbühl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. In ICCV, 2015a. 4,
Deepak Pathak, Evan Shelhamer, Jonathan Long. and Trevor Darrell. Fully convolutional multi-class multiple instance learning. In ICLR, 2015b. 4
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation. In ICLRR 2018. 1, 8
Pedro O Pinheiro and Ronan Collobert. Weakly supervised semantic segmentation with convolutional networks. In CVPR, 2015. 4,
Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with rnn based fine-to-coarse modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 754-763, 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In ICML, 2015.7 7
Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. arXiv preprint arXiv: /2/, 2018.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. arXiv preprint arXhv:1803.01840 2018. 5, 8
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning, Artificial intelligence, 112(1-2):18122) 1999.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3540-3549, JMLR. org, 2017.5
Cha Zhang, John C Platt, and Paul A Viola. Multiple instance boosting for object detection. In NIPS, 2005. 7
Dimitri Zhukov, Jean-Baptise Alayrac Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. arXiv preprint 1rXiv:1903.08225, 2019. 7
10


Reviews:
[{"review_id": "ryx5EZ5htB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of learning to label individual timesteps of sequential data, when given labels only for the sequence as a whole. The authors take an approach derived from the multiple-instance learning (MIL) literature that involves pooling the per-timestep predictions into a sequence-level prediction and to learn the per-timestep predictions without having explicit labels. They evaluate several pooling techniques and conclude that the log-sum-exp pooling approach is superior. The learned segmentations are used to train policies for multiple control skills, and these are used to solve hierarchical control tasks where the correct skill sequence is known.\n\nThis is a good application of the MIL approach. However, I have settled on a weak reject because in my view, the novelty and results are minor.\n\nThe main point of comparison is the log-sum-exp() pooling as compared to max() and neighborhood-max() pooling. However, if I understand correctly, the log-sum-exp() approach has been used successfully in several other domains including its original domain of semantic image segmentation. So I view the novelty of the approach to be fairly low.\n\nIn addition, although the superior pooling method (which already exists in the literature) does outperform the alternatives evaluated here, the results are somewhat underwhelming, at only ~35-60% validation accuracy. How does this compare to a fully-supervised oracle method trained with per-timestep labels?\n\nThe behavioral cloning results are also fairly underwhelming, and the experiments are not very clearly described. Am I correct in my understanding that the learned skills are composed to solve a task where the correct sequence of skills is known, but is longer than the training sequences? A success rate of 50% on this task seems rather low. How does this compare, as above, to a fully-supervised oracle baseline? Why is there no success rate reported for the CCNN baseline?\n\nI think this is a good application of weakly-supervised MIL, but I find the specific contributions to be lacking in novelty and impressiveness of results. There are several directions that I think could improve the work:\n- oracle fully-supervised results, to indicate the gap between the fully- and weakly-supervised case\n- more thorough baselines on the behavior task, such as Policy Sketches [1]\n- perhaps the temporal aspect of the problem could be incorporated into the pooling approach more directly to produce a more novel algorithmic contribution\n\n[1] Andreas, Jacob, Dan Klein, and Sergey Levine. \"Modular multitask reinforcement learning with policy sketches.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1lG65CZKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a weakly supervised learning approach for trajectory segmentation, which relies on coarse labelling about the occurrence of a skill in a demonstration to segment trajectories. This is accomplished through a recurrent model predicting skill categories for each step in a trajectory, and a trajectory level loss function that penalises the probability of seeing a given skill in a trajectory.\n\nOverall, I like the idea of identifying skills in this manner, and think that this is an important problem to address. However, I have concerns about it's feasibility when a large number of skills are present. It seems that there are certain requirements of skill occurrences in datasets that need to be met if this approach is to be feasible. For example, consider the dataset of skill sequences:\n\n[111 222 333]\n[444 222 333]\n[222 333 555]\n\nIt seems that it will never be possible to learn to identify skills 2 and 3 from this dataset. This paper would be greatly strengthened if the minimum dataset requirements to learn all skills were enumerated, or some theoretical bounds provided around when this loose labelling could possibly be successful provided.  The paper glosses over this point by suggesting that \"data\" makes this a non-issue, but the paper would be much stronger if these limitations were confronted and some bounds on the chances of meeting the requirements needed for learning provided. \n\nThe classification results seem extremely poor, and it is hard to assess these on the basis of accuracy alone. For example, in the Robosuit example, the test results could potentially have been obtained by simply making the same prediction throughout the test, and there is no indication that anything sensible was actually learned. Confusion matrices, or precision and recall metrics, are required to avoid this. At present it is impossible to know if these errors are due to noisy predictions, the dataset limitations described above, or simply a poor classifier.\n\nAlong these lines, qualitative results in the form of trajectory classification (say following the presentation conventions in \n\nBolanos, 15, https://arxiv.org/pdf/1505.01130.pdf \nRanchod, 15, https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7353414&tag=1\n\nwould also help to address these concerns. \n\nA question regarding the dial jaco videos experiments, why does the classifier not predict a 5 when moving from 4 to 6? I would expect a very jumpy prediction here, but the prediction looks very smooth - is this a filtered result?\n\nFinally, baseline experiments are limited to other weakly supervised learning segmentation approaches,  but I think comparisons with unsupervised clustering methods would also be useful.\n\nUnfortunately, due to the lack of evidence that the proposed approach is able to learn effectively, I am inclined to reject this paper. Clarifying the bounds, and presenting stronger evidence (beyond accuracy) would make this paper stronger.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJxlyIDAYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a weakly supervised method for segmentation of trajectories into sub-skills inspired by multi-instance learning (MIL) in image classification by Andrews et al. (2002). This is done via training a classifier to label each observation per time-step with the probability of skills corresponding to that observation. These predictions are then accumulated throughout the trajectory to compute the probability of the skill in that trajectory. There is only a trajectory level supervision provided which specifies which skills are present with no specification of the order in which they appear. They empirically show that their model can achieve decent skill level classification scores on multiple environments provided that there is a large variety of demonstrations provided.\n\nIn its current form, I would recommend this paper to be rejected because 1) the framing and motivation of the paper does not correspond to the results and experiments reported and hence seems misleading 2) the paper is limited in scope 3) further experiments and comparisons to relevant baselines are needed to support the claims made in the paper.\n\nThe problem that they are proposing is interesting and is of clear value, however, the paper falls short in addressing this problem. In particular, the paper is framed as a way to learn re-usable useful sub-skills that can help generalize to new situations in control. However, the method presented provides a per time-step labelling of each observation with the associated most likely sub-skill. Having the per time-step labelling of the trajectory provides no indication that the data could be useful for learning reusable skills for downstream tasks later. One very basic experiment could be to train a behaviour cloning (BC) agent on the observation-action pairs conditioned on the sub-skill (or a separate network per sub-skill) and show that the learned policies can be leveraged in solving the tasks presented. For instance, one can train a meta-controller that can switch between these learned sub-skills to successfully perform the task. Training such sub-skills from weakly supervised skill annotations has been successfully done by Shiarlis et al. (2018). It should be noted that in their setting, the annotations are ordered which simplifies the problem to some extent.\n\nIgnoring the motivation and focusing only on the problem setting addressed which is annotation of trajectories with skill labels, the experiments seem very restrictive to me. To my understanding there are at most 4-6 primitive skills present in each environment investigated. Looking at the videos linked, it feels like there is some overfitting to the trajectories provided as for example in the case of the video with the red bowl (Reach and Stir inside Cup), the classifier is predicting 'Stirring' from the first frames where there is not much information present in the scene and 'reach to object' could also be plausible for example. It would have been nice to see the logits of predictions per classes for these examples to understand the confidence of the model particularly when some of the skills could be equally likely (e.g. first few frames).\n\nI found the experimental setup unclear. Particularly, details regarding the task setup, how many demonstrations are needed per task/skill, architectural choices and hyper-parameters are lacking and makes experiments hard to follow and understand. I would have also liked to see more analysis regarding the segmented trajectories, particularly how consistent these predictions are through time. To my understanding there is nothing that keeps a skill annotation consistent over some period of time (e.g. the model could keep switching its prediction every time-step). This would be quite unsatisfactory if one would like to use this to actually segment sub-trajectories associated with a given skill that could be useful for training policies. They report applying Gaussian smoothing to filter out noisy predictions but there are no details provided on how this is tuned and how much this affects the quality of segmentations.\n\nOverall, the paper seems to me very limited in its scope and experimental results. The claims made throughout the paper are not supported empirically or theoretically. There is not enough evidence for me to assess the significance of the proposed method and know whether this is indeed useful in practice. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a multiple instance learning-based approach that uses weak supervison (of which skills appear in any given trajectory)  to automatically segment a set of skills from demonstrations.  The reviewers had significant concerns about the significance and performance of the method, as well as the metrics used for analysis.  Most notably, neither the original paper nor the rebuttal provided a sufficient justification or fix for the lack of analysis beyond accuracy scores (as opposed to confusion matrices, precision/recall, etc), which leaves the contribution and claims of the paper unclear.  Thus, I recommend rejection at this time.
Decision: Reject