Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning
Authors: Sachit Kuhar
Shuo Cheng
Shivang Chopra
Matthew Bronars
Danfei Xu
Venue: CoRL 2023 Poster (2023)
Keywords: Imitation Learning
Preference Learning
Manipulation

Abstract: Practical Imitation Learning (IL) systems rely on large human demonstration datasets for successful policy learning. However, challenges lie in maintaining the quality of collected data and addressing the suboptimal nature of some demonstrations, which can compromise the overall dataset quality and hence the learning outcome. Furthermore, the intrinsic heterogeneity in human behavior can produce equally successful but disparate demonstrations, further exacerbating the challenge of discerning demonstration quality. To address these challenges, this paper introduces Learning to Discern (L2D), an offline imitation learning framework for learning from demonstrations with diverse quality and style. Given a small batch of demonstrations with sparse quality labels, we learn a latent representation for temporally embedded trajectory segments. Preference learning in this latent space trains a quality evaluator that generalizes to new demonstrators exhibiting different styles. Empirically, we show that L2D can effectively assess and learn from varying demonstrations, thereby leading to improved policy performance across a range of tasks in both simulations and on a physical robot.

Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning
Sachit Kuhar, Shuo
Shuo Cheng, Shivang Chopra, Matthew Bronars, Danfei Xu
Georgia Institute of Technology
(kuhar, shuocheng, shivangchopra11, mbronars, danfei Sanfei)@gated @gatech. edu
Abstract: Practical Imitation Learning (IL) systems rely on large human demon- stration datasets for successful policy learning. However, challenges lie in main- taining the quality of collected data and addressing the suboptimal nature of some demonstrations, which can compromise the overall dataset quality and hence the learning outcome. Furthermore, the intrinsic heterogeneity in human behavior can produce equally successful but disparate demonstrations, further exacerbating the challenge of discerning demonstration quality. To address these challenges, this paper introduces Learning to Discern (L2D), offline imitation learning frame- work for learning from demonstrations with diverse quality and style. Given small batch of demonstrations with sparse quality labels, we learn latent rep- resentation for temporally embedded trajectory segments. Preference learning in this latent space trains a quality evaluator that generalizes to new demonstrators exhibiting different styles. Empirically, we show that L2D can effectively assess and learn from varying demonstrations, thereby leading to improved policy per- formance across range tasks in both simulations and on physical robot.
Keywords: Imitation Learning, Preference Learning, Manipulation
Introduction
Imitation Learning (IL) allows robots to learn complex manipulation skills from offline demonstra- tion datasets [1, 2, 3, 4, 5]. However, the quality of the demonstrations used for will significantly influence the effectiveness of the learned policies [2, 6, 7]. Practical imitation learning systems must amass demonstrations from broad spectrum of human demonstrators [8, 9], ranging from novices experts in given task [10. I1]. Moreover, even experts may complete task differently, resulting in disparate demonstrations of equal quality [12, 13. 14]. Therefore, effective IL algorithm must learn from demonstrations of varying quality, each further diversified by the unique skillset of the individual demonstrators.
However, mainstream II. algorithms often make the simplifying assumption that all demonstrations are uniformly ideal [11]. As result, policies trained with these algorithms may unknowingly learn contradictory supervisions [11, 14, 15]. Recent work attempts estimate expertise by developing unsupervised algorithms to quantify action distribution and variance [6] or actively soliciting new demonstrations that match with known expert policy [16]. While these methods are general in principle, two critical limitations remain. First, estimating demonstrator expertise without any external quality label is an inherently ill-posed problem. As noted above, demonstrated behaviors can vary drastically across demonstrators and even across different trials from the same demonstrator. Neither action variance nor similarity to an expert sufficient to capture expertise in such settings. Second, these works only consider state-level features, while behaviors are better represented in temporal sequences.
from suboptimal
"work done while at Georgia Institute of Technology.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Preforence Leaming
GMM
Learning
Figure 1: L2D: Our framework proceeds in three primary stages during training. First, we aug- ment trajectory segments with temporal embeddings and employ contrastive learning to map these segments to latent space. Next, use preference learning in this latent space to train quality critic on sparse preference labels. Finally, we train a Gaussian Mixture Model (GMM) on the critic's 's outputs where the different modes represent demonstrator quality.
To this end, present Learning Discern (L2D), completely offline method for learning from heterogeneous demonstrations. 1,2D can efficiently estimate expertise from limited demonstration quality labels based on scquence-level latent features. By incorporating temporal embeddings into trajectory segments, we are able to learn a more meaningful latent representation for long-horizon manipulation tasks. Preference learning in this latent space trains a quality evaluator that generalizes across diverse styles of task completion while only requiring a subset of coarsely ranked data. We demonstrate that our method can not only discern the quality of in-domain data but also identify high-quality demonstrations from entirely training. We show this empirically in both simulated and real- world robot settings. With I.2D, it becomes feasible to filter high-quality demonstrations from the vast and diverse datasets that are needed for data-driven IL.
demonstrators with modes of expertise unseen in
2 Related Work
Imitation Learning with Suboptimal Data. Imitation Learning (IL) [17, 18, 19, 20, 21, 22] from sub-optimal data is an active area of research in robotics. Various works show that near-optimal policies can be trained if demonstrations are ranked based on their quality [4, 11]. However, manual annotation of dense reward labels is costly and unscalable. CAIL. [23] extrapolates demonstration quality from a subset of labeled data D', but still requires a dense ranking of D'. Other works estimate demonstrator quality in an unsupervised manner (ILEED [6]) or compare demonstrations expert policy (ELICIT [16]). Such unsupervised algorithms do generalize equal quality demonstrations from different modes of task completion. This motivates the need for an IL algorithm that learns generalizable quality estimator from a subset of coarsely ranked data.
Preference Learning from Human Demonstrations. Preference learning is a form supervised learning that learns rankings from pairwise comparisons [24, 25, 26, 27, 28, 29]. When applied human demonstration data, preference learning can effectively train reward function based on rankings of trajectory segments [30]. Various frameworks utilize this reward function train poli- cies through inverse reinforcement learning [31, 32]. To limit manual annotations, some methods synthetically generate lower-quality data through noise injection [7]. These methods are hampered by the fact that noise level does not directly translate to preference ranking of trajectories [33]. Our method, L.2D, ranks offline demonstrations using preference learning in a latent space. This allows us to generalize across diverse styles and modes of task completion with sparse preference labels.
Representation Learning for Data Retrieval. Representation learning, especially with focus robotics, has been widely studied. This area typically explores learning from visual inputs [34], data

---

MAPPING TO LEARNED LATENT SPACE
CRITIQUING
QUALITY
Figure 2: Filtering Unseen Demonstrations: When faced with unseen demonstrations, L.D parti- tions the trajectory into segments and augments each with its chronological ordering in the sequence. The segments are mapped to the latent space learned during training and ranked by the quality critic. After calculating the mean and variance of ranks in full trajectory, the trained GMM is employed to predict a preference label for the unseen demonstration.
augmentation [35], goal-aware prediction [36], and domain-specifee information [37]. Recent work such as that by Du et al. [38] enhances the performance of robot manipulation tasks by first mapping offline data latent space using contrastive learning and then retrieving offline data using that latent space with task-specific data as queries. Yet, these works do not account for the suboptimality of given demonstrations in task-specific data and offline data.
3 Problem Setting
We study the imitation learning problem in environments modeled as Markov Decision Processes (MDP), where S and A denote the state space and the action space, respectively. P: x A x S [0, 1] is the state transition function. The environment emits binary reward upon task completion. The learning algorithm does not have access to this signal.
We assume access small set of demonstration Dknowa b where a trajectory тi is quence of transitions with variable length T.y; given by (8o, 0o 81, 01,.......................... 8 -1, OL,, oL,,-)) Each trajectory in Denown is associated with a quality label E L. where L is the set of possible labels indicating the quality of trajectory. We divide Dxnown into sub-datasets i.e., (DA, DR, De....} of similar quality levels, based on the quality labels associated with each trajectory. If dataset A is considered be of superior quality compared to B, we denote it as > B. Similarly, we assume TA > TB, provided тλ E and TB B.
In this paper, we develop framework for imitation learning that achieves state-of-the-art perfor- mance on sub-optimal data by estimating the quality of new demonstrations. More specifically, our aim to learn a representation that captures the multimodality from a large dataset of unknown quality Dunknown and can critique the quality of new demonstrations within the same context. Im- portantly, our approach operates in an offline learning setting where we do not have access to the environment, reward signals, or any successful completion of trajectories [6, 1]. This constraint further underscores the need for our method's ability to discern and evaluate the quality of demon- strations without direct interaction or feedback from the environment.
4 Learning to Discern
Imitation Learning (IL) with suboptimal data often leads to inferior task performance. Evaluating the quality of a demonstrator without supervision is challenging, and manually assigning quality to demonstrations is impractical. The core component of our method a preference network Q that learns to evaluate the quality of a demonstration 7 by estimating its ranking label 1. As mentioned above, the key challenges of learning generalizable demonstration quality estimator are that (I) it is difficult to capture behavior-level features by assessing individual state-action pairs, especially for long-horizon manipulation tasks and (2) the demonstrated behaviors are diverse even among

---

demonstrators of similar expertise. In this section, we describe each challenge in more detail and introduce the components in our method 1.20 that address the corresponding challenges.
Extracting Behavior Feature through Temporal Contrastive Learning
As we mentioned earlier, estimating the quality of demonstration by merely observing variance in states is poorly-posed problem. Instead, we propose learning temporal latent space for quality critique. We utilize a neural network encoder E 141x0ts.Dim + D that maps trajectory segments {ar = {5t { f==1 for this learning using triplet margin loss [38] poses a challenge: naive sampling based on quality labels may degrade representation quality. Specifically, long-horizon tasks like Square [1 I] contain bottleneck states indifferent to demonstration quality, leading to task-specific regions that don't 't ma- jorly impact overall quality. We use these sampled segments create triplets of anchor trajectory segment a. a positive trajectory segment op. and negative trajectory segment on which are used in different contrastive learning strategies:
1
(i=num_egmmmns into a d-dimensional latent space. Sampling triplet pairs
Strategy Arbitrary Sampling from Specific Quality Sets: ent trajectories from demonstration set A. We then sample the negative trajectory segment oN from different demonstration quality set B.
We sample oa and op from differ-
Strategy 2: Specific Segment Sampling from Arbitrary Quality Sets: We leverage domain- specific knowledge that certain segments, like initial or final segments, do not influence demon- stration quality. For instance, in the case of a square task with complete demonstrations, we know that all demonstrations conclude with the robot placing the square the cylinder. Thus, we sam- ple the final segments finat from 71 and finull from 72. The negative trajectory segment on is sampled such that it is not from the same region but can be sampled from any subset as the segmented regions themselves do not impact quality.
Position Encoding for Capturing Non-cyclic Behavior
Preference learning is effective for evaluating the quality of unknown and diverse demonstrations, but we find it struggles with long-horizon manipulation tasks. We posit that this is due the intrin- sic non-cyclical nature of these tasks can lead to identical state-action pairs having different quality labels depending on the trajectory context. To navigate this inherent complexity, we introduce data augmentations like position encoding for each demonstration state with respect to the entire trajec- tory. We find that this provides the necessary context to the latent space for comprehensive task understanding. Let our original observation at any time-step be denoted as o, E Robe.dim We introduce position encoding, represented as real number pr indicating the normalized time-step (i.e.. t/T, where T is the total number of time-steps in the demonstration). We then formulate the augmented observation of which becomes a vector in Robs don+1 defined as of [o,,Pe].
Training Quality Critic
After training the encoder E learn latent space that is cognizant of quality-sensitive regions of the demonstrations, our approach leverages Quality Critic network Q, defined function Q RR + R. that serves as regressor, mapping the d-dimensional trajectory segment embedding to continuous scalar value indicative of the quality of the demonstration. In the training phase, we utilize pairs of trajectory segments, σ and d, randomly sampled from datasets DA and DB, respectively. Each segment first converted into an embedding and subsequently passed through Q to obtain quality score. The pairwise ranking loss [39, 30, 31], capitalizes on these pairs to compare and learn the relative quality differences between the segments. During the inference stage, the model computes the quality score of individual segments.
Respresenting Suboptimality as Gaussian Mixture Model
Demonstrations of varying quality may exhibit mixtures of behaviors different regions. By being receptive to individual demonstration regions, our approach allows to estimate the quality of these mixed-qualis demonstrations, the overall quality of a demonstration can be viewed as a majority vote of its constituent parts. This concept can be naturally represented by Gaussian distribution,

---

where the mean of the Gaussian represents the average quality of segments in a demonstration. After learning the latent space encoder E and quality critic Q, we train a Gaussian Mixture Model (GMM) G. We sample a sufficient number of segments ( f. x k)) from each demonstration in Known and pass each segment through encoder and quality critic Q to obtain a sealar value. These values represent the quality of different regions within the demonstration. We then map these scalar values into sets that correspond demonstrations of different qualities (i.e., DA, DB, De, etc) and train the GMM where different Gaussians correspond to different qualities.
Estimating Quality of Unseen Demonstrations
Given a new unseen demonstration T, generate of scalar values using (((((smplerr(7) with each value estimating the quality of specific region within the demonstration. We then esti- mate the probability of each value with respect the trained GMM and assign each scalar to the set with the maximum probability. Finally, we use a heuristic to determine whether the demonstration should be used for training the policy. As an example, in our robomimic experiments, for an unseen demonstration, we count the percentage of segments assigned to the 'good' quality set. Demonstra- tions are then rated based on this count. We subsequently rank all demonstrations based on this score and select the top demonstrations for training the IL algorithm BC-RNN. This procedure ensures the inclusion of demonstrations that predominantly exhibit desirable behavior.
5 Experimental Evaluation
We empirically validate whether L2D can learn effective representations that enable accurate dis- crimination of high-quality demonstrations from the pool of available demonstrations, which vary in quality. We show that distinct components of the multi-component architecture of L2D signifi- cantly contribute to its effectiveness with an ablation study. We demonstrate that 1.2D can discern high-quality demonstrations from both seen and unseen demonstrators and compare against base- lines. Finally, we collect human demonstrations for real and simulated manipulation tasks to test the method's applicability in more realistic scenarios.
Experiment setup. We use the Robomimic [11] for all experi- ments, which include a range of tasks like Lifting, Square Nut As- sembly, and Can Pick-and-Place collected from multiple human op- erators of varying skill levels. We specifically focus the Square Nut Assembly task where Robomimic provides us with 300 demon- strations, with 50 from each of the 6 operators. These 300 demon- strations are categorized into 3 sets 100 demonstrations each and are labeled as 'good', 'ok', or 'bad' based on the execution quality. Using higher quality demonstrations for training leads to a higher success rate in robomimic tasks [11]. To evaluate the different fil- RNN-based behavioral cloning (BC-RNN)
Figure 3: Good (green) and Bad (red) demonstrations for the Robomimic's Square task.
tering methods, to train policy on selected demonstrations and report the success rate (SR) (see appendix D for details).
We bifurcate the identification of high-quality demonstrations into two scenarios: (A) Familiar demonstrators. The demonstrations in each set originate from the same demonstrators, i.c., both Dknown and Dunknown have demonstrations each for varying quality from the same demonstrators. (B) Unseen demonstrators. The demonstrations in each set originate from different demonstrators, i.c., both Dxnown and Dunknown have demonstrations each for varying quality, but they are provided exclusively from different demonstrators. We adapt the robomimic dataset for these scenarios as fol- lows: In both cases, we divide the 300 available demonstrations into two sets of Dxnowa and D Dunknown- In the first case, we perform this division uniformly such that all 6 operators provide an equal num- ber of demonstrations into either set. However, the second case, the demonstrations in each set originate from different demonstrators, i.e., both Denowa and D Dunknowa have 50 demonstrations each for varying quality, but they are provided exclusively from different demonstrators.
5

---

Baselines. We benchmark the performance of our proposed method, L2D, against two state-of-the art methods for learning from mixed-quality demonstrations. ILEED [6] employs an unsupervised expertise estimation approach to identify good demonstrations. ELICIT [16] actively filters new demonstrations based on whether the state-action pairs match the prediction of an expert policy. We also compare against adaptations of preference learning and contrastive learning methods. Prefer- ence Learning baseline
best effort adaptation of TREX [31] reward learning method and uses pairwise ranking loss [30, 40J. Contrastive Learning uses triplet margin loss samples from distinct quality sets [38] for training and performs filtering by using cosine-similarity between encodings of trajectory segments. As additional baselines, we compare against naive uniform sampling strategy from the dataset of unknown quality demonstrations, as well as an oracle baseline representing the ideal scenario where the highest-quality demonstrations can be perfectly identified.
5.1 Component Ablation
In our ablation study, we dissect L2D to evaluate the individual contribution of each component. By comparing the Wasserstein distances between the distributions corresponding scores generated by Q, for sub-dataset with specific labels within Dunknown- we quantify the quality of the representa- tion learned. A higher Wasserstein distance indicates enhanced separability between trajectories of different qualities, making it easier to identify high-quality trajectories. Robomimic tasks, better quality demonstrations result in a higher SR when training policy using BC methods [11]. There- fore, higher total Wasserstein distances indicate more effective learned representations for filtering.
Table 1 and Figure 4 show our findings and the corresponding distribution visualizations, reveal that L.21) consistently outperforms the preference learning approach in unseen demonstrator experiment setup 5.2. The histogram visualizations plot the distributions of trajectory segment scores output by Q for each quality subset in Dunknown- Adding the S1 Contrastive component and positional encoding enhances the quality of the learned latent space, confirming the necessity of contextual understand- ing for long-horizon non-cyclic robot manipulation tasks. The final addition of the S2 Contrastive component underlines the benefits of using well-crafted positive and negative pairs for contrastive learning. These pairs, even from demonstrations of differing quality. lead to better representations, highlighting the importance of understanding that some regions of demonstrations do not influence the quality. This lends support to the idea of leveraging commonalities across all demonstrations to regularize our learning process, thereby avoiding overfitting specific demonstrator biases.
Method
Good vs. Okay f Good vs. Bad?
Okay vs. Bad
Total Distance
Preference
0.130
0.143
0.147
0.420
S1 Contrastive
0.337
0.213
0.414
0.964
Position Encoding
0.535
0.649
0.589
1.773
+ S2 Contrastive
0.604
0.848
0.417
1.869
Table Wasserstein Distance Comparison for the Robomimic Square Task: The table showcases capability of different methods, including the incremental introduction of contrastive components, to discern among demonstrations of quality levels (good, okay, bad) from unseen demonstrators.
Pretibiera on Unieroom usity onteesccentyy
Pradictiro Unbroem uaig
Tote
Can
Fow
Opra
Figure 4: Histogram Distribution Visualization: Comparing demonstration quality scores (good, okay, bad) for the Square task from unseen demonstrators. The left histogram represents conven- tional preference learning approach, while the right highlights the efficacy of our method, L2D

---

5.2 Main Results
L2D can identify unseen high-quality demonstrations from familiar demonstrators, We trained L2D the known demonstrations Danwwn and tested its ability filter unseen demonstrations Dunknown from the same demonstrators. Our results, as presented in Table 2, indicate that I.21) successfully identifies 43 out of the top 50 high-quality demonstrations from the pool. We find that our method outperforms ILEED [6], reinforeing the idea that using preferences over an unsuper- vised approach leads better representations. Finally, we observe that our method achieves an oracle-level success rate (SR) when training the policy on filtered data.
Method
Success Rate
Method
Good
Okay
Bad
Naive
0.44
Naive
68
16
16
ILEED 0.54
Our Method
93
4
3
Our Method
0.66
Oracle
100
Oracle
0.66
Table 2: Identifying Unseen High-Qualit Demonstrations from Familiar Demonstrators: Comparison of methods to identify high-quality demonstrations in an unseen dataset. The right table presents the quality categorization of selected demonstrations with the best 50 demonstrations from D Dknwwn while the left displays the imitation learning policy's 's success rate when trained on a combined set of best-known and selected demonstrations.
L2D can identify high-quality demonstrations from unseen demonstrators. In a practical situ- ation, policy may be trained on data provided by previously unseen demonstrators. This is chal- lenging because each demonstrator may possess unique style of execution and a diverse range of demonstration quality. We evaluated L2D in this setting by training it and the baselines on Dknown and then using them to filter the top 50 demonstrations from Dunknown- Table 3 shows that our proposed method I.2D significantly outperforms both contrastive learning and preference learning methods, which in turn outperforms naive sampling. In comparison with the ELICIT method [16], which learns only from high-quality data, L2D demonstrates superior performance, suggesting that more adaptable to variation in demonstration quality and demonstrator style. Notably, the success rate achieved by our method is only marginally inferior to that of the oracle, further supporting our claim that L2D can effectively discern high-quality demonstrations from unfamiliar demonstrators.
Method
Success Rate
Method
Good
Okay
Bad
Naive
0,20
Naive
18/50
16/50
16/50
Contrastive
0.36
Contrastive
23/50
20/50
7/50
Preference
0.38
Preference
27/50
20/50
3/50
Elicit 0.38
Elicit
23/50
20/50
7/50
Our Method
0.44
Our Method
39/50
8/50
3/50
Oracle
0.46
Oracle
50/50
0/50
0/50
Table 3: Identifying Unseen High-Quality Demonstrations from New Demonstrators: A com- parison of methods to discern top demonstrations from an unfamiliar source. The left table displays the imitation learning policy's s success rates, while the right categorizes the quality of demonstra- tions selected. L2D's performance approaches that of the oracle, highlighting its effectiveness in recognizing quality across unfamiliar demonstrator styles
5.3 Learning from
Demonstrations
We evaluate the ability of L2D to estimate the quality of demonstrations in realistic scenarios, com- pared to controlled simulated environments. We gathered new human demonstrations for tasks such as Square Nut Assembly simulated environments while Lifting and Stacking were demonstrated in real-world environments using actual robots, all performed by a diverse group of operators. These operators performed independently, without leveraging knowledge from others' demonstrations. We then trained our method on subset of these demonstrations and used it to filter the unseen dataset
7

---

0.25
87 0
U
Figure 5: Demo Quality Estimation. We show the predicted quality for real-world demonstrations of the Stack task. The trajectory segments with less optimal behaviors (e.g., jittering or waving) will be assigned with lower scores (marked with red bounding boxes).
for high-quality demonstrations. Please refer to Sec. in the Appendix for more training details. The effectiveness of our method evaluated based on the improvement in success rate after the integration of these newly identified demonstrations. Our experimental results, depicted in Table 4, suggest that L2D substantially improves the performance of imitation learning policy in realistic scenarios. In Figure we visualize the quality prediction for two stacking demonstrations and show that L,D can correctly identify high-quality demonstrations based on sparse labels.
Environment
Task
Method
High-Quality Demo Selection (%)
SR (%)
Simulated
Square
Naive Sampling
37
10
Simulated
Square
Our Method
95
32
Real
Stack
Naive Sampling
33
50
Real
Stack
Our Method
97
80
Real
Lift
Naive Sampling
33
30
Real
Lift
Our Method
45
53
Table Performance comparison of naive sampling and our method in terms of high- demon- stration selection from an unknown dataset and SR increase for two tasks (Can Pick and Place and Lift) on a real robot and one task (Square) simulation.
Limitations
Our method filters out entire demonstrations based on their assessed quality, ensuring that only high-quality data used for analysis. However, this approach may overly simplistic, assuming that low-quality demonstration contains no high-quality segments. In reality, demonstrations often encompass diverse tasks and behaviors, and a low-quality demonstration may contain sections method assumes clear distinction between
that could provide useful training data. Additionally, high and low-quality demonstrations, but the line separating the two is often blurred, depending on the complexity of the task and the specificity of the demonstrator's style. Addressing these limitations could lead to even better-learned representations.
7 Conclusion
We presented Learning to Discern (L.2D), novel method for efficiently estimating expertise from heterogeneous demonstration data in IL without access to the environment and reward signals. L2D employs sequence-level latent features and temporal embeddings to learn robust latent represen- tation of complex tasks. With a quality evaluator trained this latent space, our method can gen- eralize across varied task completion styles and identify high-quality demonstrations, even from new, unseen modes of expertise. This study demonstrates the potential of L2D filter high-quality demonstrations from vast and diverse datasets, marking significant advancement in the field of data-centric robotics.

---

References
[I] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint urXiv:2005.01643 2020.
[2] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. arXiv preprint (tXXix2218.002990 2021.
[3] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rys: What essential for offline via supervised learning? arXiv preprint arXiv:21 12. 10751, 2021.
[4] A. Kumar, J. Hong, A. Singh, and S. Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618, 2022.
[5] D. Ghosh, A. Ajay, P. Agrawal, and S. Levine. Offline rl policies should be trained to be adaptive. International Conference on Machine Learning, pages 7513-7530. PMLR, 2022.
[6] M. Beliaev, A. Shih, S. Ermon, D. Sadigh, and R. Pedarsani. Imitation learning by estimating expertise demonstrators. In International Conference on Machine Learning, pages 1732- 1748. PMLR, 2022.
[7] D. S. Brown, W. Goo, and S. Niekum. Better-than-demoonsttttr imitation learning automatically-ranked demonstrations. Proceedings of the 3rd Conference on Robot Learning, 2019.
[8] A. Brohan, N. Brown, Carbajal. Y. Chebotar, J. Dabis, C. Finn, Gopalakrishnan, K. Haus- man, A. Herzog, Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068 17, 2022.
[9] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Ay- tar, D. Budden, M. Vecerik, al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv: 12000, 2019.
[10] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, F. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
[11] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[12] A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations, Robotics: Science and Systems, 2020.
[13] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 0122955-22966 2022.
[14] C. Chi, S. Feng. Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.041337 2023.
[15] Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158 168. PMLR, 2022.
[16] K. Gandhi, S. Karamcheti, M. Liao, and D. Sadigh. Eliciting compatible demonstrations for multi-human imitation learning. In K. Liu, Kulic, and Ichnowski, editors, Proceedings of The 6th Conference Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1981-1991. PMLR, 14-18 Dec 2023. URL https::/ /tpp:///////cceings..omm mlr. press/v205/ 112205/anndi23... html.

---

[17] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-4833 2009.
[18] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation.
Neural computation, 3(1):88-95 -7, 1991.
[19] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceed- ings, 2011.
[20] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[21] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[22] Y. Ding, C. Florensa, Abbeel, and M. Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
[23] S. Zhang, Z. Cao, D. Sadigh, and Y. Sui. Confidence-aware imitation learning from demon- strations with varying optimality. Advances in Neural Information Processing Systems, 34: 12340-12350, 2021.
[24] T. Joachins. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133- 142, 2002.
[25] Fürnkranz and E. Hüllermeier. Pairwise preference learning and ranking. In Machine Learning: ECML 2003: 14th European Conference Machine Learning, Cavtat-Dubrovnik, Croatia, September 22-26, 2003. Proceedings 14, pages 145-156. Springer, 2003.
[26] W. Chu and Z. Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning, pages 137-144, 2005.
[27] N. Houlsby, F. Huszár, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classifi- cation and preference learning. arXiv preprint 1eXXi::1112.5744 2011.
[28] K. Lee, I.. Smith, and P. Abbeel. Pebble: Feedhack-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.0509), 2021.
[29] C. Kim, J. Park, J. Shin, H. Lee, P. Abbeel, and K. Lee. Preference transformer: Modeling human preferences using transformers for arXiv preprint arXiv:2303.00955, 2023.
[30] P. F. Christiano, J. Leike, T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[31] D. S. Brown, W. Goo, P. Nagarajan, and S. Nickum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 783-792, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http:h//pocceediing..e////// mlr. press/ html.
[32] C. Wirth, R. Akrour, G. Neumann, J. Fürnkranz et al. A survey of preference-based reinforce- ment learning methods. Journal of Machine Learning Research, 18(136):1-46 2017.
10

---

[33] L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self- supervised reward regression. In Conference on robot learning, pages 1262-1277. PMLR, 2021.
[34] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2203. 2022.
[35] M. Laskin, K. Lee, A. Stooke, Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884-19895, 2020.
[36] Nair, S. Savarese, and Finn. Goal-aware prediction: Learning to model what matters, In International Conference on Machine Learning, pages 7207-7219, PMLR, 2020.
[37] R. Jonschkowski and O. Brock. Learning state representations with robotic priors.
Autonomous Robots, 39:407-428, 2015.
[38] M. Du, S. Nair, D. Sadigh, and C. Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv preprint arXiv:2304.08742 2023.
[39] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[40] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations atari. Advances neural information processing systems, 31, 2018.
[41] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv:1412.69980, 2014.
arXiv preprint

---

A Additional Ablation Studies
To further investigate the effect of different architectural components and design choices in our pro- posed method, L2D, we focus specifically on the roles of S2 contrastive learning and data augmen- tation. The experimental setup is the same as described in Section 5: identification of high-quality demonstrations from unseen demonstrator experiment the Square task. We LSTM+Self At- tention architecture with CLS token for contrastive learning followed by simple MLP encoder for preference learning. The baseline for this experiment does not use S2 contrastive learning or data augmentation. We study the following design choices of L2D: (1) S2. Initial::: Initial trajectory segment similarity; (2) Final: Final trajectory segment similarity; and (3) Data augmentations: Decation-sensitvve time warping for segments used S2 contrastive. As done in Section 5.1, we sys- tematically examine the increase in separability of demonstration quality from new demonstrators by measuring the Wasserstein distances between trajectories of different qualities under different configurations. Higher distance indicates better separability and quality of learned representations.
Initial
S2.Final
Data Aug
G vs 0
G vs B
O vs B
Sum(1)
No
No
No
0.12
0.24
0.36
0.72
Yes
No
No
0.19
0.28
0.4
0.87
Yes No
Yes
0.16
0.4
0.36
0.92
Yes
Yes
Yes
0.28
0.48
0.48
1.24
Table 5: Impact of Trajectory Segment Similarity and Data Augmentation. The ablation study demonstrates the value added by incorporating initial and final trajectory segment similarity using S2 contrastive and location-sensitive time warping into our method. Fach component enhances the separability of demonstration quality from new demonstrators, leading to superior representations and increased total Wasserstein distance.
Table 5 shows that integrate each element of L2D, the total Wasserstein distance increases, indicating improved separability of demonstration quality. The full configuratio with both forms of trajectory similarity and time-warping augmentation achieves the highest total distance. This val- idates the effectiveness of L2D's design choices in improving the quality of learned representations.
B Real Robot Demonstrations
A+..
ALd
U
m
Figure 6: Real Robot Execution. We show the rollout results of the policies trained with the demonstrations selected by our method and naive sampling for the Stack task.
We collected physical robot demonstrations for the Lift and Stack tasks and simulated demonstra- tions for the Square task to provide a more realistic setting for evaluating our method. The Lift task had 60 demonstrations the training set Danown and 60 demonstrations in the test set D Dunknown- The Square and Stack tasks had 100 demonstrations each Dnnown and Dunknown- The labeled demon- stration data had three quality types: good, okay, and bad. The real robot demonstration data was high-dimensional, containing camera images, gripper position, end-effector pose, joint positions, and joint velocities of the robot arm. After selecting high-quality demonstrations using L2D approach and baseline approaches, we trained an imitation learning (IL) policy on the best demon- strations from Dinown combined with the selected demonstrations from Dunknown- We report the success rate of the trained II. policy on the real robot task.
For the Stack task, we compare the performance of our method with naive sampling. Our results show that policies learned from higher-quality demonstrations exhibit robust and efficient task com- pletion, achieving 80% success rate over 10 trials. Conversely, policies learned from demon-
12

---

strations selected through naive sampling occasionally display peculiar behaviors, such as waving around the target position or dropping the grasped cube at unsafe heights. Consequently, these behaviors contribute to a lower success rate of 50%. Our results demonstrate that the quality of demonstrations plays crucial role the performance of learned policies on real robots. By lever- aging our L2D approach identify high-quality demonstrations, we can train policies that achieve more robust and efficient task completion.
C L2D Hyperparameters
Hyperparameter
Default
discriminator learning rate Loosel Q discriminator mum semmeeneets discriminator disrriiinatorssgmmentLooo segment_lngth discriminator batch size discriminator training steps initial arajectory-segmeen
0.0001
0.1 [31]
20000
48
128
500000
12
final trajeatrajttrrrooo _ttajecctorrsseeeeennt segment_len use_ .oc_pposnnnoding discriminator httpiiiiinnor.mmmmdddinn....oo Architecture
6
True
12
[LSTM,2 MLPs.Flatten),, 2 MLPs] [2MLPs]
Table 6: Hyperparameters used for training L2D to perform Robomimic based ablations.
D Imitation Learning Policy Training Details
We choose the default BC-RNN model setting from Robomimic benchmark [11] for learning policies. For simulation tasks the agents only low-dimensional observation (e.g., robot proprioception, object poses), the epochs are set to 2000 with batch size 100, and we test the agents by running rollout with a maximal hori- zon 500 in every 50 epochs. For real-world tasks, we also include the RGB color observa- tions from the three cameras (two at the side, one at the robot's wrist) and the epochs to 500. All networks are trained using Adam optimizers [41].
Hyperparameter
Default
Batch Size
16
Learning rate (LR)
le-4
Num Epoch
1000
Train Seq Length
10
MLP Dims
[400, 400]
Image Encoder Wrist View
ResNet-18
Image Feature Dim
64
GMM Num Modes
5
GMM Min Std
0.01
GMM Std Activation
Softplus
Table 7: Hyperparameters for L,2
For the Robomimic Square task on Multi-
Human Dataset experiments in Section 5, there is an equal 150-150 demonstration split. L2D trains on 150 known demonstrations and is tasked select the demonstrations that belong to the "good" quality from the set of 150 unknown demonstrations. Familiar demonstrator experiment in Table 2, aims to test the II. policy's performance when it has access combination of known and unknown demonstrations. Specifically, BC-RNN is provided with 100 demonstrations: the top 50 from D Danown and another 50 chosen from Dunknown- For the unfamiliar demonstrator experiment in Table 3, BC-RNN trains exclusively on 50 demonstrations selected from Danknown-
Once have filtered demonstrations using different methods (including proposed method and base- lines) and consistent heuristics across methods, we employ Behaviour Cloning for training. We use RNN-based behavioral cloning (BC-RNN) to evaluate algorithm performance using subsets based on demonstrator quality as prior works show [6, 11, 16] it is effective at handling mixed quality data.
13


Reviews:
[{"review_id": "ldf6fSKItf-", "reviewer": "Reviewer_pGpd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Imitation learning requires large diverse datasets, but these datasets will come from a variety of demonstrators, and the quality cannot be guaranteed to be uniformly high. This paper proposes a method to select high-quality demonstrations that can be used for training policies. The approach relies on a small subset of the demonstrations being labeled with their quality, from which a latent representation is learned, followed by a quality critic. The critic is then used to pick the top quality trajectories from the unlabeled set, which can then be used to train a policy.\n\nSimulation results show that the proposed method outperforms previous work as well as different ablations of it. Some real-world experiments exist as well, but the details are unclear (see details below).", "summary_of_recommendation": "The paper has good ideas, but is significantly weakened by its presentation. Many critical aspects of the algorithmic and experimental details are not provided in the main text or the appendix. There are also severe inconsistencies about which experiments were actually conducted in the real-world, which leads me to discount those results entirely until more evidence is shown.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The detailed list of quality/clarity issues, and some experimental issues were listed above, and could be fixed during the revision period.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Quality and Clarity\n\nOverall, the writing in the paper needs a lot of improvement to make it clear. Currently, the methods and evaluations are not as clear as they should be, and this is impacting the quality of the paper as well. There could be an opportunity to tighten this up during the rebuttal phase. The details are listed below:\n\nLine 138 to 143 (Training Quality Critic): It’s unclear if this is a classifier or a regressor (since many other parts of the text talk about scalar values being assigned to the quality labels). Similarly, is the added label noise in the discrete label space or the continuous scalar space?\n\nLine 176: “filtered demonstrations”: the filtering procedure is not clear. In the previous section, the paper says that “we use a heuristic to determine whether the demonstration should be used for training the policy”, and provides an example heuristic. However, for experiments, a description of the actual heuristic used needs to be shown so that the results can be correctly interpreted.\n\nLine 177: “different methods” – it’s unclear what methods we’re talking about here. Maybe you could instead say that the “filtered demonstrations” are obtained through different methods, including the baselines and the proposed method.\n\nLine 187: “quality subset distributions” – it’s unclear what this means. It’s probably better described as  “histogram of estimated values Q() for each subset of trajectories with specific labels”. \n\nSection 5.1: The baselines should be described better. What is “the basic preference learning approach” (line 190)? Does it involve passing the trajectory directly into the quality critic, instead of the embedding? Or is it a prior method from the literature?\n\nFigure 3: This should probably be split up into a separate table and figure. The table should be captioned separately. “G vs O” etc titles are unclear; you should be able to fit in “good vs ok” and have a better description in the caption as to what the numbers mean.\n\nFigure 3: These distributions seem to show a bucketed version of an estimated scalar value (since we see multiple bars for each integer). However, the previous section (line 156, “Estimating Quality of Unseen Demonstrations”) says that each demonstration is evaluated by sampling segments from it and getting a scalar score from each of them. How is an aggregate value obtained for a single demonstration? Or does the histogram show the estimated values of each segment? It would also be good to understand the actual labels that were used (i.e., was it bad = 1, ok = 2, good = 3?)\n\nMain results: (paragraph starting line 201) – the description of the experiment is not very clear. It should probably be mentioned that $D_{known}$ is used for training, and that the “pool” being tested is $D_{unknown}$, but that’s just my assumption – it should be clarified in the text. \n\nLine 208: Please use the term ILEED instead of just “[6]”, since you also use it in the table and elsewhere.\n\nLine 209: “SR” is used before defining it. Also, this section should refer to Figure 4 in the text, since that’s the raw data based on which the claim is made.\n\nFigure 4: The table on the right needs to be explained in more detail – it looks like you run each method on the entire dataset (training and test), and look at the ground truth labels of the top 100/300 high-quality trajectories? (I tried to infer this from the values in the table, but please describe this explicitly). Or is it the top 100/150 from $D_{unknown}$? (In any case, it should probably be restricted to $D_{unknown}$ since evaluating on the training set would not be informative).\n\nIt would also be good to evaluate this on more than just a single task from Robomimic – at least expanding this evaluation to the other three Robomimic tasks would convince the reader that this result and algorithm can work in various settings.\n\nIdeally, it would be great to also include a result from behavior cloning on *all* demonstrations as a baseline, not just the naive sampling. Yes, it will be trained on a larger number of demonstrations than the others, but in my opinion this would be a fairer (and more realistic) comparison than randomly sampling a smaller set, since the full dataset is available and there’s no reason to subsample if quality predictions are not available. This should be the bar to beat – by sub-selecting the “good data”, can you outperform a model that’s simply trained on all the data?\n\nSec 5.3: Learning from real-world demos: Please specify the number of demonstrations per task that were used for training and test.\n\nLine 239: please add the table number as reference.\n\nThere are many inconsistencies in the specification of which tasks were actually conducted in the real world. Line 231 says “tasks such as Square Nut Assembly, Lifting, and Stacking in both simulated and real-world environments”. The Table 1 Caption says that only Lifting was in the real world, and the other were in sim. And the table itself says that Lift and Stack were in the real world, while Square was in sim. The level of inconsistency here, combined with the lack of any real-world videos leaves me suspicious about which experiments were actually conducted in the real world. If the first two rows of Table 1 were indeed in a simulated environment, what is this result trying to show over what was already shown in Section 5.2? Is this about using an image-based end-to-end policy in simulation, vs the previous section being state-based? \n\nAlso, line 236 (in the real-world experiment section 5.3) is where I realized for the first time that the previous experiments (5.1, 5.2) were not using images. The inputs/outputs/action spaces for the tasks are not described anywhere, but they need to be.\n\nMany tables are captioned as “Figure xx” - please fix them, and the references to them (some references are called Table xx, which currently don’t exist.)\n\n### Originality\n\nThe proposed approach has novelty compared to prior approaches, at least based on my knowledge of the field. However the quality and clarity issues mentioned above significantly weaken the paper.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "cWYpKj6neV", "reviewer": "Reviewer_xRQR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method to learn from a variety of demonstrations. The approach uses a small number of quality labels on demonstrations to train a latent space and an evaluator that uses sequence-level latent features to predict demonstration quality. By training only on good demonstrations, the authors show improvements over prior work.", "summary_of_recommendation": "I think the paper addresses an important problem. But, as written, I do not think it is ready for publication. There are too many missing details that make it hard to evaluate.\n\n--\npost rebuttal: the paper has improved but can still be further improved with more details. however, the overall idea has merit so I am leaning towards acceptance, but wouldn't argue against rejecting the paper.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Given that you have quality labels already. Why not train a classifier to map directly from segments to qualities and then just train on new demos that have high-quality predicted labels? This seems like it could be a strong baseline to compare against. Also, why not just compare against something trained to map directly from segments to scores via preference learning and use the raw scores without the GMM to compute better and worse trajectories?\n\nFig 3. It is not clear what G vs O or G vs B represent. Also the captions say Good, Okay, Poor, but not Bad. \n\nNo discussion of user study details, IRB approval, participant information, number of participants, training routine, etc. \n\nIt is unclear what the ablations mean and how they are implemented. Also, I see that they lead to better distance in the latent space, but I don't see where it is shown whether this distance correlated with higher success.\n\nIt is unclear what the tasks are, but the authors assume familiarity. Please add figures and descriptions before discussing nuances of the environments. E.g., line 108 talks about square but this doesn't make sense unless readers are already familiar with this env, which they probably aren't. Also, bottleneck states are very important and I don't see why they are indifferent to demonstration quality. Usually bottle neck states are the ones that actually matter.\n\n\nLine 208. Need to reference authors or algo name for [6] and describe this baseline. Where are the results for this claim?\n\nLine 209 What is SR?\n\nWhat kind of labels are provided? Good, Okay, Bad? Is this provided for each trajectory or each segment? Based on section 3 is seems that each trajectory gets a high-level quality label, but then how do you get D_A, D_B, etc which seem to correspond to trajectory segments of the same quality.\n\nLine 110: How do you know how to sample quality influencing trajectory segments? Do you train a classifier?\n\nStrategy 2 is confusing as written. Why not sample from final segment or same region?\n\nLimitations talks about using high-quality sections of bad demos. This seems possible since the training data has segments of good and bad portions of full trajectories and that the method predicts the quality of segments so why not do this? What is the barrier?\n\nHow is the GMM used to predict a trajectory-level preference label? It seems that it is trained on trajectory snippets.\n\nHow is the encoder and preference model trained? What are the loss functions?\n\nWhat is the heuristic used for determining whether the demonstration should be used for training the policy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like the overall idea; however, there are lots of typos and the paper is lacking many important details. It is unclear whether Strategy 1 and 2 are used together or whether one is chosen. The domains deserve more description before discussing their nuances. Also, the way in which the latent space and quality predictor are trained and used is not rigorous enough and makes it hard to understand. I like that the authors use real demonstrations and test on a real robot. The problem is important and the method seems to work well, but the writing needs work and many details are missing.\n\n---\nPost rebuttal: I still think there are many details lacking: data on users who provided demonstrations, specific formulas used for the different training losses, how the Wasserstein distance was calculated, pseudo-code for the different parts of the algorithm, etc. I am updating my score to a weak accept, since I think these issues can be corrected in the camera ready draft. I strongly encourage the authors to significantly expand the appendix to foster reproducibility as well as to release code since it seems very hard to reproduce given how the paper is currently written.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "LwJxZPHaoJ", "reviewer": "Reviewer_L98X", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In their paper, the authors propose a novel method for learning a latent representation of demonstrations, which temporally embeds their segments. The learned latent space is then used to train a preference model, enabling the discernment of high-quality demonstrations. Subsequently, the authors employ these filtered demonstrations to train an agent using a behavior cloning method. To evaluate their proposed approach, the authors conducted experiments in simulation as well as on a real robot, demonstrating its superior performance when compared to the baseline methods", "summary_of_recommendation": "The paper is not clearly written, evaluation is incomplete, and the contribution is not significant.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Could the authors reference results presented in the tables in the text of the paper?\n\nCould the authors evaluate the additional baseline that is mentioned in the review?\n\nWhy did the authors choose BC-RNN algorithm? Did they consider other BC methods?\n\nPlease improve the writing by addressing points in the review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The clarity of the paper can be significantly improved. At the end of the review is a list of specific suggestions and recommendations\n\nUtilizing a contrastive loss to embed demonstration segments into a latent space proves to be a valid approach. Prior works, such as [1], have explored the use of embedding task progression for agents with varying embodiments. Overall, this topic holds significant relevance, and further research in this direction could offer valuable insights for the field of imitation learning.\n\nIn Lines 128-129, the authors assert that preference learning is not effective in manipulation tasks. However, this claim appears to be at odds with the conclusions drawn in various papers on preference learning [2][3][4], where manipulation tasks have been explicitly taken into account. Therefore, it is essential for the authors to acknowledge and discuss these relevant works in the Related Work section\n\nIn the paper, the authors made reference to a relevant work [5], which could have served as a suitable baseline for their work. Unfortunately, they did not include a comparison with this paper. Additionally, the lack of clarity in their writing makes it difficult to provide further comments on their results. Specific points can be found at the end of this review.\n\nThe authors should provide reasoning for choosing BC-RNN as their BC method. Moreover, an ablation study encompassing various BC approaches would constitute a valuable contribution.\n\nPresently, the authors assess their method under the condition where an equal number of demonstrations exist within each quality group, namely \"good,\" \"okay,\" and \"bad.\" However, it is beneficial to extend the evaluation to encompass scenarios where the relative quantity of \"good\" demonstrations is reduced. Such case is relevant as it mirrors real-world situations in numerous tasks.\n\nSuggestions for improving the clarity:\n\nIn Line 18, the authors introduced the acronym IL for imitation learning; however, they do not use it consistently afterwards. Overall, the authors are advised to adhere to the standards of academic writing when using acronyms\n\nFigure 3 consists of a table and a figure, and should be appropriately separated. In Line 189, authors referenced Table 3, but in the paper no table is labeled in such manner. Most likely the authors discussed the table in Figure 3, but that should be clearly written.  Moreover, Figures 4 and 5 are actually Tables. Most of the figures are never referenced in the paper. \n\nAuthors introduce some notation which can be confusing when considering other papers in the domain. Specifically, they represent the preference network with 'Q' (line 95), which is commonly used to denote the state-action value function in Reinforcement Learning\n\nMinor writing comments:\n\nLine 108: the authors mention the task Square, a reference and short short description of the task would increase the readability. \n\nLine 172: A reference to the robomimic dataset should be included.\n\nLine 248: Sentence does not have a verb in it.\n\nLine 294: Name of an author is unnecessary bold.\n\n[1] K.Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, D. Debidatta, XIRL: Cross-embodiment Inverse Reinforcement Learning, CoRL 2021\n\n[2]K. Lee, L. Smith, P. Abbeel, PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training, ICML 2021\n\n[3]A. Taranovic, A. Kupcsik, N. Freymuth, G. Neumann, Adversarial Imitation Learning with Preferences, ICLR 2023\n\n[4]C. Kim, J. Park, J. Shin, H. Lee, P. Abbeel, K. Lee, Preference Transformer: Modeling Human Preferences using Transformers for RL, ICLR 2023\n\n[5] D. S. Brown, W. Goo, and S. Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations, ICLR 2019", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "8yIgCaG2GV", "reviewer": "Reviewer_kfod", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper learns to identify  the quality of an unseen demonstration and thereby decide whether to use it for imitation learning. To do so, this work learns a quality critic from a small set of ranked demonstrations. This quality critic can evaluate segments of length L from each demonstration, which are encoded into a position-aware latent representation. The latent representation is trained first using contrastive strategies.  The quality critic is trained by casting the predicted labels from all the segments in the demonstration into a Gaussian Mixture Model  where the average predicted label from the GMM is matched with the true demonstration label. During test time, the suitability of an unseen demonstration for imitation learning is decided by counting the quality critic labels assigned to the highest quality set.", "summary_of_recommendation": "The paper was easy o read and well motivated in my opinion with nice experiments, but still has some clarity issues that need to be addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "    1. Why is it important to separate the demonstrated trajectories into subsets D_A, D_B, D_C etc.? If there are a limited number of individual labels, why can these be not directly used to train the quality critic? Related questions: \n        1. How many individual labels and how many subsets are there in your experiments\n        2. How are the subsets chosen or arranged?\n    2. From the section “Training Quality Critic”, the quality critic seems to be a multi-class classifier to me. I do not understand then why comparing pairs of datasets necessary (lines 142-143) . Also confusing - Is the input to the quality critic the trajectory segment or a pair of trajectory segments?\n    3. CAIL sounds like a better baseline for comparison to me – since it also assumes access to demonstration labels. The authors have mentioned this paper in related works and seem to have dismissed it as a baseline citing its requirement for “dense labels”. However, the authors also assume access to quality labels for each of the demonstrations in their known dataset.   \n    4. Are the suboptimal demonstrations simply used for training the quality critic and not in any way used for imitation learning?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the paper was well organized and fairly easy to read and well motivated. While the individual concepts introduced by the paper are not particularly novel, the motivation behind putting them together (for the most part) to solve the problem of imitation learning from heterogenous distributions is explained convincingly. The experiments and the ablation studies  were also very interesting. However, there are some foundational steps (see questions for rebuttal) that are unclear.\n\n**Strengths:**\n- ablation studies and experiments look very convincing\n- results on hardware\n\n**Weaknesses:**\n- Some clarity issues (See questions for rebuttal).\n- I am not very convinced with the choice of baselines. ELICIT does not seem very suitable as a baseline since the objective of the method is quite different, maybe something like CAIL and TREX is better suited.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper offers an intersting approach to offline imitation learning with convincing experiments. While initially, the reviewers have identified several concerns regarding the choice of the baseline, the number of experiments (e.g. for robomimic) and the presentation of the approach, these concerns were adequately addressed by the reviewers. The authors clarified their choice of baselines which and added new experiments with additional imitation learning datasets. While there is still one reviewer with a "weak reject", I consider most of the points addressed, where the only comment not addressed is the use of more recent imitation learning algorithms as backbone (e.g. diffussion policies have shown to outperform BC-RNN by a large margin). Yet, this can be left as future work in my opinion and I would still recommend the paper for publication.
Decision: Accept (Poster)