KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidance
Authors: Jingxian Lu
Wenke Xia
Dong Wang
Zhigang Wang
Bin Zhao
Di Hu
Xuelong Li
Venue: CoRL 2024 (2024)
Keywords: Online Imitation Learning; Robotic Manipulation

Abstract: Online Imitation Learning methods struggle with the gap between extensive online exploration space and limited expert trajectories, which hinder efficient exploration due to inaccurate task-aware reward estimation.
    Inspired by the findings from cognitive neuroscience that task decomposition could facilitate cognitive processing for efficient learning, we hypothesize that an agent could estimate precise task-aware imitation rewards for efficient online exploration by decomposing the target task into the objectives of "what to do" and the mechanisms of "how to do".
    In this work, we introduce the hybrid Key-state guided Online Imitation (KOI) learning approach, which leverages the integration of semantic and motion key states as guidance for task-aware reward estimation.
    Initially, we utilize the visual-language models to segment the expert trajectory into semantic key states, indicating the objectives of "what to do". 
    Within the intervals between semantic key states, optical flow is employed to capture motion key states to understand the process of "how to do".
    By integrating a thorough grasp of both semantic and motion key states, we refine the trajectory-matching reward computation, encouraging task-aware exploration for efficient online imitation learning.
    Our experiment results prove that our method is more sample efficient than previous state-of-the-art approaches in the Meta-World and LIBERO environments. We also conduct real-world robotic manipulation experiments to validate the efficacy of our method, demonstrating the practical applicability of our KOI method.

KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidance
Jingxian Luk,, Wenke 12*f Dong Wang Zhigang Wang', Bin Zhao² Di Hul Xuelong Li4
'Gaoling School of Artificial Intelligence, Renmin University of China Shanghai Artificial Intelligence Laboratory *Northwestern Polytechnical University *Institute of Artificial Intelligence, China Telecom Corp Ltd
Abstract: Online Imitation Learning struggles with the gap between extensive online exploration space and limited expert trajectories, hindering efficient explo- ration due to inaccurate reward estimation. Inspired by the findings from cognitive neuroscience, we hypothesize that agent could estimate precise task-aware re- ward for efficient online exploration, through decomposing the target task into the objectives of "what to do" and the mechanisms "how to do". In this work, we introduce the hybrid Key-state guided Online Imitation (KOI) learning method, which leverages the integration of semantic and motion key states guidance for reward estimation. Initially, we utilize visual-language models to extract semantic key states from expert trajectory, indicating the objectives of "what to do". Within the intervals between semantic key states, optical flow is employed to capture motion key states to understand the mechanisms of "how to do". By integrating a thorough grasp of hybrid key states, we refine the trajectory-maten computation, accelerating online imitation learning with task-aware exploration. We evaluate not only the success rate of the tasks in the Meta- World and LIBERO environments, but also the trend of variance during online imitation learning, prov- ing that our method is more sample efficient. We also conduct real-world robotic manipulation experiments to validate the efficacy of our method, demonstrating the practical applicability of our KOI method. Videos and code are available at /
reward
gi github.cc. io/Keystate__aeee 0_ee___nneeeeeeeieeeee Imi Amitation/.
Keywords: Online Imitation Learning, Robotics Manipulation
Introduction
Imitation Learning has achieved significant success in various robotic manipulation tasks through offline and online modes [1, 2, 3, 4]. The offline approach [5, 6, 7, 8, 9] seeks to learn policy by mimicking state-action pairs from expert demonstrations through supervised learning. Due to its independence from interaction with environments, offline imitation learning necessitates extensive robotic data, which requires expensive costs associated with worl-world data acquisition [10, 11, 12]. In contrast, online imitation learning [13, 14], driven by reward function informed by expert demonstrations, is able to formulate policies via exploration. Although has the potential re- fine policies with minimal expert input, the considerable gap between substantial online exploration space and limited expert trajectories challenges the estimation of exploration reward, which hin- ders the efficiency of online imitation learning. To estimate fine-grained reward, Optimal Transport (OT) [15, 16, 17, 18] is proposed to facilitate trajectory-matching reward by measuring the distance between exploration trajectories and expert demonstrations. However, their insufficient grasp of the semantic comprehension tasks makes them easily converge on sub-optimal solutions [15].
Equal contribution Work done during internship Shanghai Artificial Intelligence Laboratory I Corresponding Author (wangdong @ pjlabborg.cnn didu@ruc.edu.cc)
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Pivotal research in cognitive neuroscience [19, 20] suggests that humans could enhance cognitive processing and learning efficiency via task decomposition. Inspired by these findings, we hypothe- size that agents could optimize online imitation learning efficiency by task decomposition. Specifi- cally, by gaining a holistic grasp of both the objectives ("what to do"" and the mechanisms ("how to do") of the task, an agent could be encouraged conduct task-aware exploration.
To this end, we propose the hybrid Key-state guided Online Imitation (KOI) learning method, which effectively extracts the semantic and motion key states from expert trajectory to refine the reward es- timation. We initially utilize the rich world knowledge of visual-language models to extract semantic key states from expert trajectory, clarifying the objectives of "what to do". Within intervals between semantic key states, optical flow is employed to identify essential motion key states comprehend the dynamic transition the subsequent semantic key state, indicating "how to do" the target task. By integrating both types of key states, we adjust the importance weight of expert trajectory states in OT-based reward estimation to empower efficient online imitation learning.
To validate the efficiency of our KOI method, we conducted comprchensive experiments across Mera-World manipulation tasks and long-horizon LIBERO tasks. The analysis of experiments highlights the strength of our semantic and motion key state guidance, which enhances the sample efficiency of KOI compared previous methods. In the end, real-world experiments conducted to prove that method could also generalize to real complex scenarios.
Related Work
Online Imitation Learning. Online Imitation Learning seeks to develop policies through online ex- ploration, guided by a reward function informed by expert demonstrations [21, 22, 23, 24). Recog- nizing the difficulty of obtaining state information real-world settings, some researchers [25, 26] explored the use of visual and language cues inform task-aware reward estimation. However, these works mainly focused task semantic knowledge, overlooking detailed trajectory informa- tion. In this study, we adopt the trajectory-matchin reward estimation method, integrating task semantic comprehension with fine-grained trajectory motion for efficient online exploration.
Optimal Transport for Imitation Learning. Optimal Transport (OT) computational frame- work that addresses the problem of finding the most efficient way to transform one distribution into another, minimizing a predefined cost for this transformation [27, 28, 29]. Prior works borrow vari- ous distance metrics to measure the distance between the online exploration trajectories and expert demonstration for fine-grained imitation reward estimation [17. 18, 30, 15]. However, they regret- tably ignore the semantic information of manipulation tasks, leading to convergence on suboptimal solutions [15]. In this work, we propose to combine the finegrained eajectory-matching reward with semantic and motion key states, facilitating the efficiency of online imitation learning.
3 Method
During online imitation learning, a significant challenge lies in the gap between extensive online exploration space and limited expert trajectories, resulting in inefficient exploration due to imprecise imitation reward estimations. In this work, to precisely estimate task-aware reward, we extract the hybrid key states from semantic and motion aspects by the Semantic Decomposition Module (SDM) and Motion Capture Module (MCM), indicating "what to do" and "how do" for the target task. By integrating both types of key states, we adjust the importance weights of expert trajectory within the OT-based reward estimation method, enhancing efficiency of online imitation learning.
3.1 Background
The goal of online imitation learning is train a policy n" through exploration with the reward re informed by the expert demonstrations T The i-th state from exploration trajectory T is denoted as sf. and the j-th state from expert demonstration 4 is denoted as sy. The similarity between them

---

Description
Expert Demonstration
Exploration Trajectory
Put the cabe
Sumantic Index
Key
Online Imitation Learning
doppant
Exploration observation
OT resord
Semantic
Pick up
cube
Decompoation
cumulettv
Importance
opticall
Adaptin
MA
Moriion
Motion Index.
Motion
Expert Importance Distribttio
Computation
The pipeline of hybrid Key-state guided Online Imitation (KOI) learning method. We first extract semantic key states with the Semantic Decomposition Module. Within intervals between semantic key states, the Motion Capture Module is proposed to identify motion key states. Further, we adjust the importance weight in OT-based reward estimation with these hybrid key states, enable task-aware exploration for efficient online imitation learning. The color intensity of OT matrix represents the value of estimated reward.
Figure 1:
measured by function C, which is defined as cosine similarity in this paper. Accordingly, the cost of transporting from source sf. destination sy is calculated follows:
Cjj 1 C(s 19)
(1)
Given the cost above, methods based Optimal Transport (OT) [15, 16] aim to find the optimal transport matrix 17" to estimate reward re of each state. For example, the reward of exploration state at timestep i is estimated using the subsequent formula:
Nd
(s
Cij
(2)
j=1
where 77 NN* xX' and each element Tky represents the amount of data transported from to j. Ne and N are lengths of exploration trajectory T and expert demonstration 7 respectively. The op- timal transport matrix 1y" is obtained by measuring the distance between importance distribution μ of exploration trajectory T and importance distribution v of expert demonstration 7 with Sinkhorn algorithm [27] employing an entropy regularization H to enhance computational efficiency:
N
N
71" min
Ci (1)
1=1
Nd
s.t.
Ml
for 1,... No
(3)
/=1
Σ
for 1...., Nd
Tkj Vi
However, previous OT-based methods assume the distribution of expert demonstration uniform distribution for each state, i.e., ạ 1. While this assumption facilitates fine-grained rewards, overlooks the importance among states. Such oversight hampers exploration efficiency for lack of task-aware state comprehension, revealed in the qualitative analysis of Section 4.4.
To integrate trajectory maatchn reward with task-aware information, we extract the semantic key states, highlighting the objectives of "what to do"., and identify the motion key states, revealing the mechanisms of "how do'. With the guidance of both types of key states, we adjust the importance distribution of expert trajectory for accurate reward estimation described in Section 3.4, which encourages the agent conduct task-awar exploration to facilitate online imitation learning.

---

3.2 Semantic Key-state Extraction
Foundation Models have demonstrated their rich world knowledge and strong generalization pabilities in various visual-language tasks [31, 32]. In this work, we propose utilizing GPT-4v to design the Semantic Decomposition Module (SDM) for task-specific subgoal extraction.
Concretely, we initially instruct GPT-4 to decompose the task description I, into K subgoals, de- noted as L = ( (i,, Subsequently, GPT-4v is employed to locate corresponding semantic key states of the expert demonstration. To manage the number of input images, sample states every T' step from expert observations, forming the query sets O. To ensure the temporal consistency of located key states, both sampled expert observation O and subgoal language descriptions I are fed into GPT-4v to get the semantic key state index set I
P = GPT-4v L*).
(4)
Consequently, SDM produces a temporally coherent sequence of indices 1 that precisely match the semantic key states to subgoal descriptions, clearly defining "what to do" at each stage of the task. Due to the length constraints of the paper. please refer to Appendix A for detailed implementation.
3.3 Motion Key-state Identification
While the SDM effectively recognizes semantic key states, they primarily outline the task objectives, without delving into manipulation details required for transition to these target states. To enhance the grasp of mechanisms of "how to do" the task, propose the Motion Capture Module (MCM) to identify the motion key states within the interval defined by two semantic key states.
Specifically, we measure the intensity of motion between two consecutive states by the absolute value of optical flow their corresponding observations, which is captured by the Farneback [33] optical flow estimation algorithm F. The p-th motion key state represents the critical transition from semantic key state ST, to SIş, whose index 1 (p) is identified by following formula:
1 arg max (E(
|F(o
(5)
By identifying the most intense optical flow, we select the related states as the motion key states, with index set 1" which indicate "how to do the target task.
3.4 Importance Weight Adaption
Based on the extracted hybrid semantic key states and motion key states, we develop the Importance Weight Adaption method to guide the agent in online exploration. Previous studies [15] typically assumed that the importance weights assigned to each state in expert trajectory / in Equation are equal, namely,
N² While it aids vajectory-matchingg the estimated reward under this assump- tion disarmed of task awareness. In this module, enhance task-aware reward by emphasizing the guidance of semantic and motion key states. Thus, design the importance weight of expert demonstration, which was decomposed into K subgoals, Gaussian Mixture Distribution D:
K D AL o3) + +K K
ATT
ΣK AK
K Am
where we adjust the semantic variance to be smaller than the motion variance om. guiding the agent to pay more attention to semantic information. The weights of these two types of key states A and A are introduced to ensure the dominance of task completion status when reward estimation. The choice of hyperparameters is detailed in Appendix B.2
Additionally, replace the uniform distribution assumption of expert demonstrations with the normalized distribution D, enhancing task-aware exploration during online imitation learning.

---

3.5 Learning Paradigm
To fully utilize the expert demonstrations, we employ an diline-to-onling methodology, initially training a behavior cloning policy n. using the state-action pair (sa, da) in expert trajectory 7
LBC ((arna))T ||aa (aaa->((sa)) (sa))
Initialized our exploration policy x" as the behavior cloning policy n², the agent could explore the environments with task-aware knowledge and measure the similarity between exploration trajec- tory and expert demonstrations, Further, we utilize the n-step Deep Deterministic Policy Gradient (DDPG) [34] with our estimated exploration reward re to update policy л", and Q-function Qo:
π = arg max [(L -
[Qa(se ae)] aa(TT (( I|a - (saa)l]] (8)
In this step, we employ the adaptive function following the Regularized Optimal Transport (ROT) [15] to effectively balance the weight between BC regularization and online exploration:
A(x)) = E (a.)~To [ Pooooo (ons4.too))))
However, the limited expert demonstrations commonly fail to offer task-aware reward 7 for exten- sive online exploration space. The KOI method we proposed release this issue via the guidance of hybrid semantic key states and motion key states, accelerating the online imitation learning.
Experiments
4.1 Experiments Setting
To comprehensively evaluate our key-state guided method, we initially conducted experiments across 6 tasks from the Meta-World suite [35] to validate the efficacy of our method. Further, we sess KOI's performance in complex manipulation tasks, conduct tasks from the LIBERO suite [36]-notable for its complex scenarios, large action spaces, and long task sequences.
In offline imitation phase, we collected demonstration and 50 demonstrations per task in the Meta- World suite and LIBERO suite to train BC policy, respectively. For online imitation phase, we loaded pretrained BC policy and initialized critic network. Details of settings are attached Appendix B.
Comparison Experiments
To evaluate our methods, we conducted comparison experiments with various methods, all methods initialized with pretrained behavior cloning policy x to ensure fair comparison.:
BC represents the pretrained behavior cloning policy nb.
UVD [37] utilize pretrained visual encoder R3M [38] for task decomposition in image-goal reinforcement learning.
RoboCLIP [25] leverages the pretrained video encoder S3D [39] estimate the task-aware reward function instead of the reward of each state.
ROT [15] employs Optimal Transport to estimate the distance between exploration trajec- tory and expert demonstrations, which focus on fine-grained trajectory matching.
As the results presented in Figure 2 shown, the UVD method, which emphasizes semantic subgoals rather than the mechanisms "how to do", often fails to accomplish the target task. Similarly, RoboCLIP leverages a video encoder to capture trajectory motion, but the absence of per-state re- ward hinders its ability guide online imitation. Although ROT employs fine-grained reward and shows commendable performance, it sadly neglects task-aware reward, thereby impeding efficient exploration. This issue is highlighted in the qualitative analysis detailed in Section 4.4.
5

---

metaworld bin picking
metaworld drawer open
libero put wine to cabinet
1.0
1.0
1.0
0.8
0.8
succesS Eate
success rate
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.5
1.5
frame
1e5
frame
1e5
frame
1e5
RoboCLIP
KOl (Ours)
Figure 2: A subset of experiment results on Meta- World and LIBERO suites, with exploration 5 le5 and 2x le5 timesteps respectively. The shaded region represents = I standard deviation across seeds. The results prove that our KOI method excels in sample efficiency compared to others.
In contrast, KOI method achieves tasks such as "drawer open". For the challenging "bin picking" task, characterized by multiple ma- nipulation subgoals, only the fine- iine-grained reward estimation methods (our method and ROT) have achieved success. Furthermore, our method outperforms ROT by leveraging a thorough comprehen- sion of both semantic and motion key states, providing distinct advantage.
efficient online imitation learning in simple World
The tasks in the LIBERO suite present increased difficulties, due to longer task sequences and larger action spaces. The results demonstrate that KOI not only yields more efficient learning but also maintains greater stability, benefiting from the comprehensive guidance from the hybrid key states.
metaworld bin picking
open
librraanv put wine
0.20
Vgllange
0.15
0.15
0.15
0.10
0.10
0.10
0.05
0.05
0.05
0.00
0.00
0.00
3
4
0
0.5
1 frame
1.5
2
frame
1e5
frame
1e5
BC
RoboCLIP
ROT
KOI (Ours)
Figure 3: The trend of variance during online imitation learning. The variances of KOI continuously decreases over time.
We further discuss the trend of variance over time. As shown in Figure 3, the variance KOI de- creases continuously during exploration while the performance increases, which indicates that our method can enhance model performance across diverse range of environments, thereby demon- strating its robustness and stability.
4.3 Ablation Experiments
Our method proposes the Semantic Decomposition Module (SDM) and Motion Capture Module (MCM) to provide task-aware knowledge for effectiveness of each module, we conducted ablation experiment with the following three settings:
reward estimation. To evaluate the
w/o. Both: This setting selects the state every 10 timesteps the key state and utilizes them adjust the importance weight of expert demonstrations.
w/o. MCM: This setting only uses states selected by visual-language models to adjust the importance weight of expert demonstrations.
w/o. SDM: This setting removes the semantic subgoals inferred by visual-language models and selects motion key states within every 10-state intervals.
As shown in Figure 4, the absence of either semantic motion key states detrimentally affects the efficiency of online exploration, particularly in the complex "bin picking" manipulation task. No-

---

metaworid bin
1.0
1.0
Success rate
0.8
success rate
0.8
0.8
0.6
0.6
success
0.0
0.0
0.0
1e5
frame
1e5
frame
w/o. Both
MCM
w/o. SDM
(Ours)
Figure 4: Ablation experiments of our proposed Semantic Decomposition Module (SDM) and Mo- tion Capture Module (MCM). The shaded region represents + standard deviation across 3 seeds.
tably, both the Semantic Decomposition Module and Motion Capture Module we proposed surpass "w/o. Both" setting in most tasks, demonstrating the effectiveness each module.
Besides, the performance of the "w/o. MCM" setting comparable the "w/o. SDM" setting, demonstrating the equal importance of both types of key states. Additionally, the shaded region for the "w/o. MCM" setting is larger than the other methods, indicating that motion key states contribute to stabilizing online exploration by providing detailed guidance on "how to do" tasks.
4.4 Qualitative Analysis
(a) Expert demonstration
-2.5
-05
reward
reward
-1.0
-1.0
-22
-2.6
timestep
20
timestep
60
timestop
(b) Failure Case
(c) Partial Success Case
(d) Success Case
ROT
KOI (Ours)
Motion Key-state
Semantic Key-state
Figure 5: The qualitative analysis of our method. (a) demonstrates the selected semantic and motion key states. (b)-(d) illustrates three representative cases and estimated reward values in "bin picking" tasks. The triangular and pentagonal icons represent two objectives of this task, with the filling denoting completion status, the corresponding estimated rewards are linked by dashed lines.
We visualize the semantic and motion key states selected by our method. As shown in Figure 5(a), our semantic key states effectively extract subgoals of the task. Our motion key states, capture the most intensive optical flow between semantic subgoals, allowing agents to grasp motion dynamics.
Furthermore, present three representative cases in the "bin picking" task. As shown in Fig- ure 5(b), when the agent fails to complete any subgoals, our method estimates lower reward to
7

---

discourage unsuccessful exploration. In cases where the agent completes only the first subgoal of Figure S(c), the estimated reward is high during the initial stage but lower at the final stage. In contrast, ROT incorrectly estimates a high reward for the final stage. The last case, shown in Fig- ure S(d), demonstrates that our method provides a high reward estimation when all subgoals of this task are completed, thereby facilitating effective online exploration.
4.5 Real-world Results
As shown in Table 1, We conduct real-world experiments using XARM robot equipped with Robotiq gripper. To comprehensively evaluate the performance, the entire imitation learning is di- vided into three stages: offline, online, and test. For each task, we collect 10 human expert demon- strations to train the Behavior Cloning (BC) policy during the offline phase and use only expert demonstration to estimate reward during the online phase. All the evaluations are conducted at varying initial object positions. Implement details can be found in Appendix C.
method and previous trajectory matchiing method [15], reporting the success rates of each phase in Table 1. As the results show, the ROT method commonly struggle to provide effective reward estimation for unseen initial positions. How- ever, our method utilizes task-aware information to improve the reward estimation,, thereby enhanc- ing the agent's exploration efficiency and performance. The differences are most significant in the task "put the cube on the scale", which requires the highest precision manipulation, making difficult complete the task by simply matching trajectories with expert demonstrations.
We conduct comparative experiments between
Take the rag off
Put the tape in the box
Put the cube on the scale
Method
Ofline
Online
Test
Offlne
Online
Test
Offine
Online
Test
BC
50%
40%
30%
ROT
50%
60%
70%
40%
45%
60%
30%
20%
30%
Ours
50%
70%
100%
40%
70%
90 %
30%
40%
50%
Table The demonstrations and comparative results of our 3 real- robotic manipulation tasks. The initial state of each task is represented in lighter shade, while the completed state is depicted darker shade. The results indicate that our method achieves more efficient exploration and better overall performance.
5 Conclusion and Limitation
In this work, we propose the hybrid Key-state guided Online Imitation (KOI) learning method, which extracts the semantic and motion key states for trajectory-matchie online imitation learning. By decomposing the target task into the objectives of "what to do" and the mechanisms of "how to do"", we refine the trajectory-matchhe matching reward estimation to encourage task- aware exploration for efficient online imitation learning. The results from simulation environments and real-world scenarios prove the efficiency of our method.
reward estimation for
Limitations. The online imitation learning in this study initializes an critic, shown in our results, which negatively impacts early exploration. In future work, offline reinforcement learning could be employed to pretrain both the actor and critic, potentially enhancing initial online learning.

---

ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China (NO.62106272), the National Natural Science Foundation of China (62376222), Shanghai AI Laboratory. National Key R&D Program of China (2022/D01160101) and Young Elite Scientísts Sponsorship Program by CAST (2023QNRC00),
References
[1] N. Ratliff, J. A. Bagnell, and S. S. Srinivasa. Imitation learning for locomotion and manipu- lation. In 2007 7th IEEE-RAS international conference on humanoid robots, pages 392-397. IEEE, 2007.
[2] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic ma- nipulation. International Journal of Intelligent Robotics and Applications, 3:362-369, 2019.
[3] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108. 032998, 2021.
[4] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey learning methods. ACM Computing Surveys (CSUR). 50(2):1-35, 2017.
[5] S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong. J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. arXiv preprint 0rX1v:2403.01823, 2024.
[6] Octo Model Team, Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, Dasari, Hejna, C. Xu, J. Luo, T. Kreiman, Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. https: model gi github..... io, 2023.
[7] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 2xXi::2303.04137, 2023.
[8] X. Pang, W. Xia, Z. Wang, B. Zhao, D. Hu, D. Wang, and X. Li. Depth helps: Improving pre- trained rgb-based policy with depth information injection. arXiv preprint arXiv:2408.05107. 2024.
[9] W. Xia, D. Wang, X. Pang. Z. Wang, B. Zhao, D. Hu, and X. Li. Kinematic-aware prompt- ing for generalizable articulated object manipulation with 1lm.. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2073-2080. IEEE, 2024.
[10] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.1333 2021.
[11] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale, arXiv preprint arXiv:22 (tt::2222.068/1 068177 2022.
[12] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint ar 13..08844 2023.
[13] S. Arora and P. Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297: 297:103500, 2021.
[14] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International conference machine learning, pages 783-792. PMLR, 2019.

---

[15] S. Haldar, V. Mathur, D. Yarats, and L. Pinto. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning, pages 32-43. PMLR, 2023.
[16] S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:3333.01997, 2023.
[17] R. Dadashi. L. Hussenot, M. Geist, and O. Pietquin. Primal wasserstein imitation learning. arXiv preprint arXiv:2006. 04678, 2020.
[18] G. Papagiannis and Y. Li. Imitation learning with sinkhorn distances. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 116-131. Springer, 2022.
[19] U. H. M. Alexander Renkl, Robert K. Atkinson and R. Staley. From example study to problem solving: Smooth transitions help learning- The Journal of Experimental Education, 70(4): 293-315, 2002. 1oi: 000100008000225500099555511
[20] E. Pollock, P. Chandler, and J. Sweller. Assimilating complex information. Learning and instruction, 12(1))61-86, 2002.
[21] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: survey of learn- ing methods. ACM computing surveys, 50, 2017. ISSN 0360-0300. doi: 10. 145/3054912. URL http:://dh hthp:///hlhhaadlee handle net/10059/2200 cle details complete; not marked as pending following upload 10.05.20 20.05.2017 GB - -Now on ACM website, issue still in progress 9/5/2017 LM Not on journal website 24/2/2017 LM Info from contact 10/2/2017 LM ADDITIONAL INFORMATION: Elyan, Eyad Panel B.
COMPLETED Issue in progress, but arti-
[22] M. Zare, M. Kebria, A. Khosravi, and S. Nahavandi. A survey of imitation learning: Algo- rithms, recent developments, and challenges. arXiv preprint arXiv:2309.09) 02409,,, 2023.
[23] J. Ho and S. Ermon. Generative adversarial imitation learning, Advances in neural information processing systems, 29, 2016.
[24] W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell. Deeply aggrevated: Dif- ferentiable imitation learning for sequential prediction. In International conference on machine learning, pages 3309-3318, PMLR, 2017.
[25] S. Sontakke, Zhang, S. Arnold, K. Pertsch, E. Bryık, D. Sadigh, C. Finn, and L. Itti. Robo- clip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36, 2024.
[26] A. Adeniji, A. Xie, C. Sferrazza, Y. Seo, S. James, and P. Abbeel. Language reward modulation for pretraining reinforcement learning. arXiv preprint arXiv:2308.12221 2023.
[27] M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[28] M. Sommerfeld, J. Schrieber, Y. Zemel, and A. Munk. Optimal transport: Fast probabilis-
tie approximation with exact solvers. Journal of Machine Learning Research, 20(105): 2019.
[29] G. Peyré, M. Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-00 607, 2019.
[30] H. Xiao, M. Herman, J. Wagner, S. Ziesche, J. Etesami, and T. H. Linh. Wasserstein adversarial imitation learning. arXiv preprint arXiv:1906.08// 2019.
[31] Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
10

---

[32] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, I.. Hasenclever, J. Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint (rXiv:2306.08644 2023.
[33] G. Farnebäck. Two-frame motion estimation based on polynomial expansion. In Image Anal- ysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29- 29-July 2, 2003 Proceedings 13, pages 363-370. Springer, 2003.
[34] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2015.
[35] Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[36] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
[37] Z. Zhang, Y. Li, O. Bastani, A. Gupta, D. Jayaraman, Y. J. Ma, and L. Weihs. Universal visual decomposer: Long-horizon manipulation made easy. arXiv preprint arXiv:2310.08581, 2023.
[38] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXi::2203.12001 2022.
[39] Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offss in video classification. In Proceedings of the European conference on computer vision (ECCV), pages 1011,21, 2018.

---

Appendix
A
Details of Semantic Decomposition Module
Task decomposition
Key state selection
Task definition prompt:
Task definition prompt:
Suppose you are an observer, watching robot do task. You should divide your observation multi shots based on observed rabot's actions, just describe looks like.
There is robot doing a task, which can be divided multi steps:
(task subgoals) The following pictures process sequentially, output index of the most relevance picture for each step.
the ending each shot
during this
In nonttetttt exemplar:
For example, task that pick up cube and move to a goal position, the output can be as follows: [11": "Gripper grasps the cube.", :
Inference rules:
Output should formatted a python pictures' index.
Inference rules:
divided as possible.
Expert Demonstration:
These endings should be static physical states, dynamic
Task description:
(task description)
Figure 6: The pre-defined prompt used in our Semantic Decomposition Module. Some keywords can enhance the quality of decomposition and selection.
In the Semantic Decomposition Module (SDM), we employ GPT-4v to extract the semantic key states from expert demonstrations, whose rich world knowledge and strong generalization capabili- ties have been proven various visual-language tasks [31, 32]. The process of extraction includes two stages: task decomposition and key state selection.
The first stage takes the description of the task as input, decomposing the manipulation process into multiple subgoals. Concretely, we use the name of the task as the task description. These subgoals will be the criteria for selecting key states from expert demonstrations.
The key state selection stage would take the observation queries and subgoal descriptions into GPT- 4v, and get the key state indexs. The prompts for the two stages are shown in Figure 6. Due the API and context length restrictions, select states from expert demonstration every T step to form the query expert observation sets. In our experiment, T is assigned 10. Empirically, we found that there will be chronological confusion if only one subgoal is passed in each query, egg., selecting the fifth frame for the first subgoal while the third frame for the second subgoal. Passing all subgoals in one go solves this problem effectively.
B Details of Experiments
B.1 Experiments setting
As illustrated in Section 4, the LIBERO suite [36] is notable for complex scenarios, large action spaces, and long task sequences. Thus, to promote policy learning, the policy in the LIBERO suite utilizes both image and proprioceptive as input, while the policy in the Meta- World suite only utilizes image as input. Besides, 50 expert demonstrations are borrowed from their open geen-source release for BC training in LIBERO suite, while the estimation of the agent's exploration reward only based on of them, which reduces the time cost of computation reward. In addition, when the task is finished, each state of the successful exploration will gain task-finia fini-finish reward, to promote policy learning. More details of environment setting can be found at Table 2.
12

---

During imitation learning phase, each task was trained using batch size of 256 over 50k epochs. For the online imitation learning phase, we loaded the pretrained BC policy and initialized the critic network. The model then interacted with the environment over 500k timesteps in the World suite and 200k timesteps the LIBERO suite, allowing us to observe the efficiency of different online imitation learning methods. For a fair comparison ROT [15]. we train the policy using stack of 3 consecutive RGB frames in Meta-World suite, and each action in the environment is repeated 2 times.
Suite
Parameter
Value
Meta-World
Use proprioceptive
False
Image size
84 84
Action shape
4
Frame stack
3
Action repeat
2
Seed frames
12000
Task finish reward
0
Demonstration($) for BC
1
Demonstration(s) for online finetuning
LIBERO
Use proprioceptive
True
Image size
128 x 128
Action shape
7
Frame stack
Action repeat
1
Seed frames
24000
Task finish reward
5
Demonstration(s) for BC
50
Demonstration(s) for online finetuning
5
Table Details of environment settings.
B.2 Hyperparameters
The complete list of hyperparameters is provided in Table 3. As shown, all the methods differ only in reward estimation, i.e., components like encoder and RL backbone are the same. When modeling the importance weight of expert demonstration, we apply distinct weights and standard deviations for semantic and motion key states, respectively. The weight assigned to the ultimate goal of the task set higher than that of the subgoals, thereby enhancing the agent's awareness of task completion.
B.3 Results of Comparison Experiments
In addition to the results provided in Section 4.2, Figure 7 shows the performance of KOI on all tasks 6 from the Meta- World suite [35] and from the LIBERO suite [36]. On all tasks, KOI has shown its sample efficiency compared to other reward estimation methods [15, 37, 25]. However, the "door unlock task, as shown in Figure 12, requires the robot arm to rotate the handle to unlock the door. However, during online exploration, the agent could complete this task by using its body instead of its gripper to rotate the handle, due to the imprecise physical simulation. The tricky policy can complete the task without imitating the expert's trajectory, resulting in the degradation
13

---

Method
Parameter
Value
Common
Replay buffer size
150000
Learning rate
1e-4
Discount
0.99
R-Step returns
Mini-batch size
256
Agent update frequency
Critic soft-update rate
0.01
Feature dim
50
Hidden dim
1024
Optimizer
Adam
Exploration steps
0
DDPG exploration schedule
0.1
Target feature processor update frequency(steps)
20000
Reward scale factor
10
Fixed weight o
0.03
Linear decay schedule for A(TT)
lincar (,0,1.1000))00000)
KOI
Weight of semantic key states A (i N - 1)
0.15
Weight of the last semantic key state Afy.
0.35
Weight of motion key states A
0.05
Standard deviation of semantic key states 0j
10
Standard deviation of motion key states d2
25
Table 3: List hyperparameters.
of performance during online finetuning with any reward estimation method as compared to the pretrained policy.
B.4 Ablation Semantic Decomposition Module
We provide the ablation study of the Semantic Decomposition Module, which select the semantic key states with different methods:
Human Annotation: We manually annotate the key states the expert demonstration.
UVD R3M: Utilizing the pretrained R3M visual model as a visual encoder for task de- composition, we select semantic key states to adjust the importance weights in the expert demonstration.
UVD -ttaiiide:: We utilize the visual encoder from the Behavior Cloning model, which is trained the expert demonstrations, to select semantic key states to adjust the importance weights of the expert demonstration.
The results in Figure demonstrate that our model, designed based on GPT-4v, can fully leverage the comprehension capabilities of large multimodal models, thereby achieving better decomposition effects than pre-trained models and trained sequential models. Additionally, our model achieves
14

---

metaworld bin picking
metaworld button-press
metaworld drawer
1.0
1.0
0.8
0.8
0.8
siccess rate
success rate
success rate
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
4
3
frame
1e5
frame
1e5
frame
1e5
metaworld door unlock
metaworld drawer close
metaworld door open
1.0
1.0
1.0
0.8
0.8
success rate
success rate
success rate
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
frame
frame
frame
1e5
libero put bowl plate
libero put bowl stove
libero put wine cabinet
1.0
1.0
1.0
0.8
M
M
succes Tale
success rate
succoss rate
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.0
0.5
0.5
1.5
1e5
frame
1e5
frame
1e5
RoboCLIP
ROT
(Ours)
Figure 7: The experiment results 6 tasks from Mera-World environments and 3 tasks from LIBERO environments. The shaded region represents standard deviation across 3 seeds.
results comparable to manually selected key frames, while reducing labor costs and enabling easier scalability in future processes.
1.0
metnworid bin picking
metaworld button press
metaworid bin picking
3.0
metaworid butten, press
0.8
0.8
5.8
0.0
0.0
3.0
8.0
fome
Tome
from
frome
trained
Figure 8: The ablation experiment of Semantic Decomposition Module.
B.5 Ablation of Motion Capture Module
We provide the ablation study of the Motion Capture Module (MCM), which selects the motion key states with different methods:
w/o. MCM: This setting only uses states selected by visual-language models to adjust the importance weight of expert demonstrations.
uniform Motion Capture: This setting selects the motion key states every 5 steps within the interval defined by two semantic key states.
15

---

As shown in Figure 9, the uniform motion selection method would choose noisy keyframes, affect- ing the reward estimation. However, our motion capture module can extract useful information to facilitate online exploration.
metaworld bin bin_picking
metaworld button _
1.0
1.0
0.8
0.8
success rate
success rate
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
1
2
3
frame
le5
frame
1e5
w/o. motion
uniform motion
Ours
Figure The ablation experiment of Motion Capture Module.
B.6 Ablation Both Modules
We also provide the ablation study of the Both Module:
w/o. Both: This setting selects the state every 10 timesteps as the key state and utilizes
them adjust the importance weight of expert demonstrations.
w/o. MCM: This setting only uses states selected by visual-language models adjust the importance weight of expert demonstrations.
w/o. SDM: This setting removes the semantic subgoals inferred by visual-language models and selects motion key states within every 10-state intervals.
As shown in Figure 10, the absence of either semantic or motion key states detrimentally affects the efficiency of online exploration, particularly in the complex "bin picking" manipulation task.
Besides, the performance of the "w/o. MCM" setting is comparable to the "w/o. SDM" setting, demonstrating the equal importance of both types of key states.
Additionally, the shaded region for the "w/o. MCM" setting larger than the other methods, indicat- ing that motion key states contribute stabilizing online exploration by providing detailed guidance on "how do" tasks.
Notably, both the Semantic Decomposition Module and Motion Capture Module we proposed sur- pass "w/o. Both" setting in most tasks, demonstrating the effectiveness of each module.
C Real-World Experiments
In this work, conduct 3 real-world robotic manipulation tasks, shown in Figure 1. We conduct our real-world experiments using XARM robot equipped with a Robotiq gripper. To facilitate efficient policy learning, we limit the robot's action space to 4 dimensions (x, y. z, gripper) and utilize both image and proprioceptive inputs for the policy. To ensure the safety of the exploration, we restrict the robot arm's gripper to operate within certain safety range.
For each task, collect 10 human expert demonstrations train the Behavior Cloning (BC) policy during the offline phase and use a single expert demonstration to estimate online imitation reward
16

---

metaworld bin, bei_biiinniiiie picking
metaworld button, press
putton_press
metaworld drawer open
1.0
1.0
success rate
0.8
success rate
0.8
success rate
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
le5
1e5
le5
metaworld door open
door
metaworld
2.0
1.0
1.0
0.8
success rate
SUCCESS rate
success rate
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
1
3
1
3
5
Trame
le5
frame
1e5
frame
1e5
w/o. Both
w/o. MCM
w/o. SDM
KOI (Ours)
Figure 10: The ablation experiment of both modules.
during the online phase. To evaluate the pretrained BC policy, conduct 10 experiments varying initial object positions to calculate the average success rate. We then conduct 20 times of online explorations, recording their average success rates. Finally, we load the updated policy and conduct 10 offline tests to assess its performance.
Offiine: For image data, we resize them 120 160 and extract 128 128-dimensional representations through shallow CNN network. For proprioceptive data, collect the 6D pose information of the robot arm's end-effector, the 7-axis degrees of freedom joint information, and the gripper's ss angle. We concatenate the proprioceptive and then map them to the 128-dimensional representation through a fully connected layer. We train the BC policy nhe with 10 expert demonstrations over 200 epochs and evaluate the performance at 10 various initial object positions.
We load the pretrained policy hhe and interact with environments within 20 times during online exploration. To ensure the safety of the robot, we restrict the operation range of the gripper to prevent it from colliding with the table.
Online:
As mentioned the limitation, the mismatch between pretrained BC policy and initial critic would negatively impact early exploration learning. Consequently, to ensure both safety and efficiency during real-world exploration, we substitute the adaptive function À in Equation 8 with a constant value of 0.9 and provide sparse reward indicate whether the task has been completed. Through this modification, enable the robot to explore the environment stably and develop the exploration policy =².
To ensure the fairness of the experiments, we also set the adaptive weights of ROT in the comparative experiments as the constant value of 0.9.
Test: We load the updated exploration policy nE to evaluate the performance at 10 various initial object positions.
17

---

a)
Behavior Cloning
fail
success
b)
fail
fail
Online Exploring
fail
success
c) Finetuned policy
success
success
Figure 11: Overview of our real-world experiment on "take the rag off" task. KOI significantly accelerates the online finetuning, guiding the agent to complete the task even when objects are placed in different positions.
18

---

Bin picking
Button press
Drawer open
Door open
Drawer
close
Door
unlock
Put
bowl
to
plate
Put
bowl
to stove
Put
wine to
cabinet
Put
tape
in box
Take
rag
off
Put
cube on
scale
Figure 12: Example trajectories for tasks from Meta-World suite, 3 tasks from LIBERO suite, and 3 real robot tasks, sequentially.
19


Reviews:
[{"review_id": "5WLJqqfyu7", "reviewer": "Reviewer_5oNU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a hybrid Key-state guided Online Imitation (KOI) learning approach to improve the efficiency of online imitation learning. The authors claim that this addresses the challenge of limited expert trajectories in extensive exploration spaces. The KOI method decomposes tasks into \"what to do\" and \"how to do\" aspects, utilizing visual-language models to identify semantic key states for the former and optical flow analysis to determine motion key states for the latter. This dual approach enhances task-aware exploration by refining trajectory-matching reward computations. Experiments in Meta-World and LIBERO environments show that KOI significantly boosts sample efficiency compared to previous methods, and its effectiveness is further validated in real-world robotic manipulation tasks.", "summary_of_recommendation": "I believe the authors' idea is interesting. However, the experiments have to be more extensive to provide clear results.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "My primary issue is with the evaluations and I would recommend the authors perform more trials in sim to obtain more reasonable variance bounds and also perform a comparative study with the real robot to strengthen their claims.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "shLBdFTprX", "reviewer": "Reviewer_APBJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel approach called hybrid Key-state guided Online Imitation (KOI) learning to improve online imitation learning. The method addresses the challenge of efficiently exploring large state spaces with limited expert demonstrations by decomposing tasks into \"what to do\" (objectives) and \"how to do\" (mechanisms).", "summary_of_recommendation": "An interesting idea with room to improve on presentation and evaluation of the method", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "It seems like the semantic key states are identified entirely from the language model. How would your approach work in a task where there are multiple plausible ways to accomplish it, but the expert demonstrations select one? Would you still learn effectively in this situation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "drg90iKipO", "reviewer": "Reviewer_raZY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a simple modification to an exisiting online Imitation Learning algorithm, by incorporating task decomposition and subsequent estimation of task aware rewards using Optimal Transport. The task decomposition is achieved by leveraging exisitng vision language models and optical flow etimation techniques. The experiments show how the proposed modification with task decomposition as an inductive bias for learning give superior quantitative and qualitative results over valid baselines.", "summary_of_recommendation": "I recommend for a Weak Accept with a lower confidence.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Is the use of pre-trained vision-language model the only way to achieve this task decomposition. Can a sequential model be explicitly trained to decompose the task in case the pre-trained model fails at this task?\n\n2. What would happen if the tasks are not inherently hierarchical? Let's say the simpler tasks like a cheetah run from dmc, as performed in the original ROT paper?\n\n3. Can't we let a human do the task decomposition instead of the GPT model?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DAR7O3fLNn", "reviewer": "Reviewer_3bAV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel approach called Hybrid Key-state guided Online Imitation (KOI) learning to enhance the efficiency of Online Imitation Learning (OIL). The main idea is to address the challenges of task-aware reward estimation by decomposing tasks into \"what to do\" (semantic key states) and \"how to do\" (motion key states). By integrating these key states, the method refines reward computation, encouraging more efficient online exploration. The KOI method leverages visual-language models and optical flow for key state extraction, which is validated through experiments in simulated environments (Meta-World and LIBERO) and real-world robotic manipulation tasks, demonstrating improved sample efficiency and practical applicability.", "summary_of_recommendation": "I believe this is good idea and I do appreciate the unique use of higher level reasoning to faciliate reward estimation. There are some components that are not clear within the paper to fully convince me of the key contributions and impact of this approach on the online imitation learning setting and how this method can scale/generalise to other tasks.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Can you provide more detailed explanations or examples for the mathematical formulations, especially the OT-based reward estimation and the Importance Weight Adaption?\n2. The paper mentions a reliance on pretrained behavior cloning policies. How significant is this dependency, and what would be the potential impact on early exploration efficiency without such pretraining?\n3. Could you provide more insights or results regarding the choice of variance parameters \\( \\sigma_1 \\) and \\( \\sigma_2 \\) for the Gaussian Mixture Model? How sensitive is the model's performance to these parameters?\n4. Can you give more detailed descriptions of the real-world experiments, including the setup, specific tasks, and any encountered challenges?\n5. How does the KOI method scale with more complex tasks? Any additional insights or experiments on generalization capabilities would be valuable.\n6. How does the semantic decomposition from GPT-4 impact performance. What is the variation in this process? Would something like UVD be more suitable to extend to more complex tasks where the semantic keypoint starts arent as obvious to a vision model? Explain how reliant/robust your method is to the selection of these keystates and comment comment on how general it is to scale to arbitrary tasks.\n7. In figure 3, could you include a graph without both the Semantic Decomposition Module (SDM) and the Motion Capture Module (MCM) to better illustrate their individual contributions?\n8. Can you clarify the representation, dimensionality, and usage of \\( I^s \\) and \\( I^m \\)? How are these terms specifically utilized to derive the reward, and what is the justification for the choice of sigma values?\n9. Can you explain why the KOI method shows slightly higher rewards compared to ROT, and how this result is attributed to task-dependent key state information for reward estimation?\n10. Could you provide experimental results demonstrating how the number of demonstrations used for reward estimation affects downstream task performance with and without the proposed semantic keypoints? What is considered 'limited' in terms of expert demonstrations?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The presented approach to improve online imitation learning is very interesting and generally well presented. However, some design choices could be better motivated and discussed in more detail. The reviewers highlighted several issues that should be addressed to further improve the paper. For instance, additional clarification regarding the large variance in KOI’s performance would improve the discussion of the evaluations. Furthermore, the real-world experiments are not described very well and raise some concerns regarding the claims made. The paper also lacks comparison to other hierarchical or subgoal-based methods.

## Post Rebuttal
The rebuttal addressed all major concerns and further improved the already strong work. The reviewers unanimously agree that this work should be presented at CoRL 2024.
Decision: Accept