AutoCGP: Closed-Loop Concept-Guided Policies from Unlabeled Demonstrations
Authors: Pei Zhou
Ruizhe Liu
Qian Luo
Fan Wang
Yibing Song
Yanchao Yang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Self-Supervised Manipulation Concept Discovery
Concept-Guided Policy for Robotic Tasks
Primary area: applications to robotics, autonomy, planning

Abstract: Training embodied agents to perform complex robotic tasks presents significant challenges due to the entangled factors of task compositionality, environmental diversity, and dynamic changes. In this work, we introduce a novel imitation learning framework to train closed-loop concept-guided policies that enhance long-horizon task performance by leveraging discovered manipulation concepts. Unlike methods that rely on predefined skills and human-annotated labels, our approach allows agents to autonomously abstract manipulation concepts from their proprioceptive states, thereby alleviating misalignment due to ambiguities in human semantics and environmental complexity. Our framework comprises two primary components: an *Automatic Concept Discovery* module that identifies meaningful and consistent manipulation concepts, and a *Concept-Guided Policy Learning* module that effectively utilizes these manipulation concepts for adaptive task execution, including a *Concept Selection Transformer* for concept-based guidance and a *Concept-Guided Policy* for action prediction with the selected concepts. Experiments demonstrate that our approach significantly outperforms baseline methods across a range of tasks and environments, while showcasing emergent consistency in motion patterns associated with the discovered manipulation concepts. Codes are available at: https://github.com/PeiZhou26/AutoCGP.

Published as a conference paper at ICLR 2025
AutoCGP: CLOSED-LOOP CONCEPT-GUIDED POLICIES FROM UNLABELED DEMONSTRATIONS
Pei Zhou Ruizhe Liul.*
Qian Luol55+
Fan Wang
Yibing Song
Yanchao Yang!.2
1 HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong
*Department of Electrical and Electronic Engineering, The University of Hong Kong DDMM Academy, Alibaba Group Huupa Lab TTranscengram
(pezhou, zr11rz360, gianloo hku.hk [fan. w, songyibing bbb.://bbbb......l.bbbbbba-aaaa. inc. com, yanchaoy@hku.
ABSTRACT
Training embodied agents to perform complex robotic tasks presents significant challenges due to the entangled factors of task compositionality. environmental di- versity, and dynamic changes. In this work, we introduce a novel imitation learning framework to train closed-loop concept-guided policies that enhance long-horizon task performance by leveraging discovered manipulation concepts. Unlike methods that rely on predefined skills and human-annotated labels, our approach allows agents to autonomously abstract manipulation concepts from their proprioceptive states, thereby alleviating misalignment due to ambiguities in human semantics and environmental complexity. Our framework comprises two primary components: an Automatic Concept Discovery module that identifies meaningful and consistent manipulation concepts, and a Concept-Guidd Guided Policy Learning module that effec- tively utilizes these manipulation concepts for adaptive task execution, including a Concept Selection Transformer for concept-basel guidance and Concept- Guided Policy for action prediction with the selected concepts. Experiments demonstrate that our approach significantly outperforms baseline methods across range of tasks and environments, while showcasing emergent consistency in motion patterns associated with the discovered manipulation concepts. Codes are available https:: pettps:/ggitae gighhb....... Heimm/PeiZhouui AutoCGP.
INTRODUCTION
The pursuit of developing robotic systems that can perform wide array of tasks in diverse environ- ments is a core mission in embodied AI and robotics. While imitation learning has been employed to derive policies from extensive robotic datasets (Collaboration et al., 2024; Kim et al., 2024), remains challenging for embodied agents to effectively learn policies for long-horizon tasks. Errors and inconsistencies can accumulate, making situations progressively more dynamic and complex to manage. Additionally, imitation learning faces further difficulties due to the varying and unpredictable conditions encountered at each step (Brohan et al., 2022: 2023; Padalkar et al., 2023; Fu et al., 2024).
To tackle long-horizon tasks, it is essential to divide them into sequential sub-tasks with specific goals. This strategy offers two main advantages: first, breaking down a complex task into simpler sub-tasks makes the learning process more managcable by focusing overall complexity and enhancing feasibility. Second, these sub-tasks often involve common actions, such "grasping," which are frequently repeated in various tasks. By learning these common skills, the training dataset more effectively utilized, allowing for reuse in new, unseen tasks. Therefore, beneficial learn representation that captures the intrinsic patterns of these common actions, referred to as manipulation concepts (Shao et al., 2021; Liu et al., 2024a; Jia et al., 2024; Zhou & Yang, 2024). These concepts analogous human motor skills like grasping, throwing, pushing, and pulling (see Fig. 1). Just as a set of basic skills can facilitate humans to perform complicated tasks such as cooking or assembling furniture, the learned manipulation concepts can also be composed or reused to address complex long-horizon robotic manipulation tasks.
attainable goals, thereby reducing
First Authors

---

Published as a conference paper at ICLR 2025
Task: "Prepare coffee using the coffee maker."
Human Annotation
Self- Self-Supervised Discovery
'Grasp"
"Transit"
"Push"
R
R
Manipulation Concepts
Human
Machine
Step1:
"6resp"
"Gnasp"
Stepl:
Step2: "Transit"
Shop2:
"Trnsii*
StepN:
"Push"
Effector
StepN:
Effector
"Pusk""
Using Manipulation Concepts for Action
Figure 1: Comparison of human defined manipulation concepts with those derived by method. Human manipulation concepts, or motor skills, can be described in natural language and combined for long-horizon tasks. In contrast, method applies a self-supervised learning framework to discover meaningful manipulation concepts as feature vectors (see 3. 3.1). These manipulation concepts can be applied manipulation tasks via composition. For humans, manipulation concepts inform planning, which guides specific actions. Similarly, our model execute policies guided by the discovered manipulation concepts (see Sec. 3.2).
Many prior works have relied human semanties, e.g., using natural language, to label sub-processes and enhance imitation learning by training hierarchical policies. These approaches often depend on manually crafted annotations, heuristic rules, or large models trained for encoding human prior (Mo et al., 2019; Ahn et al., 2022; Eisner et al., 2022: Li et al., 2023c; Di Palo et al., 2023; Ding et al., 2023; Lin et al., 2023). This reliance poses significant challenges: the extensive manual effort required for data collection is time-consuming and resource-intente to manipulation concepts that do align well with the robot's configuration or its operational environment, ultimately hindering effective learning and performance.
and the inherent subjectivity may lead
To address these challenges of manual annotation, task complexity, and environmental unpredictabilityy we propose novel framework that combines automatic concept discovery with closed-loop concept- guided policy learning. Our approach autonomously extracts and utilizes manipulation concepts directly from the robot's proprioceptive states (Please refer to Sec. A.l for the rationale behind using proprioceptive states), thereby eliminating the dependency on predefined skills and cuman-annotater labels. This self-supervised method not only mitigates the misalignment caused by ambiguities in human semantics but also adapts dynamically to unforeseen situations.
Specifically, framework consists of two main components. The first is the Automatic Concept Discovery module, which enables robots to derive manipulation concepts in a bottom-up manner without resorting human annotators. This module focuses on key characteristies shared by wide range of skills and sub-goals, allowing for the abstraction of high-quality and consistent manipulation concepts across various tasks. The second component is the Closed-Loop Concept- Guided Policy Learning module. This module establishes an adaptive policy by employing Concept Selection Transformer (CST) propose and adjust manipulation concepts in real-time during the robot's interactions with the environment. This dynamic selection enables the system to adapt to environmental changes and select the most appropriate concept for the current situation. Subsequently, the Concept- Guided Policy (CGP) utilizes the selected manipulation concepts predict actions based on instantancous visual inputs, leveraging a diffusion policy to generate efficient and high- dimensional actions. This closed- approach ensures that the policy can dynamically adjust and refine its actions based on continuous feedback, effectively addressing the challenges of dynamic and unpredictable environments.
The experimental results demonstrate that our pipeline significantly improves manipulation task performance, surpassing baseline methods across variety of tasks and environments. We observe
2

---

Published as a conference paper at ICLR 2025
emergent consistency within these manipulation concepts, particularly in terms of motion patterns. Our contributions can be categorized into three major aspects: 1) A novel closed-loop concept-guidee learning pipeline that discovers and utilizes manipulation concepts from unlabeled demonstrations; 2) A manipulation concept discovery method that abstracts high-quality, consistent manipulation concepts in various tasks, reducing dependence on human prior knowledge; and 3) A concept- guided policy that incorporates learned manipulation concepts with diffusion policy, achieving state-of-the-art performance on various benchmarks.
2 RELATED WORK
Concept discovery in manipulation tasks. The manipulation concepts we propose are discrete symbolic representations, akin to natural language and codes, which are widely used and beneficial for learning manipulation tasks. Despite advances in improving manipulation tasks through costly manual labeling (Mo et al., 2019; Eisner et al., 2022; Li et al., 2023c), unsupervised concept discovery remains under-explored. Recent advances in Large Language Models (LLMs) show promise for automated labeling (Ahn et al., 2022: Di Palo et al., 2023; Ding et al., 2023; Lin et al., 2023), but they lack sufficient grounding in real-world sensor data (Zhou Yang. 2024). Studies adopting self-supervised methods to discover manipulation concepts (Sermanet et al., 2018; Yan et al., 2020; Zambelli et al., 2021; Morgan et al., 2021; von Hartz et al., 2022; Weng et al., 2023; Liu et al., 2024a) include utilizing information-theoreds
tools such as mutual information (Gregor et al., 2016; Hausman et al., 2018; Hu et al., 2024) and modeling temporal relationships and latent features using approaches like cime-contrastive learning (Nair et al., 2022; Ma et al., 2023). identifying critical temporal junctures (Jayaraman et al., 2018; Neitz et al., 2018; Pertsch et al., 2020; Zhu et al., 2022; Caldarelli et al., 2022), and predicting future time-steps (Starke et al., 2022; Li et al., 2024). Many of these methods also integrate geometric (Morgan et al., 2021; Zhu et al., 2022; Shi et al., 2023) and physical (Yan et al., 2020) constraints. Several approaches struggle to generalize due to data inefficiency. heuristic designs and reliance on predefined primitives tailored to specific tasks (Wang et al., 2024), while others rely on overly broad self-supervised principles that lack focus on the intended concept use cases. Recent methods like BeT (Shafiullah et al., 2022) and VQ-BeT (Lee et al., 2024) utilize self-supervised learning to derive discrete symbols from actions. However, their focus remains on mitigating continuous noise in neural network outputs through action discretization rather than on capturing higher-level abstractions, such as motor skills. The proposed discovery mechanism efficiently manages generalization across robots with the same morphology while capturing key features of diverse manipulation skills beyond the original action encoding.
Manipulation policy learning. A significant body of research is devoted to learning manipulation policies through methods like imitation learning (Argall et al., 2009; Rahmatizadeh et al., 2018; Zhang et al., 2018; 8; Fang al., 2019) and reinforcement learning. These approaches primarily use deep neural networks to map states to actions, enabling interaction. Key techniques in this field include the Decision Transformer (Chen al.. 2021; Team et al., 2023; Xu et al., 2023b; Zhao et al., 2023; Tanaka et al., 2024) and Diffusion Policy (Wang et al., 2022; Chi et al., 2023; Li et al., 2023b: Pearce et al., 2023: Liu et al., 2024b; Tan et al., 2024; Yan et al., 2024; Ze et al., 2024). To address more complex tasks, recent advancements have integrated self-supervised skill extraction (Peng et al., 2022: Li et al., 2023a) and hierarchical planning with concepts (Xu et al., 2018; Yang Bateebaut-Busssse et al., 2022; Jia et al., 2023; Liang et al., 2023) into policy learning,
et al., 2022: or a combination of them (Peng et al., 2022: Wan et al., 2024). Various methods have also been proposed to handle multiple manipulation concepts within a single model (Brohan et al., 2022; 2023; Driess et al., 2023). Additionally, Large Language Models (LLMs) are leveraged for the massive prior knowledge encoded to manage diverse situations in manipulation tasks (Huang et al., 2023a;b; Long et al., 2023; Wang et al., 2023; Wong et al., 2023; Xie et al., 2023; Yu et al., 2023; Izzo et al., 2024; Zhou et al., 2024). However, these methods often struggle with real-world tasks due to the lack of physical grounding. Our approach overcomes this by deriving manipulation concepts directly from robot observations and proprioceptive states, automatically ensuring physical grounding in an end-to-end manner.
METHOD
We aim to develop closed-loop soncept-guided policies, which output action by taking in a manipula- tion concept (e.g., indicating motor skill) and the current state. The manipulation concept in effect
3

---

Published as a conference paper at ICLR 2025
Automatic Concept Discovery
Concept-Guided Policy Learning
Manipulation Concepts Demo
inputs
Closed-Loop
Manipulation Concept Assignment
Mani
Con.
Daug
Sp
01
d2
0g
CST
Interaction
labels
Refine Assignment
Self -Spervised Learning
Demos D
Detection
Evalustion
Corsolidation
C1
Q2
CEg
CGP
Figure The proposed pipeline for Learning Closed-Loop Concept-Guided Policies from unlabeled demonstrations. The Automatic Concept Discovery module labels demonstrations in D with manipulation concepts using three strategies (see Sec. 3.1 and Fig. 3). These strategies form self-supervised learning process, refining the assignment of manipulation concepts with E. The labeled demonstrations form Daug providing observations (o, and st) and labels (at) for Concept-Guided Policy Learning. which trains the Closed-Loop Policy comprising CST and CGP (see Sec. 3.2). In this setup, CST selects the appropriate manipulation concept (ax) to guide CGP in generating actions.
is adjusted according to the task and the current situation (i.e., closed-loop). Moreover, these manipu- lation concepts are discovered self-supervised manner without resorting to human annotations. Specifically, our pipeline consists of two key modules as shown in Fig. 2. The first module, Automatic Concept Discovery, detects segments of similar motion patterns from unannotated demonstrations to derive manipulation concepts. The second module, Concept- Concept-Guided Guided Policy Learning, leverages these manipulation concepts to train policy that adapts to environmental changes by selecting the most appropriate manipulation concept (Concept Selection Transformer (CST)) and predicting actions with the proposed manipulation concept (Concept-Guude Guided Policy (CGP)).
3.1 AUTOMATIC CONCEPT DISCOVERY
Given unannotated demonstrations D, where T E D is a trajectory, i.e., T {(sT, of. a[) of the environmental observation, of the agent's proprioceptive state, a, the action at time-step 4, and T(r) the length of r), the Automatic Concept Discovery (ACD) module aims to detect manipulation concepts by assigning each state trajectory to one of the K learnable embeddings A {ax) { /= We denote K { assignment, we instantiate an encoder E, such that for every T E D, & maps its state at f to an index kj E K. corresponding to the embedding of. Let Og be the parameters of & (excluding K. and A),
T(+) (with
the index set for the manipulation concepts. Specifically, to perform the
then we have:
k E(,
af okj
(1)
Note that sequences sharing an index & I. belong to the same manipulation concept represented by a₃, depicting the behaviors initiated by it. Additionally, the assignment encoder & prioritizes the proprioceptive information to facilitate deriving coherent manipulation concepts, since proprioceptive motion sequences exhibit high consistency across tasks and different data types.
The structure of (Fq. 1) adapted from the VQ-VAF VAF architecture (Van Den Oord al., 2017), which selects index from AC {K]K the manipulation concept for time step t in a demonstra- tion т. This selection matches the feature vector generated by the encoder to the nearest embeddings in A {ax [k 1
K = E(t); = arg min [ē(t)r: - axll. x. =0k]
(2)
kek. Here, & is the sub-module that takes the demonstration E D input and outputs feature of the same dimension as the approximation of the embedding vectors A at each time step 4. The discrete nature of the VQ-VAE encoder, discussed in the following sections, naturally enables the segmentation of sub-trajectorics that achieve sub-goals represented by the same manipulation concepts, facilitating the identification of the moment when the sub-goal is accomplished during training. Please refer to Sec. A.2 for a detailed description on the implementation of E, especially for the preservation of gradients.
Goal State Detection. Given the current state, a manipulation concept or sub-goal enables the envisioning of the future state upon its accomplishment. To encourage the discovery of meaningful
4

---

Published as a conference paper at ICLR 2025
Goal State Detection
Goal State Evaluation
k peoo-proces Manipulation Concept
k'
km Sub-process of Manipulation Concept
K
Goal state !
Goal state
G prediction
k
Goal:
Goal Consolidation
Manipulation Concept
Emb.
Gool shate
Seq.
Reconstructor R
V,V(s, gi ax)
-0
-0
Relatively higher value
Whole Demo (Proprioceptive States)
Guide Transition
Value of v at States
Figure 3: Automatic Concept Discovery Strategies. The Automatic Concept Discovery (ACD) module employs three key strategies. The Goal State Detection strategy maps each state, along with its manipulation concept (embedding), to the goal state of its sub-process, defined at the junction between segments with different manipulation concepts (k and k', see Eq. 4). The Goal State Evaluation strategy uses the value function V to evaluate both the Alignment and the Completion of proprioceptive state based on the manipulation concept and goal state, Alignment indicated by the value difference between states within and outside specific manipulation concept segments. Completion derived from the gradient of V, guiding state transitions. The Goal Consolidation strategy maps manipulation concept embeddings (i.e., Emb.) back to their proprioceptive states.
manipulation concepts, we leverage the above phenomenon strategy, which learns by training network G that maps each state of with the putatively associated manipulation concept embedding to the goal state (ending state) of the corresponding sub-sequence. This allows manipulation concept embeddings (A)) and the assignment encoder (E) to be refined through back proo-propagation. The objective is the loss between the goal states predicted by and those defined by the partitioning of the manipulation concepts:
design a Goal State Detection
Cad |1((1 of (TT (of TT
(3)
Here of is computed according to Eq. and gf denotes the proprioceptive state at which the sub sub-goal for time-step t in is achieved. Leveraging the discrete embeddings in E, once indices from IC are assigned as manipulation concepts to the states demonstration т, the state at the sub-goal's accomplishment be detected retrieving the nearest time step where the manipulation concept switches, indicating the completion of the current sub-goal:
gi arg min, (wou2t,k (cx AkT)
(4)
Please refer Fig. for an illustration of the Goal State Detection procedure.
Goal State Evaluation. Given a state depicting when the sub-goal indicated by a manipulation concept achieved,
assess whether state is well-aligned towards achieving the sub-goal, and can further evaluate the progress toward its completion at the current state. To further enhance the quality of the discovered manipulation concepts, we develop Goal State Evaluation strategy. This involves using the assigned manipulation concepts to train a value function >/ that evaluates both the alignment and the completion degrees for proprioceptive state conditioning on the specific manipulation concept and its associated goal state:
Vs 1 gh. a4) + [0, 1J, where ag E A.
(5)
detailed implementation Eq. can be found in Sec. A.3, and V, in theory, shall differentiate the compatibilities of state against various manipulation concepts by their values. States compatible with the manipulation concept ag that aiming for the goal state g,, should be assigned higher values, while others with lower values, due either mismatch of the manipulation concept resulting
5

---

Published as a conference paper at ICLR 2025
in the goal state (Goal State Evaluation in Fig. 3). This discriminative ability (for alignment) of V could be acquired by training with the following binary classification objective':
{ {
[loo log V (sT,9]saa)
i k and g 94
(s) i
log (1 -- (sT (l:)))
otherwise.
(6)
= =
= T ==1 [[K + k M c (st
However, the compatibility learned according to the above may not faithfully assess the completion degree of a state regarding the sub-goal state, Eq. 6 does not explicitly enforce that higher score represents greater progress towards the sub-goal state, thus the ambiguities in the evaluation.
To address this issue, further constrain the gradient field of V to enforce accurate completeness evaluation for more informative manipulation concepts. We propose that the progress toward completion should be correlated with the increase decrease in V's value, which can be effectively captured through its gradients. Moreover, when of and 0g are fixed in Eq. 5, the gradient field with respect the current state s;, while ag is in effect, should guide the state transition, represented by Vss (Goal State Evaluation in Fig. 3). Thus, we train a network II, which maps the current state and its gradient to this state transition follows:
(T() II ( V.; of: af) C
where the state transition VsT can be computed as the discrepancy between the states from two consecutive time steps, i.e., finite difference.
Goal Consolidation. Furthermore, propose that all states associated with given manipulation concept shall exhibit similarity to some extent, as they share the same sub-goal or skill. To utilize this characteristic, introduce reconstruction network R trained to map the embedding vector back to its corresponding state:
_ (ir(())) - R (oq)Tr))
8
We denote this as a consolidation aimed at promoting the consistency of sub- sequences assigned with the same manipulation concept, ensuring that similar states serve similar purposes.
Together, Eq. 9 constitute the training objective for the self- discovery of manipulation concepts from unannotated demonstrations:
LCD
Led + L + Aent (E + LV) + AgaLE
Here L the combination the commitment loss and vector quantization loss in Van Den Oord al. (2017), and Aent and Age are positive hyperparameters. Next, we describe the learning of closed-loop concept-guided policies with the discovered concepts.
3.2 CLOSED-LOOP CONCEPT-GUIDEI POLICY LEARNING
With the self-supervised discovered manipulation concepts, we augment the unannotated demonstra- tions D (T ((5], of, az) (r)) by tagging each state in a demonstration with manipulation concept kj E K. resulting in the augmented dataset Daug = {T = {(8T oj, a7 k)) With
T())
Daug, we can utilize the discovered and grounded manipulation concepts K. to learn closed-loop visuomotor policy:
~ pcs Occs))
(10)
aploo, 8, k; Ocup)
where OCST the parameters of PCST and OcGP the parameters of wD. The proposed framework enables the policy to dynamically select and adjust the manipulation concept in response to envi- rommental changes with the Concept Selection Transformer (CST): k PCST (k o, s; Ocsr), and to execute appropriate actions based on real-time visual inputs and corresponding manipulation concepts through the Cenneptt-uuider Policy (CGP): a ( 8, k; OcGP) the following.
The loss L in Fq. 6 focuses on the time step when the manipulation concept ID switches, similar to Eq. 4. Once the switch is determined, the binary classification objective is applicd every time step of the trajectory.

---

Published as a conference paper at ICLR 2025
Concept Selection Transformer (CST), The Concept Selection Transformer (CST) determines the manipulation concept based on run run-time observations: K, ~ (est(klor, 82), where o and &t denote the observation and proprioceptive state, respectively, and k, is the selected manipulation concept from K. This design allows adjustments at any time step, enabling the system to adapt to environmental changes and select the most appropriate concept for the current situation. The training leverages cross-entropp loss to align the predicted probability distribution over manipulation concepts with the one-hot labels corresponding to the manipulation concepts provided by &:
LEST = -E---, Post (k = kj 10,, sT: OCSt).
(11)
Concept-Guided Policy (CGP). The Sencept-Guided Policy (CGP) TD utilizes manipulation con- cepts predicted by PCST within imitation learning framework. We adopt the Diffusion Policy (Chi et al., 2023) for its ability to generate complex, high-dimensional actions from learned distributions. Given varying stages and sub-goals in manipulation process, CGP adjusts itself with respect the conditional manipulation concept provided by the CST module. Since the subtrrajcctories labeled with specific manipulation concept are always shorter than the original 7, the learning of the skills indicated by these manipulation concepts is usually more efficient due to reduced complexity and can lead to improved performance due to compositional generalization.
Specifically, we leverage diffusion policy TD conditioned on the observed environment, the robot's s proprioceptive state, and the selected manipulation concept. It then denoises the action from a random sample drawn from (O, I) using DDPM framework with the denoising module 'D (Eq. 13 and Eq. 14). As stated in Chi et al. (2023), this approach yields actions that are more robust to multiple viable options and dynamic environments compared to other methods. To enhance the coupling between PCST and TD. we utilize the entire manipulation concept distribution T (o,, $t; OCSt) predicted by PEST rather than simply selecting the manipulation concept with the highest probability:
ag TD (alor, St, Tor, 81; OCsT); Ocar),
(12)
where T(o,, s6; OCST) [PCST (klo,, sy OCST)l E R
This enables the joint training of PCST and TD. allowing PCST to refine its manipulation concept (distribution) prediction with not only the augmented manipulation concepts from Automatic Concept Discovery, but also with the feedback from training TD using the action prediction loss.
More precisely, for the DDPM training of the denoising module with demonstrations T E Daug- we minimize the prediction loss between the injected random noise ((n) (for time step n) and the predicted noise from the noisy action (_(")) where at is perturbed by e(*). The DDPM loss is jointly optimized with the loss for the Concept Selection Transformer in Eq. 11 to enhance Pest's 's selection of manipulation concepts and its coordinated behavior with Tp!
LCGP E |DCc] (*) ("),, sT T(o,, sT OcST): Ocap)
(13)
Cpalicy = LCGP + XLeST
Here, Lest is the loss for training Pesr (Eq. 11), and À > 0 is the hyper-parameter for weighting. For action inference, incrementally refines the generated actions, beginning with random Gaussian noise, by applying a reverse diffusion process, resulting in at as the sampled (predicted) action:
TD : (a- 3,, (a." "(e""""") (5c. Or, T(o,, st θesT): 0csr)) + (0, (14)
Leveraging the diffusion model's strong capability to model complex distributions, TD can produce actions of higher quality when conditioned on the selected manipulation concept. For the detailed architecture and training pipeline, please refer to Sec. B.
EXPERIMENTS
We evaluate the proposed pipeline for Automatic Concept Discovery and Closed-Loop Concept- Guided Policy learning various manipulation tasks. Our primary evaluation compares the per- formance of our method with major baseline approaches across diverse tasks within the Robosuite simulation environment (Zhu et al., 2020). Moreover, we conduct ablation study analyze the
7

---

Published as a conference paper at ICLR 2025
Table 1: Success rates of policies trained with the proposed method and competing baselines across diverse manipulation tasks. DP represents the standard diffusion policy, while all other methods with "+DP" build on this framework by incorporating various concept discovery approaches.
Cof.
Ham.
Stk. 3
3 Pc.
Thd.
Mug.
Do
D1
D2
Do
D1
Du
D1
Do
D1
D2
Do
D1
Do
D1
BeT
0.66 0.52
0.42
0.88
0.36
0.60
0.32
0.42
0.14
0.00
0.34
0.18
0.34
0.26
DP
0.80
0.58
0.40
1.00
0.46
0.68
0.68
0.54
0.28
0.00
0.60
0.28
0.76
0.28
InfoCon+DP
0.90
0.70
0.58
0.96
0.58
0.62
0.52
0.66
0.22
0.02
0.72
0.16
0.78
0.36
Xskill+DP
0.88
0.74
0.50
1.00
0.54
0.72
0.76
0.74
0.36
0.00
0.70
0.20
0.70
0.40
AWE+DP
0.86 0.68
0.48
0.96
0.56
0.70
0.68
0.60
0.26
0.00
0.62
0.16
0.76
0.30
Ours+DP
0.98 0.84
0.72
1.00
0.66
0.78
0.72
0.82
0.42
0.04
0.80
0.28
0.88
0.50
contribations of the different components within our manipulation concept discovery and policy learning modules. We demonstrate the advantages of the discovered manipulation concepts and validate the efficacy of the conditional diffusion policy in improving task performance under the closed-loop framework.
Implementation details. We evaluate our method on tabletop manipulation tasks as described in MimicGen (Mandlekar et al., 2023), detailed in Sec. C.I. These tasks are categorized into six types: Coffee Making (Cof.), Hammer Cleanup (Ham.), Stack Three Cubes (Stk. 3), Three Piece Assembly (3 Pc.), Threading (Thd.), and Mug Cleanup (Mug.). Each task is associated with several levels of environmental variability and noise, denoted as Do. D1, and D2. where larger subscript indicates greater variability and higher noise levels. For each task and its corresponding level of variation, we select 950 demonstrations provided by MimicGen for Imitation Learning. The training samples include front-view images, wrist-view images, robot proprioceptive state, and ground-truth actions at each timestep. We follow the procedures described in Sec. to train concept- concept-guided policies (detailed in Sec. B): 1) Automatic Concept Discovery: First, we apply our framework's 's Automatic Concept Discovery method across six task types, using only the 950 demonstrations with the highest level of variation for each task (e.g.. D2 for Cof.). Task type information is omitted in this process. We train on mixed demonstrations from all tasks, and then label all demonstrations and their respective levels of initial variation with the encoder & in Eq. 1. 2) CST and CGP: For each task, we use Concept Selection Transformer (CST) to map images, and robot state to the relevant execution concepts. Concurrently, a Concept- Concept-Guided Policy (CGP) mentioned in Sec. 3.2 trained to map the CST inputs and manipulation concepts into actions. These two processes are jointly trained to enhance synergy and performance.
front-view images, wrist-view
Baselines. We first train the baseline Diffusion Policy (DP) without leveraging manipulation concepts. We then compare our method with other major concept discovery baselines (implementation details in Sec. A.5). Here is a brief summary: 1) InfoCon (Liu et al., 2024a): InfoCon is a self- supervised framework that discovers manipulation concepts through generative and discriminative informativeness regarding the low-level physical state and state changes. 2) XSkill (Xu et al., 2023a): XSkill is an imitation learning framework that leverages self-contrastive learning transfer manipulation skills from unlabeled human and robot videos. In this case, we reimplement XSkill using only robotic demonstrations. 3) AWE (Shi al., 2023): AWE enhances robotic imitation learning by heuristically extracting minimal waypoints from demonstrations, streamlining the decision-malee making process by approximating entire trajectories through linear interpolation. We also include BeT (Shafiullah et al., 2022) as a baseline since it employs learnable discrete representations similar to manipulation concepts.
extract and
4.1 QUANTITATIVE RESULTS
Evaluation. Our primary evaluation metric is the task success rate (SR), which varies based on the specific criteria for each task (see Sec. C. for details). For instance, in the Coffee Making task, success is defined as accurately grasping the coffee and placing it into the coffee machine, while in the Mug Cleanup task, success involves moving the mug from the table into a drawer. The observation space for all tasks includes two images (each 112 x 112) from the front and wrist views, along with 9-dimensional proprioceptive information of the agent (gripper position, rotation, and opening state). During evaluations, the environment is initialized randomly. We conduct tests over 50 episodes
8

---

Published as a conference paper at ICLR 2025
Table 2: Ablation study of our proposed Automatic Concept Discovery module. We show the success
rates of ablations of different components in the module.
Cof.
Ham.
Stk. 3
3 Pc.
Thd.
Mug.
Do
D1
D2
Do
D1
Do
D1
Do
D1
Do
Do
D1
w/o GD
0.92
0.80 0.58
0.98
0.58
0.70
0.68
0.72 0.38
0.72
0.88
0.48
w/o GE
0.92 0.78
0.58
1.00
0.64
0.72
0.64
0.80
0.40
0.78
0.78
0.40
w/o GC
0.96 0.84
0.60
0.94
0.56
0.68
0.68
0.74
0.38
0.80
0.72
0.46
Ours 0.98 0.84 0.72
1.00
0.66
0.78
0.72
0.82
0.42
0.80
0.88
0.50
for each task using a set of fixed random seeds for fair comparison. Given the complexity of these multi-step manipulation tasks, we set a maximum of 600 rollout steps per episode for evaluation.
Main results. Tab. presents the task success rates (SR) of the trained policies across 50 randomly initialized environments, with variations in object positions and placement angles. We use SR to evaluate the quality of the discovered concepts, since higher success rate indicates that the manipulation concepts learned by the model can guide the policy towards task completion more effectively. The results demonstrate that our proposed method consistently outperforms DP baseline and other concept discovery pipelines on the majority of tasks. In particular, for relatively challenging tasks like the CD2. task and the (DI) task, the performance improvement is even more pronounced, indicating that our method excels handling more complex, long-horizon tasks. This suggests that our approach is highly effective at discovering intrinsically meaningful manipulation concepts that enhance the learned policies' ability to generalize across dynamic environments during robotic execution. The discovery of these meaningful concepts highlights the practical benefits of the proposed closed-loop framework in solving manipulation tasks with high variability and complexity.
Ablation study. We conduct ablation study to assess the impact of various components within our concept discovery and policy learning pipeline, with results detailed in Tab. 2. As described in Sec. 3.1, the Automatic Concept Discovery module consists of three primary components. This ablation involves systematically disabling each component to evaluate their individual contributions to concept discovery and subsequent policy training. The results, as shown in Tab, 2. demonstrate that omitting any of these components adversely affects performance across tasks. Specific settings tested include disabling goal state detection (w/o GD), goal state evaluation (w/o GE), and goal consolidation (w/o GC). The table illustrates that policies trained with fully integrated concept discovery module achieve the best performance across variety of tasks, underscoring the importance of each component in enhancing policy performance with improved manipulation concept quality.
4.2 VISUAL RESULTS
We also visualize and assess the concepts discovered by our method. First, we demonstrate the consistency of the concepts identified across different manipulation trajectories of the same task (Coffee D2), as illustrated in Fig. 4 (upper). The results highlight how well our method generalizes the identified concepts within variations of the same task. We also compare our results with those from XSkill (Xu et al., 2023a), shown in Fig. 4 (lower). For XSkill embeddings, we utilize K- means clustering to group the embeddings into separate concepts, allowing for a direct comparison between the two methods. This comparison underscores key advantage of our approach: our method consistently identifies similar, robust manipulation concepts across different trajectories of the same task, even when there are variations during execution. This suggests that our framework effectively captures the underlying motion pattern of the task. In contrast, the concepts identified by the XSkill pipeline exhibit greater variability and prone to noise. Furthermore, Fig. 5 illustrates that sub-sequences corresponding to the same manipulation concepts exhibit similarity in proprioceptive motion, even when these sub-sequences are derived from different tasks. This suggests that the discovered concepts capture underlying commonalities in dynamics, regardless of the task context. These similarities highlight the generalizability and robustness of the concepts, making them transferable across various tasks. Please see Sec. C.3 for more visualizations.
5 DISCUSSION
We develop a closed-loop concept- guided manipulation policy learning pipeline with two modules: Automatic Concept Discovery and Sencept-Guider Policy Learning, which abstracts manipulation concepts from unannotated robot demonstrations and uses them to improve manipulation policy

---

Published as a conference paper at ICLR 2025
efficacy in automatic manner. Our findings highlight the significance of using autonomously discovered manipulation concepts to enhance policy training in complex manipulation tasks. However, key limitation is the predefined maximum number of concepts due to the codebook structure. Additionally, this pipeline has not yet been tested on more complex morphology. e.g- humanoid. Future research could focus improving generalization across diverse real-world scenarios and developing automated methods to assess concept consistency, which is further discussed in Sec. D.
F97
F100
F102
F118
F139
F159
F171
F231
F248
Traj
Ours
017
elD
u15
x25
al
<10
e15
09
F134
F248
Xskill
64 al aZ 08
al
e3 aB
Traj
Figure 4: Visual comparison of manipulation concepts discovered by method versus Xskill (Xu et al., 2023a). F(number) denotes the frame index of an image within trajectory. Our method demonstrates significantly greater consistency across different trajectories compared to Xskill.
Cof.
Ham.
Mug.
Mug.
Stk.3
Stk.3
Concept #2
Thd.
Thd.
Ham.
Pc.
Mug.
Pc.
Concept #4
Concept #10
Concept #8
Figure 5: Consistency check of the concepts discovered by our method across different tasks. The sub-sequences under the same concept from different tasks share similarities motion. Concept #4: robot arms are placing the objects in target locations. Concept #8: robot arms are moving toward the objects and preparing to pick. Concept #10: robot arms are pulling the drawer handles.
10

---

Published as a conference paper at ICLR 2025
This work is supported by the Early Career Scheme of the Research Grants Council (grant # 27207224), the HKU-100 Award, a donation from the Musketeers Foundation, and DAMO Academy through the Alibaba Innovative Research Program.
Acknowledgement:
Ethics Statement: Our research explores concepts that may go beyond common semantics. For now, we are concentrating on extracting concepts from basic manipulation tasks and will maintain this focus. While the algorithm holds potential for advancing toward more complex concepts, we intend to proceed cautiously, carefully considering ethical implications and the need for responsible control mechanisms.
Reproducibility Statement: We provide our implementation details of our Manipulation Concept Discovery (including baselines) and Concept-Guidet policies in the appendix (Sec. A and Sec. B) and the supplementary material. Our experiments use the environments based on simulator Robosuite (Zhu al., 2020). We welcome discussion and advice on better and more stable performance of our pipeline and further implementation improvement in more practical environments.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do can, not as say: Grounding language in robotic affordances. arXiv preprint 1:Xiv:2204.01699 2022.
Brenna D Argall, Sonia Chemova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-45) 2009.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-languee language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Edoardo Caldarelli, Philippe Wenk, Stefan Bauer, and Andreas Krause. Adaptive gaussian process change point detection. In International Conference on Machine Learning, pp. 2542-2571. PMLR, 2022.
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypernetworks in deep learning, Artificial Intelligence Review, 57(9):250, 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-15097, 2021.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137,, 2023.
Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acor Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben
Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le. Chelsea Finn, Chen Wang, Chenfeng Xu. Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak
11

---

Published as a conference paper at ICLR 2025
Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang. Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Iljja Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo. Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério. Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black. Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala. Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti,, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim. Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart' inn'innMaat'in in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu. Tianli Ding, Todor Davehev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang. Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh- Hua Wu, Yujin Tang, Yuke Zhu, Yunehu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open x-embodiment: Robotic learning datasets and rt-x models, 2024. URL https: / /tt.xiivvv/////////////a/////ava org/ alss 2210 08864.
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller. Towards unified agent with foundation models. arXiv preprint arXiv:2307.09668 2023.
Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2086-2092. IEEE, 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.04388, 2022.
Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3:362-369, 2019.
12

---

Published as a conference paper at ICLR 2025
Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference on Robot Learning (CoRL), 2024.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.075300 2016.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
Zhengyang Hu, Song Kang, Qunsong Zeng, Kaibin Huang, and Yanchao Yang. Infonet: neural estimation of mutual information without test-time optimization. In Proceedings of the 41st International Conference on Machine Learning, pp. 19283-19300 2024.
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping
instructions to robotic actions with large language model. arXiv preprint arXiv:2305. 1176, 2023a.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023b.
Matthias Hutsebaut-Buysee Kevin Mets, and Steven Latré. Hierarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1):172-221, 2022.
Riccardo Andrea Izzo, Gianluca Bardaro, and Matteo Matteucci. Btgenbot: Behavior tree generation for robotic tasks with lightweight llms. arXiv preprint arXiv:2403. 127611, 2024.
Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine Time-agnostic prediction: Predicting predictable video frames. arXiv preprint arXiv: 2018.
Zhiwei Jia, Fangchen Liu, Vineet Thumuluri, Linghao Chen, Zhiao Huang, and Hao Su. Chain-of- thought predictive control. arXiv preprint 2307022304.00777, 2023.
Zhiwei Jia, Vineet Thumuluri, Fangchen Liu, Linghao Chen, Zhiao Huang, and Hao Su. Chain- of-thought predictive control. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 21768-21790, PMLR, 21-27 Jul 2024. URL https: htttps://proceeeeee http://pttccee...ee/e/// mlr. press / 1iw.4c..2hhha.... html.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burch- fiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source lanionlanguage-actte model, 2024. URL htttt:::///add..or/bbbb///////// 09246.
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03/18 2024.
Chenhao Li, Sebastian Blaes, Pavel Kolev, Marin Vlastelica, Jonas Frey, and Georg Martius. Versatile skill control via self-supervised adversarial imitation of unlabeled mixed motions. In 2023 IEEE international conference robotics and automation (ICRA), pp. 2944-2950 IEEE, 2023a.
Chenhao Li, Elijah Stanger-Jone Jonee, Steve Heim, and Sang bae Kim. FLD: Fourier latent dynamics for structured motion representation and learning. In The Twelfth International Conference Learning Representations, 2024. URL https: : / openreeeeeeee ew net/ / forum? fobrummid=xeee d=xsd21 1WYSA
Xiang Li, Varun Belagali, Jinghuan Shang, and Michael S Ryoo. Crossway diffusion: Improving diffusion-baste visuomotor policy via self-supervised learning. arXiv preprint arXiv:2307.01844 2023b.
13

---

Published as a conference paper at ICLR 2025
Xiaoqi Li, Mingxu Zhang. Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16217, 2023c.
Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldif- fuser: Interpretable hierarchical planning via skill abstractions diffusion-baan task execution. arXiv preprint arXiv:2312.1 / 1598, 2023.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Tex(2otioon From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345-1365, 2023.
Ruizhe Liu, Qian Luo, and Yanchao Yang. Infocon: Concept discovery with generative and discrim- inative informativeness, The Twelfth International Conference on Learning Representations, 2024a. URL https::///// httpps:///pppeeeeeeeee openreviewwwweeeeeeeeeeeeeeeeewe net www.unniddeeeeeeerreeeeeeeeeeeee
Yijing Liu, Chao Du, Tianyu Pang, Chongxuan Li, Wei Chen, and Min Lin. Graph diffusion policy optimization. arXiv preprint arXiv:2402.163002 2024b.
Zijun Long, George Killick, Richard McCreadie, and Gerardo Aragon Camarasa. Robollm: Robotic vision tasks grounded multimodal large language models. arXiv preprint arXiv:2310.10221, 2023.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robol learning, pp. 1113-1132. PMLR, 2020.
Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Oshert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301-23320. PMLR, 2023.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023.
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 909-918, 2019.
Andrew S. Morgan, Walter G. Bircher, and Aaron M. Dollar. Towards generalized manipulation learning through grasp mechanics-based features and self-supervision. IEEE Trunsactions on Robotics, 3755):1553-1566 2021. doi: 10. 1009/TR0.001..3558800
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
Alexander Neitz, Giambatrista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. Advances Neural Information Processing Systems, 31, 2018
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. In CoRL Workshop TGR, 2023.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint urXiv:2301.10677 2023.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4), July 2022.
14

---

Published as a conference paper at ICLR 2025
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Karl Pertsch, Oleh Rybkin, Jingyun Yang, Shenghao Zhou, Konstantinos Derpanis, Kostas Daniilidis, Joseph Lim, and Andrew Jaegle. Keyframing the future: Keyframe discovery for visual prediction and planning. In Learning for Dynamics and Control, pp. 969-979. PMLR, 2020.
Rouhollah Rahmatizadch, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3758-3766 IEEE, 2018.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Tont-contr networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pp. 1134-1141, IEEE, 2018.
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Anty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning & modes with one stone. Advances in neural information processing systems, 01-22955-229663 2022.
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang. and Jeannette Bohg. Concep22robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14);1419-4444, 2021.
Lucy Xiaoyang Shi, Archit Sharma, Tony Z. Zhao, and Chelsea Finn. Waypoint-based imitation learning for robotic manipulation. arXiv preprint arXiv:2307.14326, 2023.
Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: periodic autoencoders for learning motion phase manifolds. ACM Trans. Graph., 41(4), July 2022. ISSN 0730-0301. doi: 10.1145/ 3528223.35301777 URL. ht tts:: /wwi. .or.oi..rrrrooooooooooooooooooe. orgg 10. 11 001144//522222 3530178.
Wenhui Tan, Bei Liu, Junbo Zhang, Ruihua Song, and Jianlong Fu. Multi-task manipulation policy modeling with visuomotor latent diffusion. arXiv preprini arXiv:2403.07311, 2024.
Tsunchiko Tanaka, Kenshi Abe, Kaito Ariu, Tetsuro Morimura, and Edgar Simo-Serra. Return-aligned decision transformer. arXiv preprint arXiv:2402.03923, 2024.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, et al. Octo: An open-source generalist robot policy, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jan Ole von Hartz, Eugenio Chisari, Tim Welschehold, and Abhinav Valada. Self-supervieed learning of multi-object keypoints for robotic manipulation. arXiv preprint arXiv:2005.083166 2022.
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning scale. In Conference Robot Learning (CoRL), 2023.
Weikang Wan, Yifeng Zhu, Rutav Shah, and Yuke Zhu. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 537-544. IEEE, 2024.
Liquan Wang, Ankit Goyal, Haoping Xu, and Animesh Garg. Discovering robotic interaction modes with discrete representation learning. In 8th Annual Conference on Robot Learning, 2024.
15

---

Published as a conference paper at ICLR 2025
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06199 06699223880019... 2022.
Zihao Wang. Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-wone open-world multi-task agents, arXiv preprint arXiv:2302.01560 2023.
Yijia Weng, Kaichun Mo, Ruoxi Shi, Yanchao Yang, and Leonidas Guibas. Towards learning geo- metric eigen-lengths crucial for fitting tasks. In Proceedings of the 40th International Conference on Machine Learning, pp. 36958-36977 2023.
Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S Siegel, Jiahai Feng. Noa Korneev, Joshua B Tenenbaum, and Jacob Andreas, Learning adaptive planning representations with natural language guidance. arXiv preprint arXiv: 2.. 08566, 2023.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309111889 2023.
Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 3795-3802. IEEE, 2018.
Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pp. 3536-3555. PMLR, 2023a.
Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper-decision transformer for efficient online policy adaptation. arXiv preprint arXiv:230.008487 2023b.
Ge Yan. Yuch-Hua Wu, and Xiaolong Wang. Dnact: Diffusion guided multi-task 3d policy learning. arXiv preprint arXiv:2403..mm 041155 2024.
Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg. Self-supervised learning of state estimation for manipulating deformable linear objects. IEEE robotics and automation letters, 5(2):2372-2379, 2020.
Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought imitation with procedure cloning. Advances Neural Information Processing Systems, 35:36366- 36381, 2022.
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu-We Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos, 2024. URL https: /tte:///rxxiiv aariiiiiii../... wwwiiiioorgg//eo
2410 0 11758.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huel Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language rewards for robotic skill synthesis. arXiv preprint arXiv:2306. 08647, 2023.
Martina Zambelli, Yusuf Aytar, Francesco Visin, Yuxiang Zhou, and Raia Hadsell. Learning rich touch representations through cross-modal self-supervision. In Jens Kober, Fabio Ramos, and Claire Tomlin (eds.), Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pp. 1415-1425. PMLR, 16-18 Nov 2021. URL https: https://proceedi: /ttpppppoeeenn..ee/ee/ep ml..mrr..... www.s.pressseee v1 2 zambellia 1lmmla..mm....
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy, arXiv preprint arXiv:2403.03394. 2024.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lec, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 0-28566286688 5635. IEEE, 2018.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning -ne-grained bimanual manipulation with low-cost hardware. arXiv preprint 137055 2023.
16

---

Published as a conference paper at ICLR 2025
Haotian Zhou, Yunhan Lin, Longwu Yan, Jihong Zhu, and Huasong Min. Llm-bt: Performing robotic adaptive tasks based on large language models and behavior trees. arXiv preprint arXiv:2404.05134, 2024.
Pei Zhou and Yanchao Yang. Maxmi: A maximal mutual information criterion for manipulation concept discovery. In European Conference on Computer Vision, pp. 88-105. Springer, 2024.
Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, 7(2):4126 4133, 2022.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009./2293 2020.
17

---

Published as a conference paper at ICLR 2025
A CONCEPT DISCOVERY IMPLEMENTATION
A.1 WHY WE USE PROPRIOCEPTION DATA
Our decision to utilize proprioceptive state data guided by two considerations:
Representation of Inherent Motion Patterns. Manipulation concepts are analogous to human motor skills as they primarily encapsulate proprioceptive movements. In human cogmition, we broadly categorize actions without specifying the object involved-for instance, the action is generalized as "grasping" rather than distinguishing between "grasping an apple," "grasping cloth," or "grasping a phone." This categorization emphasizes the movements of our hands rather than the objects being grasped, underscoring a strong linkage between manipulation concepts and proprioceptive states. Utilizing proprioceptive states as foundation for learning these manipulation concepts provides direct pathway to acquiring representations that are deeply integrated into the robot's 's natural motion dynamics. This method aligns with prior research (Shafiullah al., 2022; Lee al., 2024), which utilized action sequences to convert changes in proprioceptive states into discretized representations, capturing the dynamics individual time steps.
Consistency Across Contexts. By focusing on proprioceptive states, we inherently filter out the contextual variability associated with different tasks and environmental setups. This methodological choice ensures greater consistency in the manipulation concepts learned, regardless of task-specific differences. For instance, the manipulation concept of" "grasping" might be employed by a robot across various scenarios involving distinct objects. Proprioceptive sequences during these tasks are more likely remain consistent than sequences derived from environmental data, which may fluctuate with changes in object types, spatial arrangements, and background settings. Such consistency underscores potential for developing manipulation concepts that are not only more stable but also broadly transferable across different operational contexts.
We conducted an additional experiment by integrating environmental observations (images) into the concept discovery process across several tasks. The table below presents the comparative results:
Table 3: Success rates for policies trained using manipulation concepts from two different settings. Ours refers to our method leveraging proprioceptive states to discover manipulation concepts. Img refers to the approach using environmental camera images.
Cof. D2
Mug. D1
Ours
0.72
0.50
Img
0.64
0.40
The results indicate decrease in performance when manipulation concepts derived from images compared to those derived from proprioceptive states. Nevertheless, we acknowledge the importance of visual information in improving our discovery of manipulation concepts. Moving forward, we aim to conduct further research on effectively optimizing the use of visual inputs. Please refer to Sec. D for additional details.
A.2 IMPLEMENTATION OF 3
We VQ- VQ-VAE Encoder Van Den Oord et al. (2017) the basic structure of E in Eq. 1, since it can output a feature vector from an index-able code-book and preserve the gradient flow from continuous encoder (Eq. 2):
2f = E Ett θg)
kj arg min || axlls kER
(15)
a. ake
af (G( 2 +
Here implement & transformer that outputs feature vector for each time-step of 7 D, SG(-) is the stop gradient operator. The fourth line in Eq. 15 makes sure that the gradient in o, can be propagated back 2?.
18

---

Published as a conference paper at ICLR 2025
A.3 IMPLEMENTATION VALUE FUNCTION V
For implementation of Eq. 5, We utilize Hypernetwork to strike a better balance between data adaptability and parameter efficiency (Chauhan al., 2024). This deep neural network takes in proprioceptive states and outputs feature vector.
HN (sT), a E
(16)
Here HN.(.) is hyper-network, and its parameters are generated from the input of concept or A. We use the Sigmoid-processen cosine similarity to form the value function in Eq. 5. Here (5 represents the scalar product two vectors:
V( on)) Sigmoid (T
(HN (oT) HIN (oT)
(17)
HN (sF) - HN (g
Here > 0 the Syper-parameee we set 5.0 to make full use the values Sigmoid can have I+e-5o RJ 0.9933).
A.4 PSEUDO-CODES
We present the pseudocode for the Automatic Concept Discovery pipeline at Alg. 1, designed to derive manipulation concepts from demonstrations E D. The provided example assumes a training batch containing single from D, which is analogous to the case when the batch contains multiple
7.
Algorithm Automatic Concept Discovery
Input:
Demo D {T (sT af T(r)
Modules:
ε, A = fax k k= and index K. { k=
Goal State Detection: G
Goal State Evaluation: V(HN.), II,
Goal Consolidation: R
Output:
trained state encoder
for iteration of training do sample ~~ D
for f 1.2.... T(T) do
2 T(] θe)
k: arg minkek || - axl
b VQ-VAE Encoder Select Manipulation Concepts
of ok? ag (G( 27] + 2T
Preserve Gradient
end for
for i. 1, 1,2,... T(T) do Calculate of using Eq. 4
end for
Calculate Ltd S, "e L
> See Eq. 3, Eq. 6, Eq. 7, Eq. Calculate L b VQ loss and Commitment loss according (Van Den Oord al., 2017) CACD Ltd + L + dent (L + LV9) + C
Back propagation from CCD
end for
A.5 IMPLEMENTATION OF BASELINES
Our method. All the transformers used in our Concept Discovery Module refer to the structure of transformers used in (Brown et al., 2020) and have inner embedding feature of 128 dimensions with 8 heads. The network & in Eq. contains E (Eq. 2), which is a 4-layer transformer, and VQ- VAE of 30 codebook items as A. The model G in Eq. 3 is a 2-layer transformer. The hyper-network HN, in Eq. 16 is able to generate feed-forward linear network of hidden layers to former V in 5. The used in Eq. 7 is a I-layer transformer. The used in Eq. is a 4- transformer. We employ the AdamW optimizer, coupled with warm-up cosine annealing scheduler to modulate
19

---

Published as a conference paper at ICLR 2025
the learning rate. This scheduler initiates at 0.1 times the base learning rate, linearly increases the rate to the base level over the course of 1000 epochs, and subsequently reduces the learning rate to 0.1 times the base rate following cosine function. The weight decay is always 1.0 x 10 3. 10-3. We append all input sequences to the length of 440 and use a batch size of 16 during training. We train our model for 4000 epochs with base learning rate of 1.0 x 10-4, The loss term Eq. and Eq. 7 receive a weight of 1.0. The loss term Eq. 6 and Eq. 8 receive a weight of 1.0 x 10-3. The training process can be finished on single GeForce RTX 3090 in 1.5 days.
XSkill. Based on the design of Xskill Xu et al. (2023a), we implement the skill discovery framework on the Mimicgen dataset, which only contains the "robot" embodiment the Xskill pipeline. We use the default hyperparameters
in the Xskill code base. For the temporal skill encoder, 3-layer CNN network followed by an MLP layer as the vision backbone. Same as Xu et al. (2023a), we augment the images in the input video clip by randomly selected operation from set of image transformations, including random resize crop, color jitter, grayscale, and Gaussian blur. For the Skill Alignment Transformer (SAT), we also use a standard R as our state encoder. The transformer encoder is composed of 16 layers, each layer featuring transformer encoder with 4 attention heads. Additionally, the feedforward network within each layer has a dimensionality of 512. We set the training batch size to 28 and a learning rate of 1.0 x 10 The training takes 4 days to converge on a GeForce RTX 4090 GPU.
use
InfoCon. Based on the design of InfoCon, we refer to the structure of the transformer used in (Brown et al., 2020). All the size of hidden features output by transformers and concept features is 128 here?. The state encoder and state reconstructor both use 4-layer transformer. The goal-based policy uses a 1-layer transformer. The predictor for the generative goal uses a 2-layer transformer. For hyper-network used for discriminative goals, we use 2 hidden layers in the goal function. The number of concepts is fixed, the maximum number of 30 manipulation concepts for all the tasks. We employ the AdamW optimizer, coupled with a warm-up cosine annealing scheduler same as "Our method". The weight decay is always 1.0 X 10-3 3, We append all input sequences to the length of 440 and use batch size of 16 during training. We train our model for 4000 epochs with a base learning rate of 1.0 x 10-4 Other apper-parameters are aligned with the work (Liu et al., 2024a). The training process can be finished on a single GeForce RTX 3090 in 1.5 days.
AWE. In the work of AWE, they set thresholds for the end condition of the dynamic programming of finding way-points with different manipulation tasks. Here we modify the method so that it can discover fixed number (here we choose 10) of key states for all sequences. Since the dynamic programming process naturally discovers a solution with the increasing of a key state number, we can just let it stop at certain key state number.
B CONCEPT-GUIDED POLICY IMPLEMENTATION
B.1 POLICY ARCIITECTURE
For Concept Selection Transformer (CST), we employ a transformer architecture to effectively capture the temporal relationships within a sequence of historical observations and proprioceptive information. This setup is designed to predict the manipulation concepts to be executed, based on current observations. We utilize the standard transformer model, which incorporates a series of self- attention layers and feed-forward networks, as described in (Vaswani et al., 2017). This transformer has 16 standard self-attention blocks with a feedforward dimension of 512. For Concept- Guided Policy (CGP), we use the U-Net architecture, a robust CNN-based network. Initially, the front and wrist images are processed through separate ResNet-15 18 networks to extract image features. These features are then combined with proprioceptive information and the manipulation concepts predicted by CST. This combined data is repeatedly injected into each block of the policy network to enhance the model's responsiveness to conditional information. Thus, the inputs to the CGP include the concatenated features and additional noise. Furthermore, we employ sinusoidal position embedding to embed diffusion timestep information, and within each block, Feature-wise Linear Modulation is used for integrating these features, following the method introduced in (Perez et al., 2018). The architecture of CGP is illustrated in Fig. 6.
We use proprioceptive states.
20

---

Published as a conference paper at ICLR 2025
Concept-aware Diffusion Policy
2D Visual Feature
Conditioning
Proprioceptive State
Mani. Concept
FiLM Modulation
Noisy Action
Denoised
Action
Figure 6: The architecture of concept-aware diffusion policy
B.2 TRAINING DETAILS
For training, each task (along with its respective levels of variation) utilizes 950 randomly sampled demonstrations (see also Sec. 4). The policy network maintains an input observation length of 4 and an action prediction horizon 8, with a batch size set at 150. minimize overfitting, we employ the AdamW optimizer, incorporating a linear warmup of the learning rate. The initial learning rate set at 10-4 and gradually decreases following a cosine annealing schedule as the number of iterations progresses. Training consists of 100 epochs in total. Additionally, decay of I x 10-6 to improve model generalization. For diffusion model training, we employ beta schedule called squaredcos.cat .cppsvcca... with beta range of 1e-4 to 2e-2, which optimizes the generation process by smoothly and controllably adjusting the introduction of noise levels. The CST and CGP trained simultaneously. The training process be completed in less than day a single GeForce RTX 4090 GPU.
apply weight
C
MORE ABOUT EXPERIMENTS
C.1 TASK DETAILS
In our experiment, utilize the MimicGen dataset for the following six categories of tasks (Fig. 7):
(a) Coffee: Insert cylindrical coffee packet into the coffee machine and secure the lid.
(b) Hammer Cleanup: Store the hammer
drawer.
(c) Mug Cleanup: Place the mug into a drawer.
(d) Stack Three: Stack three blocks in the following order from top to bottom: blue, red, green.
(e) Threading: Thread needle through a hole.
(I) Three Piece Assembly: Assemble three components together.
C.2 ABOUT ABLATION STUDY
Here, we present an analysis on the granularity of the discovered manipulation concepts. Since each segment labeled with a specific manipulation concept corresponds to a sub-process that achieves the sub-goal associated with that concept, we calculate the average length of these segments across each task presented in Tab. 2 for both the ablation cases and our proposed approach to assess granularity. The results are summarized in Tab. 4. As observed, the manipulation concept segments in the non-ablated version are shorter than those in the ablation cases. This suggests that the performance degradation due to ablation may stem from the less fine mee-graneed manipulation concepts. This finding aligns with our hypothesis: shorter segments correspond to simpler sub-tasks, thereby facilitating enhanced policy learning.
21

---

Published as a conference paper at ICLR 2025
(a) Coffee
(b) Mug Cleanup
(c) Hommer Cleanup
Threading
Stack Three
Three Piece Assembly
Figure 7: Visualization of the 6 categories of tasks in experiments.
Table Ablation Study of the Proposed Automatic Concept Discovery Module: Average segment
lengths for ablations of different components within the module, compared to the non-ablated version.
Cof.
Ham.
Stk. 3
3 Pc.
Thd.
Mug.
Do
D1
D2
Du
D1
Do
D1
Du
D1
Do
Do
D1
GD
72.0
78.3 75.3
68.5
80.5
78.7
85.7
70.4 70.0
70.3
76.3
72.2
w/o GE
54.3
59.0
44.0
52.0
53.8
57.8
51.6
58.8
58.3
52.8
61.0
51.6
w/o GC
36.2
39.3
63.8
34.5
32.3
38.5
27.5
29.4
29.3
52.8
29.7
31.1
Ours
28.3
31.0
27.6
33.9
29.5
26.7
22.5
28.5
28.8
33.3
29.3
28.5
C.3 CONCEPT VISCALIZATIONS
In this section, we present visualizations of the manipulation concepts identified through Automatic Concept Discovery (Sec. 3.1) and demonstrate their consistency across various tasks.
Fig. 9 and Fig. 10 illustrate the proprioceptive state patterns associated with different manipulation concepts across different tasks, following a methodology similar Ye et (2024). Specifically, In Fig. 9 and Fig. 10, each row corresponds to task and each column corresponds to manipulation concept. The sub-figure at the intersection of a row and column visualizes all the proprioceptive states from the task in the row that associated with the manipulation concept in the column (Notably, the analysis begins by applying PCA to the proprioceptive states across all tasks, so the visualized states here the PCA-reduced representations). The visualization results show that the non-empty sub-figures in each column share similar patterns, highlighting the consistency of the learned manipulation concepts across different tasks.
Please also see Fig. 11, Fig. 12, Fig. 13 and Fig. 14 for visualizations video frames from sub- processes in different tasks under the same manipulation concepts, demonstrating the consistency of manipulation concepts across different tasks. We also apply the Automatic Concept Discovery process real-world data from BridgeData V2 (Walke et al., 2023), demonstrating its ability to discover manipulation concepts that exhibit consistency across diverse scenarios. The concepts are visualized in Fig. 15.
Since MimicGen exhibits different levels of variability, introduced in the Implementation details section of Sec. 4, we also observe that demonstrations from the same task can involve relatively distinct sub-processes. Specifically, we find that different sub-processes are often associated with distinct manipulation concepts. For example, Fig. 8, we illustrate two sub-processes from separate demonstrations of the task threading. Both sub-processes occur in the phase "after grasping the needle, transitioning it to align with the hole." However, due to differences in initial positions, one demonstration requires rotating the needle approximately 180 degrees, while the other involves only a slight rotation. These variations lead to the assignment different manipulation concepts to the respective sub-processes.
22

---

Published as a conference paper at ICLR 2025
Traj. Example
Traj. Example
Concept #17
Concept #1
Concept #26
Concept #15
Figure 8: We observe an interesting phenomenon: for the same task, if sub-processes differ, they are assigned different sets of manipulation concepts. In this case, both sub-processes involve the transition from grasping the needle aligning with the hole. However, in trajectory example the needle must be rotated approximately 180 degrees, while in trajectory example 2, only a minor rotation is required. For the first sub-suucce process, three manipulation concepts are assigned in sequence: #17 - #1 - #26. In contrast, the second sub-process is assigned a single concept, #15.
D FUTURE WORKS
Beyond Proprioception. We posit that a promising method to further capitalize on whole environmen- tal observations (e.g., images future research will concentrate on modeling the interrelations among various modalities, leveraging diverse sensor data from robots to enhance the Automatic Concept Discovery process. However, we face limitations with the datasets currently available; most open-source datasets for robotics primarily include vision and proprioceptive states, along with some linguistic descriptions, but often lack other modalities (such sound). Moving forward, we will treat proprioceptive states and images of environmental observations as distinct "modalities" and apply multi-modal learning techniques enhance the Automatic Concept Discovery process. We plan expand our approach include additional modalities as we collect our own data or gain access to more comprehensive datasets.
entire environment) is through a "multi-modality" approach. Our
Persuasive Analysis Manipulation Concept. Future work will also focus conducting persuasive analysis of manipulation concepts. While task success rates are a common metric in robotics research, they may introduce unfairness due variations in how different policy types align with specific manipulation concepts. Some sets of manipulation concepts may be better suited to particular types of policies, while others align more effectively with different policies. This underscores the need for a more rigorous and insightful investigation into manipulation concepts themselves, as well as a deeper examination of the relationship between manipulation concepts and policies. Such exploration is critical for developing more reasonable and coherent integration of the two. These efforts aim to build a stronger foundation for understanding manipulation concepts and enhancing their integration with policies.
Potential of pearning-from-plee (Lynch et al., 2020). We would like to emphasize the similarity between the learning-frone from-play settings and our settings. Our method shares key characteristics with a bearning-fron from-play scenario, as does not depend on task descriptions or explicit task objectives (Please refer the Implementation details in Sec. 4), such natural language descriptions of tasks. This approach essentially mirrors the collection of an unannotated dataset comprising several demonstrations. In this setup, the Automatic Concept Discovery process operates without assuming any knowledge of the specific goals being pursued in any given demonstration sequence. This is strongly analogous the learning-from-plae from-play setting, where the robot observes sequences or demonstrations that are not tied to explicit objectives (Lynch et al., 2020). This resemblance highlights the relevance and adaptability of our method to contexts similar to learning- stom-play Moving forward, we plan to explore the application of our method directly within a fearning-from- scenario.
23

---

Published as a conference paper at ICLR 2025
Consistencyy Proprioception (coffee)
Oet
Consestency
Consistency Propricception (stack three)
Con P3
Cas #1
con #2
Con
Con
con as
Con as Cas cen
Con,
Cor aio
Con #32
Co
Consisercyy pcopriceetion perreeeeeeeeeeeeeeee jsseembtt
can con
D2e.
Consistnccy Prodriccopton imug dieauup
Crn Pg
Cas
Cte.
Gn
Cas
Con es
Gn cas at cen cen Ger,
Co 931
C51. Cr.P13 On #11
Figure 9: Consistency Visualization (part manipulation concepts #0~14)) Each row corresponds to a task and each column corresponds to manipulation concept. The sub-figure the intersection of row and column visualizes the PCA-reduced proprioceptive states from the task in the row that are associated with the manipulation concept in the column.
Corsisterccy Propeioce repprictttion (oofiee) #21
Consistency
Den
Cas 421 Doe
Consistency Propricceptin (stack, three)
con Con #55
Cas #31
Con #10
On. #12
Cas #12
can #21 Can #22 Can #13 Can woi orr Can #11
Con.e21
Gr.
Co 422
Consistncyy inoprineepiin (theee, pecpppeeeseee assembly) azz
(threading)
Cen wis
Con #ts
czs coe. #22
Can NoT
Con
Consitency Prooriocsption on Imug, cleanup:
Cen e15
On #55
Cas #37
Cee
On #1a
Cas #10 Ces es1 Ge #22 Can #13 can wai Gor ets on #2t
On On 432
Figure 10: Consistency Visualization (part 2, manipulation concepts #15~29), Each row corresponds task and each column corresponds to a manipulation concept. The sub-figure at the intersection of row and column visualizes the PCA-reduced proprioceptive states from the task in the row that are associated with the manipulation concept in the column.
24

---

Published as a conference paper at ICLR 2025
Ham. ex.1
Ham. ex.2
Mug. ex.1
Mug.
ex.2
Concept #2, Release
Figure 11: Consistency visualization of the concepts discovered by our method across different tasks. Here the robot arms release their grippers and put the objects down.
Ham. ex.1
Ham. ex.2
Mug. ex.1
Mug. ex.2
Concept #10, Pulling
Figure 12: Consistency visualization of the concepts discovered by our method across different tasks. Here the robot arms pull things out.
25

---

Published as a conference paper at ICLR 2025
Cof. ex.1
Cof. ex.2
Stk. 3 ex.1
Stk 3 ex.2
Thd. ex11
Thd. ex.2
Pc. ex.1
Concept #4, Transition
Figure 13: Consistency visualization of the concepts discovered by our method across different tasks. Here the robot arms transit the objects in hands.
26

---

Published as a conference paper at ICLR 2025
Cof. ex.1
Cof. ex.2
Mug. ex.1
Stk. ex.1
Stk. ex.2
Thd. ex.1
3 Pc.
ex.1
Concept #8, Grasping
Figure 14: Consistency visualization of the concepts discovered by our method across different tasks. Here the robot arms move towards the objects and ready to pick.
27

---

Published as a conference paper at ICLR 2025
Concept #2
Concept #4
Concept #8
Concept #9
Figure 15: Concept Visualization of the Automatic Concept Discovery process (Sec. 3.1) applied to real-world data BridgeData V2 (Walke et al., 2023). A VQ-VAE with a codebook size of 10 is employed. Four discovered manipulation concepts are visualized: Concept #2 (reaching), Concept #4 (closing the gripper), Concept #8 (placing an object), and Concept #9 (opening the gripper to release an object).
28


Reviews:
[{"review_id": "CzYtzmeCMf", "reviewer": "Reviewer_LkiS", "summary": "This paper proposes a framework for long-horizon imitation learning by combining the ‘automatic concept discovery’ module and the ‘concept-guided policy learning’ module. The former is trained without human annotations of the sub-sequences of robot trajectories. Instead, they assign the concept to the trajectories in a self-supervised manner.", "questions": "1)\tWhy is only proprioception used for the automatic concept? It seems natural to include observations in addition to proprioception to assign the abstract concept to the time step. How can we generalize it to observation? There may be a case where similar proprioception (e.g., robot pose) but a different observation (e.g., an obstacle in the environment).\n2)\tHow do you decide the total number of concepts/codebooks of VQ-VAE (K)? How robust is the proposal in terms of the number of concepts?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a method for hierarchical imitation learning by discovering the motion ‘concept’ without human annotation and motion policy conditioned on the discovered concept. The three tricks for discovering these concepts stably from trajectories seem to be reasonable.", "weaknesses": "Some literature potentially relating to the method or problem setting of this paper is missing. For example, learning abstract action with clustering (BeT[1]) and with VQ-VAE[2]; please contrast this work with others listed above.\nThe proposal is tested with the Robosuite (MimicGen) benchmark, whose task is inherently highly compositional, so the concept discovery seems to be relatively easy. It is interesting to see the result with an “in-the-wild” dataset, such as dataset collected with a ‘learning from play’ manner [3].\n\n[1] Behavior Transformers: Cloning k modes with one stone https://arxiv.org/abs/2206.11251\n[2] Behavior Generation with Latent Actions https://arxiv.org/abs/2403.03181\n[3] https://arxiv.org/abs/1903.01973", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RQZZFMx9Bb", "reviewer": "Reviewer_UouH", "summary": "## Research Question\nFor imitation learning(IL), 1)data annotation requires additional human effort; 2) the inherent subjectivity in manual data collection\\annotation leads to manipulation concepts that do not align well with the robot’s configuration, and such inconsistency will accumulate, and 3)  IL will face further difficulties due to the varying and unpredictable conditions encountered at each step. \n\n\n## Proposed Method\nTo mitigate the above challenges, the author proposed a framework that combines automatic concept discovery with closed-loop concept-guided policy learning. Such a framework will take unlabeled demonstration for training and autonomously extract and utilize manipulation concepts directly from the robot’s proprioceptive states.  Specifically, for training, the \"Automatic Concept Discovery\" module will first derive manipulation concepts automatically. Then, for both training and testing, the \"Concept Selection Transformer (CST)\" will propose and adjust manipulation concepts in real time during the robot’s interactions with the environment. Finally, the \"Concept-Guided Policy (CGP)\" utilizes the selected manipulation concepts to execute actions based on instantaneous visual input", "questions": "Dear author:\n\nGenerally, the paper is a decent work, but it needs several extra explanations and explorations.\n\nAnd I will raise my rating if the following concerns are solved:\n1. The concerns in the \"Weakness\" part.\n2. A minor point is to add an IL SOTA baseline specific to the problem, which is not limited to a Diffusion Policy basis.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea is a good catch for research questions\n\nThe manual annotation(sample efficiency), accumulative action inconsistency, and environmental unpredictability are classic bottlenecks for imitation learning,  and the general idea of \"self-discovered sub-policy\\concepts\" is also not something new. However, the author directly extracts the concepts from the robot’s proprioceptive states,  such design not only mitigates the misalignment caused by ambiguities in human semantics but also adapts dynamically to unforeseen situations. \n\n2. Appropriate baseline and good performance\n\nThe selected baselines are new and SOTA, and reasonable to compare with the proposed method.  Major advances of the proposed method are observed according to Table 1.", "weaknesses": "1. Unclear presentation \n\nThe figures are a little bit messy (e.g. lack the connection of each part in Fig 3, the Closed-Loop Policy part in Fig 2 is unclear  ), but it's a minor point. What matters are: 1) if the discovered concepts are consistent in all demos, i.e. if the concepts are in a unified, task-agnostic space for all demos; 2) For evaluation, it says 950 demos for each task, so does it mean: 2.1) the training data is 950 * |# of task| for single unified policy (set of concepts) and get tested all 6 type of tasks at once( in a mixed way). Or: 2.2) for each task,  it uses 950 demos to train and test only for the task.\n\nFor 1), I suppose the concepts are consistent in all demos according to the formula (1), but it's better to be clear at the beginning.  For 2), I'm not sure if it's 2.1) or not, if it's 2.2), the performance will be questionable as it's very close to overfitting. \n\n2. Less practical problem setting\n\nThere was a trend to label the raw robot demonstration automatically, but a more practical problem is the raw demo collection is much more expensive than demo annotation, and therefore, some researchers turn to human-play data to get more data to use. And it's the same story for this paper, the expensive robot demo collection reduces the significance of this work \n\n3. Concern about generalizability and scalability \n\nThough the paper claims better performance (compared with baselines) in a more diversified initial environment,  here the diversity is only expressed by object positions and placement angles. Some concerns are: 1)What if there is concept inconsistency in the demos,  e.g. to do the same manipulation, in a different way; 2)With the growth of the demo amount, will the # number of concepts converge or grow fast;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Rcsd7FRhq", "reviewer": "Reviewer_Wk2p", "summary": "This paper presents a method for discovering robot actions and manipulation policies from unlabeled demonstration data. It presents techniques for discovering generic action/manipulation concepts and their goals, evaluating progress towards those goals, and generating policies to achieve the goals. This is done using various modern machine learning techniques and problem/loss function formulations. For concept discovery, A VQ-VAE is used to discover concepts and put them into a finite codebook. A transformer model is used for goal state detection. A hypernetwork and another transformer are used for goal state evaluation, and a final transformer is used for goal consolidation. All of these networks are jointly trained using a large combined loss function. Then, concept selection and policy generation are learned using a combination of a transformer and a diffusion policy. Finally, the approach is validated by training on a set of 950 robot demonstrations in 6 different tasks, then evaluated with randomized starting positions (all in simulation). The success rate of different tasks learned via the proposed method is higher than for competing metrics, even/especially in the presence of perturbing noise.", "questions": "* At some point, the number of different networks + their parameters runs a risk of over-parameterization and overfitting. This is especially true since it seems that the set of demonstrations were collected over the same set of tasks as used in policy generation and success measurement. Combined with the lack of qualitative cross-task concept comparison or similarity analysis, I do have overfitting worries here. I have no evidence to directly support this flaw other than my own experience training large robotics models, but would still like to know the authors' thoughts on how overfitting is avoided in the proposed approach.\n\n* Why was a hypernetwork used for a single piece of the approach?\n\n* How were the 950 demonstrations in the experiment generated, and were they over all 6 tasks, or a different set/subset of tasks? I was not able to find this anywhere in the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper compares the proposed approach to a set of highly relevant prior work in a fairly direct head-to-head comparison and the proposed approach nearly universally outperforms the prior work by a good margin.\n\n* The approach is a novel combination of several modern techniques to solve a complex open problem in robotics. It expands upon work such as InfoCon by adding policy learning directly into the framework of manipulation concepts.\n\n* The paper is written clearly and the construction of the experiments is well-explained (training, structure of network, etc.), including the re-implementation of baselines.", "weaknesses": "* The primary weakness of this study is that it only contains a brief evaluation of the quality of the learned skills (i.e., are they generalized, are they more generalized than prior approaches), and a single visual example. Characterizing XSkill \"concepts\" via K-means clustering is perhaps not the most charitable interpretation of their concepts, as K-means clustering will naturally tend to collect noise in its clusters (noise is one of the stated drawbacks of the xskill results). Ideally there would be some sort of mean similarity score or some other comparison method - even something like a 2D embedding visualization of the different skills and concepts, if possible to generate, would give better faith in the stated quality of the learned concepts. This is a conference about learning representations, so I would love to see more focus on the quality of the representation of \"concept\". That being said, other papers, including InfoCon, also use final task success as the main comparison metric, so I don't consider this a big enough weakness to reject the paper.\n\n\n* It is not clear why such a large variety of approaches were used in this paper. Many different transformer sizes were used, in addition to the diffusion policy and the VQ-VAE. Specifically, the use of hypernetworks also seems like an unnecessary addition to what is already a diverse and expressive set of architecture choices.\n\n**Style notes**\n\n* perhaps this is a notational convention I am unaware of, but the parentheses in epsilon^(n) seem like unnecessary clutter.\n\n* Should eq 11 be negative to account for the fact that the log term will always be negative?\n\n* In eq 10, is the selected action from pi_D conditioned on the k selected by p_CST? If so, this could be made clearer, perhaps by splitting into two equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tN8EskCQRU", "reviewer": "Reviewer_Umdm", "summary": "The paper presents a novel imitation learning framework for training closed-loop concept-guided policies for robotic tasks, focusing on long-horizon tasks. The framework leverages self-supervised learning to autonomously discover manipulation concepts directly from robotic demonstrations, without the need for predefined skills or human-labeled data. The two main components of the system are:\n- Automatic Concept Discovery, which identifies manipulation concepts by analyzing proprioceptive states and assigning discrete embeddings to different task segments.\n- Concept-Guided Policy Learning, which utilizes these discovered concepts for policy learning. A Concept Selection Transformer (CST) selects relevant manipulation concepts in real time, while a Concept-Guided Policy (CGP) generates actions based on the selected concepts, using a diffusion policy to handle high-dimensional actions.\n\nThe framework is designed to dynamically adjust policies in response to environmental changes, ensuring robust performance across a variety of tasks. Experimental results show that the proposed method outperforms existing baselines in multiple robotic manipulation tasks, demonstrating improved generalization and performance in dynamic environments.", "questions": "- There is some line of work using mutual information maximization to discover concepts/skills from demonstration datasets [1, 2] or self-supervised methods [3]. It may help to include some insights into those works.\n- While the annotations used in the method section are very stringent and scientifically sound, it can be confusing for readers with less background knowledge or patience to fully understand the message the paper tries to deliver.\n- In the automatic concept assignment stage, is the encoder trained separately from other components, or are they trained simultaneously? If the former, how did you prevent the encoder from collapsing to map to the same codebook index?\n- Are all the discovered concepts meaningful and physically possible? Can I condition on an arbitrary concept index and the generated trajectories are good to use, or is the Concept Selection Transformer still necessary to act as a filter to rule out discovered but useless concepts?\n- How to deal with diversity? In skill discovery works, one common issue is a diverse set of skills/concepts can be discovered, but this diversity is canceled out by a downstream skill/concept selector due to its collapse on one or two most useful skills/concepts. How can you retain all the concepts discovered and demonstrate the diversity there?\n\n[1] Li, C., Blaes, S., Kolev, P., Vlastelica, M., Frey, J. and Martius, G., 2023, May. Versatile skill control via self-supervised adversarial imitation of unlabeled mixed motions. In 2023 IEEE international conference on robotics and automation (ICRA) (pp. 2944-2950). IEEE. \n\n[2] Peng, X.B., Guo, Y., Halper, L., Levine, S. and Fidler, S., 2022. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41(4), pp.1-17.\n\n[3] Li, C., Stanger-Jones, E., Heim, S. and Kim, S., 2024. FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning. arXiv preprint arXiv:2402.13820.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The approach is highly original in combining concept discovery with closed-loop policy learning. Instead of relying on predefined, human-annotated labels or skills, the system autonomously discovers meaningful manipulation concepts directly from demonstrations. This removes the need for manual intervention and avoids misalignment between human semantics and robot operations.\nThe use of a self-supervised concept discovery mechanism is innovative, especially in its application to long-horizon robotic tasks. The introduction of a Concept Selection Transformer (CST) for dynamic task execution is novel and enables adaptive behavior based on feedback.\n- The methodology is well-designed and robust, incorporating a clear pipeline for concept discovery and policy learning. The experimental setup is thorough, with comparisons to multiple baselines and a wide range of tasks, which demonstrates the effectiveness of the proposed framework.\nThe paper also includes an ablation study to assess the contributions of different components within the framework, which adds depth to the analysis and validation of the proposed method.\n- The paper is clearly written, with detailed explanations of the method, supported by visualizations of the discovered concepts. The distinction between the two main components (Automatic Concept Discovery and Concept-Guided Policy Learning) is clearly articulated, and the figures help in understanding the flow of the framework.\nThe technical details of the system, including the loss functions and architecture choices, are well-explained.", "weaknesses": "- Concept Interpretability: While the system autonomously discovers manipulation concepts, it would be interesting to try to explore how interpretable or human-understandable these concepts are. It would also be interesting to see if all the discovered concepts are utilized in downstream policy learning and, if not, what the unused ones look like.\n\n- Evaluation Scope: Although the paper evaluates the framework on a variety of tasks, the focus is primarily on tabletop manipulation in simulated environments. Expanding the evaluation to more complex morphologies or real-world scenarios would strengthen the claim that the approach generalizes well.\n\n- Lack of Real-World Validation: While the framework performs well in simulation, it has not been validated on real-world robotic platforms. The paper acknowledges this limitation but does not provide a clear roadmap for transitioning the system to physical robots, which is crucial for demonstrating its practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)