$\textit{HiMaCon:}$ Discovering Hierarchical Manipulation Concepts from Unlabeled Multi-Modal Data
Authors: Ruizhe Liu
Pei Zhou
Qian Luo
Li Sun
Jun CEN
Yibing Song
Yanchao Yang
Venue: NeurIPS 2025 poster ()
Keywords: Robotics
Self-supervised Learning
Concept Discovery
Primary area: applications

Abstract: Effective generalization in robotic manipulation requires representations that capture invariant patterns of interaction across environments and tasks.
We present a self-supervised framework for learning hierarchical manipulation concepts that encode these invariant patterns through cross-modal sensory correlations and multi-level temporal abstractions without requiring human annotation.
Our approach combines a cross-modal correlation network that identifies persistent patterns across sensory modalities with a multi-horizon predictor that organizes representations hierarchically across temporal scales. Manipulation concepts learned through this dual structure enable policies to focus on transferable relational patterns while maintaining awareness of both immediate actions and longer-term goals.
Empirical evaluation across simulated benchmarks and real-world deployments demonstrates significant performance improvements with our concept-enhanced policies. 
Analysis reveals that the learned concepts resemble human-interpretable manipulation primitives despite receiving no semantic supervision. This work advances both the understanding of representation learning for manipulation and provides a practical approach to enhancing robotic performance in complex scenarios.

HiMaCon: Discovering Hierarchical Manipulation Concepts from Unlabeled Multi-Modal Data
Ruizhe Liu'
Pei Zhou'
Qian Luo Sun Yibing Song Yanchao Yang!
Jun Cen
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong
Department of Electrical and Electronic Engineering. The University of Hong Kong DDAMO Academy, Alibaba Group "Transcengram {zr11r2360, pezhou, qianluo, sunl ( hku. hk [cenjun. cen, songyibing sybl@alibabe -nnc con, yanchaoy@hku. hk
Abstract
Effective generalization in robotic manipulation requires representations that cap- ture invariant patterns of interaction across environments and tasks. We present self-supervised framework for learning hierarchical manipulation concepts that encode these invariant patterns through cross-modal sensory correlations and multi- level temporal abstractions without requiring human annotation. Our approach combines a cross- modal correlation network that identifies persistent patterns across sensory modalities with multi-horizon predictor that organizes representa- tions hicrarchically across temporal scales. Manipulation concepts learned through this dual structure enable policies focus transferable relational patterns while maintaining awareness of both immediate actions and longer-term goals. Empirical evaluation across simulated benchmarks and real- worldwrld deployments demonstrates significant performance improvements with our concept-enhanced policies. Analy- sis reveals that the learned concepts resemble human-interpretable manipulation primitives despite receiving no semantic supervision. This work advances both the understanding of representation learning for manipulation and provides practical approach to enhancing robotic performance in complex scenarios. Code is available at: https: // //tg::////ttbb// gigguu......... com/coo///aoooooo com/zrllrz/ #rllrz/HiMaCon. HHiMaCnM....
1 Introduction
Robot manipulation in diverse, unstructured environments remains fundamental challenge. Despite advances in policy learning and architectures [4, 10, 19, 24], current approaches often fail when encountering unexpected variations or novel scenarios. As illustrated in Fig. 1, a policy trained to place cups into containers may succeed in familiar settings but fail when encountering unexpected barriers serriers-revealing a critical generalization gap limiting real-world deployment.
We propose that addressing this challenge requires learning transferable manipulation concepts hierarchical abstractions capturing fundamental manipulation patterns. These concepts connect low-level actions to high-level goals, enabling robust generalization. For example, the concept of "placing an object inside container" encompasses invariant relational patterns that persist whether the container has barriers or not, allowing adaptation while maintaining core manipulation strategy.
To acquire these manipulation concepts, we propose selfvsuper framework that learns hierar- chical latent representations without requiring labor-intensive human annotations [13, 28, 40J. Our approach operates through two complementary mechanisms: 1) Cross-modal correlation learning captures invariant patterns across different sensory modalities (vision, proprioception), enabling generalization across visual variations while preserving functional relationships. When placing ob-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training Data
Policy W/O
Manip.
Concepts
Blocked
Vibration
Test
Policy With
Manip.
Concepts
Figure 1: Manipulation concepts enhance generalization. Top: Training data with cups and containers without barriers. Middle: Without manipulation concepts, policies fail when encountering barriers. Bottom: With our concept enhancement, policies adapt accordingly.
jects in containers, these correlations encode the relationship between visual perception of container boundaries and proprioceptive feedback during placement, regardless of container appearance. 2) Multi-horizon sub-goal organization structures concepts hierarchically across temporal scales, from immediate actions (e.g., "align gripper with object") to extended sequences (e.g., "transport object to container"). This hierarchical representation enables policies to simultaneously reason about immediate actions and longer-term goals, maintaining task coherence even when specific execution paths require adaptation.
Our experiments across both simulated benchmark tasks and real-world robot deployments demon- strate that policies enhanced with these manipulation concepts consistently outperform conventional approaches, particularly challenging scenarios requiring adaptation to novel objects, unexpected obstacles, and environmental variations (Fig. 1). The learned concepts form interpretable clusters that resemble meaningful manipulation primitives, providing insights into how robots perceive and reason about manipulation tasks.
In summary, key contributions include: (1) self-supervised framework that extracts structured hierarchical manipulation concepts from unlabeled multi-modal demonstrations, capturing both cross-modal correlations and multi-level temporal abstractions without human annotation: (2) an effective policy enhancement approach that integrates these concepts through joint prediction, main- taining compatibility with diverse policy architectures; and (3) comprehensive empirical evidence demonstrating significant performance improvements across diverse settings, with analyses revealing how learned concepts enable more robust generalization novel environments.
Related Work
Representation Learning in Robotics Self-supervised representation learning has emerged as powerful approach for extracting meaningful skills [29, 32, 48] from robotic data, avoiding the
need for manual annotation in methods such as [13, 28, 37, 40]. Initial efforts explored single- modality approaches for vision-based [7, 11, 51, 65, 68] and proprioception-base. representation learning. Recent work integrates multiple modalities, combining vision with language [22, 36, 42, 50, 64), proprioception with vision [47, 62, 66), even richer [6, 53, 69).
[26, 39, 45, 52]
These approaches typically focus on cross-modal alignment but often overlook the structured temporal patterns inherent in manipulation tasks. Parallel developments in temporal representation learning have addressed this challenge through various approaches: time-contrastive learning [27, 34, 35, 42, 65], temporal masked auto-encoding [47], and explicit modeling of state transitions across different

---

Manipulation Concept Discovery
Concept Encoder
Capture Cross-Modal Correlation
Random Mask
E
Sec.3.2
Sec.3.3
Enhancing Imitation Learning
cluster
cluster
Shart-horizon Subgoal
Long-horizon Subgoal
Modality
Modality
Modallty 3
Policy (Eq.9)
p Action
"Clean into
Represent Muli-Horizon Subgoals
Figure The proposed self-supervised manipulation concept discovery and policy enhancement, Stage 1: The concept encoder (E) processes multi-modal robot demonstrations extract concept latents. These latents are refined through two objectives: (1) the Cross Modal Correlation Network (C) employs mask-and-predit strategy capture persistent patterns across sensing modalities (Sec. 3.2); (2) the Multi-Horizon Future Predictor (F) enables concept latents to organize hierarchically into multi-horizon sub-goals based coherence thresholds (e) (Sec. 3.3). Stage 2: The learned concepts are integrated into policy learning through backbone network (Tx) with concept (π₂) and action (Ta) prediction heads, regularizing action generation with structured manipulation knowledge (Eq. 9).
timescales [25, 44, 54. 55]. Our work advances the field by simultaneously addressing both multi- modal integration and hierarchical temporal structures, creating representations that naturally align with manipulation sub- goals at varying time horizons while leveraging cross-modal correlational patterns that persist across different objects and contexts.
Coudept-Guided Robotic Policies Concept-guided approaches enhance robotic policy performance by leveraging intermediate representations that bridge perception and action. These methods generally fall into two categories. First, two-module frameworks [3. 8, 29, 62, 68, 73] employ a dual-model architecture where one component extracts high-level task concepts while another generates the corresponding actions. While effective, these approaches often require specialized architectural designs that limit their applicability across different policy classes.
Second, in contrast, joint prediction approaches [18, 20, 56, 58, 67] integrate concept guidance by training policies to simultaneously predict both concepts and actions. This creates an implicit information pathway where concept understanding regularizes action generation. Our work adopts this more flexible approach, enabling seamless integration with diverse policy architectures while maintaining the interpretability benefits of explicit concept representations.
Method
We aim to encode robotic manipulation demonstrations into latent representations that capture task- induced patterns in molti- sensory-motor data. These representations should naturally cluster according functional sub-goals, providing insights into manipulation objectives and enhancing policy learning. We term these clusters manipulation concepts- representing action sequences targeting specific Lub-goals-and and call the learning process manipulation concept discovery.
Our self- uuf-suueeeieed approach works without explicit sub-goal annotations, addressing the challenge of capturing meaningful manipulation patterns without labels. We design objective functions enforcing latent representations that reflect both temporal structure and cross-modal correlations. Our approach ensures: (1) integration of modality-speciff features while encoding cross-modal correlations that persist across objects and contexts; (2) hierarchical organization of sub-processes representing sub- goals across temporal horizons and enabling action prediction guided by immediate and long-term

---

objectives. We validate these concepts through policy performance improvements and multiple analysis methods that demonstrate correspondence with meaningful manipulation primitives.
3.1 Problem Setup & Manippllaton Concept Encoder
Given a dataset D {T of N manipulation trajectories, each {(0), d contains observations of and actions af time For M modalities, of fo; 2,1 ,-.., ,0 o, where o; of" is the observation of the modality m. We denote of = [fo m E S] as observations of modalities in S S [M]] 2, ... M). We treat observations of both the same sensory modes (e.g.. multiple views) and different modes as distinct modalities, as they are functionally different modalities in terms of complementary information.
The Manipulation Concept Discovery process assigns latent representation of E RR2 to each timestep t of the trajectory T;, where of can be viewed as a noisy sampling of the underlying manipulation concept active Since these representations cluster based on sub-goals, we refer to 2 as ma- nipulation concept latents simply manipulation concepts. We use continuous representations for differentiability and to avoid constraints of codebook- capacity). To learn zf, we introduce a manipulation concept encoder & parameterized by θg, which maps observation sequence o; (= from trajectory T; to concept sequence [
discrete representations (e.g., finite
T;
Jt=
2, < (o,; Og)
We implement& using transformer encode temporal dependencies (details in Sec. A. Next, elaborate on the training strategies optimizing cross-modal and multi-horizon temporal correlation metrics (Sec. 3.2 and 3.3).
3.2 Capturing Multi-Modal Correlations
To enhance the utility of multi-modal information, we propose that manipulation concepts should capture cross-modal correlations rather than simply aggregating features from different modalities (e.g., concatenating multi- moltil signals [9, 71).. Physiological evidence suggests that concept formation often occurs when correlations across sensory modalities are high [1, 15, 38, 57, 63). These correlations remain consistent across scenarios involving the same concept, facilitating generaliza- tion. For instance, in container opening tasks, the correlated patterns between visual lid rotation, characteristic force feedback, and audio cues persist across different container types, enabling the transfer of the "opening" concept despite variations in object appearances.
To learn manipulation concepts that capture cross-modal correlations, we propose maximizing mutual information a metric capable of modeling diverse correlations-beteee observations from different modalities, conditioned on the associated manipulation concept. Specifically, we maximize the conditional mutual information over bipartitions of modality observations:
Σ
[ OMiys Z)
max Z
(2)
SCIM
where Os are observations from a subset of modalities, O/Myy are observations from remaining modalities, and Z is the manipulation concept. We implement Eq. using computationally efficient self-supervised mask-and- preddct approach that stochastically samples bipartitions during training. This ensures scalability despite exponentially increasing bipartition numbers while integrating cross- modal correlation learning with multi-modal information compression.
Correlation Network C (CMCN) with parameters O reconstructs full- modality observations from partial observations guided by manipulation concepts. During training, mask observations from random subset S of modalities and reconstruct all observations of using the unmasked subset o SM and concept at:
Specifically,
Pam (t, T,) = Es C (CCOoo/)) MMVS.. of: 0.) -ol
(3)
where S ~ U (2 \ (0))
is uniformly sampled non-empty subset of modality indices. By predicting full observations from partial inputs, we maximize the conditional mutual information in Eq. 2, forcing manipulation concepts z/ to capture modal correlations. Additionally, when all modalities are masked, reconstruction solely from 2f ensures these representations compress and preserve essential
information (please see See. A.1 for more details).

---

3.3 Representing Multi-Horizon Sub-Goals
To complete tasks with hierarchical structures, manipulation concepts must encode multi-horizon sub-goal information. Physiological evidence shows human actions are hierarchically organized [17, 41], with coarse-grained goals defining overall actions. These multi-horizon sub-goals link ultimate goals with low-level actions, enabling smooth transitions while enhancing robustness.
tasks and fine- gee-irraied goals informing immediate
We aim to make manipulation concepts organized to encode sub-processes across multiple temporal horizons without explicit annotations. Since concepts cluster by sub-goals, hierarchical sub-goals can emerge from these clusters at varying temporal scales. We propose that the temporal extent of sub-process is determined by concept latent coherence within clusters, yielding natural spectrum from short-horizon to long- morizho sub-goale Specifically, given manipulation concept latents z; = { from trajectory Ti, we quantify their similarities using spherical distance: (ist u) = + arccos || | )- Concepts belong to the same sub-process if their distance falls below a coherence threshold E [0, 1]. More explicitly, sub-processes are derived as:
16
h (Z:: e) { [9 [k+)) k 1,
<(z;()),
where 91 = 1, Jk+11 = max {gg g E (ge, Ti + I A Vt, (gw.9): dist 2f¹) <c)
(4)
where K K(z;,) is the number of clusters determined by e, and increasing yields sub-processes spanning from short-horizon to long-hone nonghhorizon. Please see Alg. 1 for more details.
Furthermore, we propose learning objectives ensure multi-horizon sub- processes from Eq. align with meaningful sub-gou -obaaoal completion processes. Specifically, the manipulation concept guiding each sub-process should be informative about the state achieved upon sub-task completion [5, 33, 72]. For all coherence thresholds c, current observation O and its associated concept Z should be informative of the terminal observation Ogoal(e), characterized by minimizing the following conditional entropy:
Ve, min H (gxal((e) O.
(5)
To implement Eq. 5, we train a Multi-Horizon Future Predictor JF (MHFP) to hallucinate terminal observations of different sub-processes. For time step in trajectory Tú, the terminal observation is determined by the ending time step of the interval containing t:
g(t: 2, c) = min 9++1). where t E [9k: 11+1 E h(z;c),
(6)
During training, the network F, parameterized by θ, predicts this terminal observation based on current observation of. manipulation concept z/. and coherence threshold e:
Cuh (t, T2)) = E. F (of. of, 0gy - , s(tiz..)
where ~ U([0, 1]) is sampled uniformly per iteration to improve efficiency by avoiding training over all c values. This training process iteratively improves both latents and sub-process derivation: compute manipulation concepts using the encoder (Eq. 1), determine sub-process boundaries, then update all networks, including F and the concept encoder. This improves future observation prediction and concept latents, which in turn refines sub-process derivation. By minimizing Eq. 7, of is ensured to encode multi-horizon sub-goal information, indicating hierarchical transitions to terminal states under various while adjusting sub- sub-processes by shaping concept latents for terminal state predictability. More details can be found Sec. A.
Final Objective for Manipulation Concept Discovery. We jointly optimize the multi-modal correla- tion objective (Eq. 3) and multi-horizon sub-goal prediction objective (Eq. 7) to ensure manipulation concepts generated by the encoder & (Eq. 1) satisfy both key properties:
L, (t, T,) Aum Cunn (t, Ti) + Cuth Tt),
(8)
where Amm, Amh > 0 balance the loss terms.
3.4 Enhancing Imitation Learning with Manipulation Concepts
After learning manipulation concepts through our self-supervised framework, we address how these concepts enhance policy learning. Unlike previous approaches that learn task-specific policies
5

---

directly from demonstrations [12, 28], we propose to leverage the learned manipulation concepts as an informative representation that bridges low-level actions and high-level goals.
Specifically, with manipulation concepts 2, generated by encoder &, we augment imitation learning by training policies to predict both ground-truth actions and corresponding concepts [21, 67, 70]. This approach uses concept prediction as a regularization that guides the policy to encode conceptual understanding alongside action planning:
h4 (ot, B;; 0*),,
#/ = (1) 02), = na (k(; 0%)
C (t, Te, (s) = ||ā aIll + Amell 24 2/
The policy consists of: (1) A backbone Tr processing task descriptions L; and observations of to produce shared representation hf; (2) A concept predictor ıι mapping hद predicted concepts 2 and (3) An action decoder mapping h to predicted actions at. This joint objective enforces the policy to leverage concept information encoded within h while predicting actions. Even though concepts are learned task-agnos for generalization, the policy receives task descriptions in multi-task setting, serving as a mechanism to learn the reuse of concepts. The learning objective balances action and concept prediction using Ame > O. More details are provided in Sec. A.2.
4 Experiments
evaluate our manipulation concept discovery approach through experiments addressing four key questions: (1) Do learned concepts enhance policy performance tasks used for concept discovery, validating our strategies for encoding cross- correlations (Sec. 3.2) and multi-horizon sub-goals (Sec. 3.3)? (2) Can concepts learned from one task set transfer effectively to different tasks sharing underlying manipulation patterns? (3) Does our concept discovery mechanism generalize to novel tasks with decreased overlap in manipulation patterns? (4) What interpretable properties emerge in the learned concepts that explain their effectiveness for robotic manipulation? Through these investigations, demonstrate both the immediate benefits of approach for imitation learning and its broader applicability for transfer learning and generalization in manipulation tasks.
4.1 Experimental Setup
Dataset and Environment Sec. 4.2 and 4.3 conduct experiments using the LIBERO benchmark [30], a comprehensive platform for robotic learning built on Robosuite [75]. We utilize three distinct
task sets:
LIBERO-90: diverse collection of 90 manipulation tasks serving as our primary training domain for concept discovery and initial policy learning.
novel long-horizon tasks, each composed of two LIBERO-90 tasks in sequence, designed to evaluate transfer to more complex task structures. LIBERO-GOAL: 10 tasks an entirely novel environment unseen during concept discovery, used to evaluate the generalization of learned concepts to unfamiliar contexts.
LIBERO-LONG:
Each task includes natural language description and 50 expert demonstrations. For multi-modal observations, we use: Agentview vision: 128x 28 RGB third-person camera capturing the entire environment; Eye-in-hand vision: 128x128 RGB gripper- camera; Proprioceptive state: vector encoding gripper position, rotation, and physical states.
Manipulation Concept Discovery Methods We compare our approach with several state-of-the-art concept discovery baselines (implementation details in Sec. A.3):
InfoCon [31]: A VQ-VAE type of method for angle-hierarchy concept discovery.
XSkill [65]: Contrastive learning for manipulation skill extraction from demonstration videos. DecisionNCE [27]: Learns reward-relevant representations from demonstrations with language annotations, evaluated in two variants: using task instructions (DecisionNCC-ttak. and using elementary action labels (DecisionNNCE-motion)
RPT [47]: Temporally and modality-masker autoencoder for multi-modal sequence modeling. All: A simplified variant of our approach that predicts all modalities from concepts without modeling cross-mode correlations.

---

Table 1: Evaluation of manipulation concept discovery methods across different task settings. Success rates (%) of ACT and Diffusion Policy (DP) models when enhanced with manipulation concepts from various discovery methods. All concept encoders were trained only on LIBERO-90, and evaluated on: original tasks (I.90-00, novel long-horizon compositions (L.9--L), and entirely new environments (I.90-). Values in parentheses show standard deviations across seeds. Bold and underlined values indicate best and second-best results.
DecisionNCE
L90-90 InfoCon XSkill
RPT
All
Next
CLIP DINOv2
Plain
Ours
task
motion
66.5
73.4
68.8
64.1
68.0
63.8
71.9
69.0
66.8
46.6
74.8
ACT
(0.8)
(0.8)
(0.8)
(2.0)
(0.4)
(0.5)
(0.3)
(0.1)
(0.8)
(1.9)
(0.8)
78.2
87.7
84.3
81.5
82.6
80.7
79.4
75.7
82.7
75.1
89.6
DP
(0.6)
(0.6)
(0.1)
(0.5)
(0.1)
(0.9)
(0.1)
(0.8)
(0.6)
(0.6)
(0.6)
DecisionNCE
L90-L
InfoCon XSkill
All
Next
CLIP DINOV2
Plain
Ours
task motion
55.5
55.0
59.0
55.5
55.0
51.0
55.0
53.0
49.3
54.0
63.0
ACT
(0.9)
(1.0)
(1.0)
(0.9)
(1.0)
(1.0)
(1.0)
(1.0)
(0.9)
(0.9)
(1.0)
75.0
73.0
61.3
79.3
83.0
67.0
63.0
58.7
52.7
34.1
89.0
DP
(1.0)
(1.0)
(0.9)
(0.9)
(1.0)
(1.0)
(1.0)
(0.9)
(0.9)
(1.1)
(1.0)
DecisionNCE
L90-G InfoCon XSkill
RPT
All
Next
CLIP DINOv2
Plain
Ours
task motion
67.0
77.0
75.0
69.0
71.0
77.0
77.3
70.0
75.0
57.0
81.0
ACT
(1.0)
(1.0)
(1.0)
(1.0)
(1.0)
(1.0)
(0.9)
(0.9)
(0.5)
(1.0)
(1.0)
92.7
93.0
91.5
91.0
91.3
92.0
91.0
92.0
93.0
90.7
95.7
DP
(0.9)
(1.0)
(0.9)
(1.0)
(0.9)
(0.9)
(0.7)
(0.8)
(1.0)
(0.9)
(0.7)
Next: Predicts adjacent time-step observations, common approach adopted in [7, 68).
CLIP [46]: Language-aligned visual features from a pretrained foundation model.
DINOO2 [43]: Self-supervised visual representations without temporal modeling.
Plain: Standard imitation learning without manipulation concepts.
Enhanced Imitation Learning To evaluate the effectiveness of our discov- cred manipulation concepts, we integrate them into two established imitation learning frameworks using the joint prediction approach described in Sec. 3.4:
Policies for
ACT [71]: A transformer-bel based conditional variational autoencoder that predicts action chunks. Diffusion Policy (DP) [9]: A ID convolutional UNet that generates actions through denoising.
For both policy architectures, we add the concept prediction head (æ= in Eq. 9) to predict manipulation concepts from the shared concept-aware representations. Implementation details appear in Sec. A.2. All experiments are reported with results aggregated across random seeds.
4.2 Evaluating Policy Performance with Learned Manipulation Concepts
Performance on Original Training Tasks We first evaluate our concept discovery method on the same tasks used for concept training. As shown in the 1.90-90 results (Tab. 1), our approach consistently outperforms all baselines with both policy architectures. The performance gap between our method and Next/InfoCon demonstrates the importance of haatiihherarccialll sub-goal modeling, while improvements over All highlight the value of explicitly capturing cross-men cross-modal correlations. Our method also surpasses DecisionNCE variants despite requiring language supervision, validating the effectiveness of our self-supervised objectives.
Transfer to Long- Long-Horizon Tasks To evaluate concept transferability to more complex composi- tions, we apply concept encoders trained on LIBERO-99 -00 directly to LIBERO-LONG demonstrations featuring novel long-horizon tasks. The L.90L results show our method maintains its performance advantage in this challenging transfer setting. This demonstrates that our approach learns manipu- lation concepts that effectively decompose hierarchical tasks, enabling policies to better handle novel complex task compositions requiring sequential execution of multiple sub-goals.
7

---

Table 2: Impact of modality combinations on concept discovery performance. Success rates (%) of ACT and DP policies using manipulation concepts discovered with different input modality combinations. All models were trained and evaluated on LIBERO-90, with specific modalities excluded (marked with "-"). A: agentview vision, H: eye-in-hand vision, P: proprioceptive state.
Ours
H
A
H
-
- -H-
A
ACT 74.8=0.8
7..
71.3+0.3
70.1
67.5+0.8
68.7+0.6
69.4=0.4
DP 89.6#0.6
85.8+0.2
85.640.3
84.3+0.5
84.8#0.1
83.740.1
85.3+0.5
Generalization to Novel Environments We further test generalization by applying concept encoders trained on LIBERO-90 directly to LIBERO-GOAL demonstrations featuring unseen environments and tasks. The L90-G results show our method continues to outperform all baselines in this challenging scenario. This indicates our approach discovers fundamental manipulation primitives that transfer effectively across environmental variations.
Impact of Multi-Modal Observations Our ablation study (Tab. 2) shows that performance consistently improves as more modalities are incorporated. The most significant drops occur when removing proprioceptive information, highlighting its importance for grounding visual observations with physical interaction states and confirming the value of our cross-modal correlation approach.
4.3 Analyzing Manipulation Concept Properties
Enhanced Cross-Modal Correlation To verify our Cross- Modal Correlation Network's ' effectiveness (Sec. 3.2), we measure mutual information between modalities conditioned on concept latents (Sec. A.4). Tab. shows that our approach achieves higher conditional mutual information than the All baseline. This confirms that our mask-and- ----ddidddddc strategy enables the concept encoder capture persistent cross- modal patterns that generalize across different objects and contexts, providing robust representational basis for policies.
Table 3: Conditional mutual in- formation between modality pairs. Values conditioned on concept latents from method versus the All base- line that does not model cross-modal correlations. A: agentview, H: eye-in- hand vision, P: proprioception.
Ours
All
[ 0^
z)
3.7999
2.0080
Alignment with Semantic Sub-Goals We evaluate whether our concepts align with human-understandable se- mantics by grouping latents from different demonstrations based on human-identifin sub-goals and computing similar- ities between these groupings:
(Op 0^
z)
4.8319
3.1312
I ((pp он
z)
4.8255
3.1322
(C,, C,) |C11C
Σ E &EC: ECE
2: | || 12
(10)
where C.. C, represent human-identified sub-goal categories, and 2, 2y are latents within each category (details in Sec. C.2). As shown in Fig. 4, similarity matrices consistently show the highest values along the diagonal, demonstrating that our approach discovers concepts that exhibit clustering patterns corresponding to meaningful manipulation primitives.
Multi-Level Hierarchical Structure Varying the coherence threshold in Eq. 4 reveals the hierarchical organization of our learned concepts. Fig. (and Sec. C.5) shows larger values identify coarse-grained phases, while smaller values capture fine-grained actions. This emergent hierarchy enables policies simultaneously reason about immediate actions and longer-term goals without explicit hierarchical supervision, contributing to improved performance on complex sequential tasks that require coordinated execution across multiple temporal scales.
4.4 Real-World World Validation
Real-World Generalization Study To study generalization capabilities, we deploy concept- enhanced policies on a Mobile ALOHA robot [16] in "cleaning cup" tasks (Fig. 5). Training data includes only simple container arrangements with consistent color pairings.

---

Task: "Open the top drawer the cabinet and put the bowi it'
Discovered Manipulation Concepts Multiple Granularities
Figure 3: Multi-granular task decomposition through concept latent clustering. Visualization of sub-proce sub-processes derived by clustering manipulation concept latents different coherence thresholds (c) for the task "open the top drawer and put the bowl in it." Higher values (top rows) produce coarser decompositions, while lower values (bottom rows) yield finer-grained segmentation. The emergent sub-processes naturally align with semantic task components, for example, the third segment in row corresponds to "put bowl in drawer," while the second segment in 4 corresponds "pull drawer open." This demonstrates our method's 's ability to discover hicrarchical, human- iittaninnterretable task structures without explicit supervision.
We test on six increasingly challenging varia- tions: (1) Novel Placements: Cups and con-
reach the top drower #1
tainers in unseen arrangements; (2) Color Com-
open the top drawer
0.28
0.06
0.16
position: Altered cup-container color pairings; (3) Novel Objects: Entirely unseen containers,
0.08
0.14
0.08
put
cups, and plates; (4) Obstacles: Objects be-
reach the bettom drawer #s 0.01
tween the robot and the cups obstructing vision;
close the bettom drawer es
0.16
0.33
0.10
(5) Barriers:
Internal dividers within containers
impeding placement; (6) Grasping Together: Two adjacent cups requiring simultaneous grasp.
close
0.07
0.10
0.18
put
0.16
As shown in Tab. 4, policies enhanced with our manipulation concepts consistently outperform baselines across all scenarios, with advantages in challenging conditions. We suggest that the following two mechanisms behind learned ma- nipulation concepts improve generalization:
Figure 4: Semantic alignment of learned con- cepts. Cosine similarity between concept latents grouped by human-defined sub-goals. Diagonal patterns demonstrate that our approach discov- concepts that exhibit clustering patterns cor- responding to meaningful manipulation primitives.
Relational focus: Concept-enhanced policies
prioritize transferable relational patterns (e.g., "object inside container") over surface features. Our cross-modal correlation learning (Sec. 3.2) enables this capability by identifying patterns that remain invariant across modalities. This relational emphasis explains the stronger performance scenarios that alter visual appearance while preserving task structure. For instance, while Novel Placement tests spatial variation alone. the other protocols introduce substantial visual perturbations (different colors, objects, or occlusions) that shift the appearance distribution. The consistent performance gains across these visually diverse scenarios (Tab. 4) suggest that the learned concepts successfully capture the underlying relational invariant placing cups into containers rather than memorizing superficial visual patterns.
Hierarchical awareness: Concept-enhanced policies exhibit more systematic failure recovery than baselines, suggesting better tracking of sub-goal completion. Baseline failures frequently exhibit

---

Real Robot
Training
Test on Unseen Setups
Novel Placements
Color Composition
Novel Objects
Obstacles
Barriers
Grasping Together
Figure Real-world generalization evaluation with Mobile ALOHA robot. Left: Mobile ALOHA robot setup for cup cleaning tasks. Center: Training conditions with simple, consistent cup-container color pairings. Right: Six test variations with increasing complexity: novel placements, altered color combinations, unfamiliar objects, external obstacles, internal barriers, and simultaneous grasping of multiple cups. These variations test the policy's ability to generalize beyond training conditions by systematically introducing new challenges.
premature task abandonment: the robot moves toward containers without having grasped objects, or hovers near placement locations without executing placement. In contrast, when concept-enhanced policies fail initial grasp attempts, they consistently retry grasping (typically 2-3 attempts) before proceeding, demonstrating recognition of incomplete sub-goals. Although these recoveries ultimately fail due to time limits object displacement, they reveal structured task progression rather than blind action execution.
These mechanisms may enable manipula- tion concepts to promote policy generaliza- tion by encoding fundamental spatial and functional relationships that remain consis- tent across environmental variations. De- tails are provided in Sec. C.6.
Table 4: Real-world generalization success rates (%) for ACT policies with and without manipulation con- cepts (MC). Test conditions: Placements (novel layouts), Color (new pairings), Objects (unseen items), Obstacles (external barriers), Barrier (internal dividers), and Multi- grasp (two cups simultaneously).
Multi-Horizon Goal Prediction Visual- ization To visualize the temporal infor- mation encoded in our manipulation cepts, we examine outputs from our Multi- Horizon Goal Predictor (MHGP, F in Eq. 7) using the BridgelDataV2 dataset [60]. Fig. 7 (Sec. C.7) shows predicted goal states when conditioned on the current observation, manipula- tion concept, and various coherence thresholds (E).
Place
Color
Obj.
Obst.
Barr.
Multi
w/o MC w/ MC
53.3
46.7
40.0
20.0
0.0
0.0
73.3
60.0
53.3
33.3
20.0
13.3
The predictions capture essential task structures such anticipated arm trajectories and object interactions - rather than attempting pixel-perfect reconstructions. This abstraction scene-specific details in favor of functional relationships is crucial for cross-environment generalization. Importantly, as increases, the predictions correspond to states progressively further into the future, with smaller values showing immediate next steps and larger values revealing final goal states. This demonstrates that our learned concepts encode meaningful temporal structures at multiple time horizons, enabling policies simultaneously reason about immediate actions and longer-term objectives. Details provided in Sec. C.7.
5 Discussion
We demonstrate that self superviee discovery hierarchical manipulation concepts significantly enhances robot policy performance across original tasks, novel compositions, and entirely new environments. Three key strengths emerge: (I) our representations naturally resemble semantically meaningful manipulation primitives without requiring explicit labels, as evidenced by diagonal clustering in similarity matrices; (2) the concepts bridge low-level actions and high-level goals through hierarchical organization, enabling reasoning at multiple temporal scales; and (3) concept- enhanced policies focus on transferable relational patterns rather than superficial features, explaining their robust generalization to scenarios with substantial distribution shifts. These findings highlight the potential of learning manipulation concepts from unlabeled multi-modal demonstrations for creating more adaptable and interpretable robotic systems. Limitations are discussed in Sec. D.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the Early Career Scheme of the Research Grants Council (RGC) grant # 27207224, the HKU-100 Award, a donation from the Musketeers Foundation, in part by the JC STEM Lab of Roboties for Soft Materials funded by The Hong Kong Jockey Club Charities Trust, and DAMO Academy through the Alibaba Innovative Research Program.
References
[1] Nikolai Axmacher, Florian Mormann, Guillen Fernández, Christian E Elger, and Juergen Fell. Memory formation by neuronal synchronization. Brain research reviews, 52(1):170-18) 2006.
[2] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018.
[3] Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024.
[4] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A vision-lagguage-cction flow model for general robot control. arXiv preprint arXiv:2410.24166 2024.
[5] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[6] Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, and Ashish Kapoor. Pact: Perception-action causal transformer for autoregressive robotics pre-training. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3621- 3627. IEEE, 2023.
[7] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[8] Qingwen Bu, Hongyang Li, Li Chen, Jisong Cai, Jia Zeng, Heming Cui, Maoqing Yao, and Yu Qiao. Towards synergistic, generalized, and efficient dual-system for robotic manipulation. arXiv preprint arXiv:2410.08001, 2024.
[9] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.0413: 2023.
[10] Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Mad- dukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-LLimerien
Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe. Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp. Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang. Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chake Jachyung Kim,
11

---

Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao. Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang. Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart in-Mart in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque. Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore. Shikhar Bahl, Shivin Dass, Shubham Son- awani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davehev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng. Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma. Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yuch-Hua Wu, Yujin Tang. Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open x-embodiment: Robotic learning datasets and rt-x models, 2024.
[11] Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An unbiased look at datasets for visuo-motor pre-training. In Conference on Robot Learning, pages 1198-1198. PMLR, 2023.
[12] Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2086 2092-22922222 IEEE, 2023.
[13] Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.043350 2022.
[14] Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang. Dieter Fox, Ranjay Krishna, and Jiafci Duan. Sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation. In 7th Robot Learning Workshop: Towards Robots with Human- Abilities.
[15] Juergen Fell and Nikolai Axmacher. The role of phase synchronization in memory processes. Nature reviews neuroscience, 12(2):105-118, 2011.
[16] Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipu- lation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401 102117,,,, 2024.
[17] Scott T Grafton and Antonia de C Hamilton. Evidence for a distributed hierarchy of action representation in the brain. Human movement science, 26(4):590-61) 2007.
[18] Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
12

---

[19] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. To.5 langua-a action model with open-world generalization. arXiv preprint arXiv:2504. 16054, 2025.
[20] Zhiwei Jia, Vineet Thumuluri, Fangchen Liu, Linghao Chen, Zhiao Huang, and Hao Su. Chain- of-thought predictive control. In Forty-first International Conference on Machine Learning, 2024.
[21] Zhiwei Jia, Vineet Thumuluri, Fangchen Liu, Linghao Chen, Zhiao Huang, and Hao Su. thaoott predictive control. In International Conference on Machine Learning, pages 11789-2179.. PMLR, 2024.
[22] Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint arXiv:2302.12766. 2023.
[23] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025.
[24] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09226, 2024.
[25] Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzales Edward Grefenstette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418-3428. PMLR, 2019.
[26] Seungjae Lee, Yibin Wang. Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03181 2024.
[27] Jianxiong Li. Jinliang Zheng, Yinan Zheng, Liyuan Mao, Xiao Hu, Sijie Cheng, Haoyi Niu, Jihao Liu, Yu Liu, Jingjing Liu, et al. Decisionnce: Embodied multimodal representations via implicit preference learning. In International Conference on Machine Learning, pages 29461-29488 PMLR, 2024.
[28] Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Maniplim: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16217 2023.
[29] Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-basee task exccution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16467-16476, 2024.
[30] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 44776-44791. Curran Associates, Inc., 2023.
[31] Ruizhe Liu, Qian Luo, and Yanchao Yang. Infocon: Concept discovery with generative and discriminative informativeness. In The Twelfth International Conference on Learning Representations, 2024.
[32] Yuyao Liu, Jiayuan Mao, Joshua B Tenenbaum, Tomás Lozano-Péres and Leslie Pack Kael- bling. One-shot manipulation strategy learning by making contact analogies. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 15387-15393. IEEE, 2025.
[33] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. Conference on robot learning, pages 1113-1132. PMLR, 2020.
13

---

[34] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine I Learning, pages 23301-23320. PMIR, 2023.
[35] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint wtXiv:2210.00000000 2022.
[36] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierr Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances Neural Information Processing Systems, 36:655-677, 67755677,220 2023.
[37] Jiayuan Mao, Tomás Lozano-Péree Joshua B Tenenbaum, and Leslie Pack Kaelbling. Learning reusable manipulation strategies. In Conference on Robot Learning, pages 1467-1483, PMLR, 2023.
[38] Lucia Melloni, Carlos Molina, Marcela Pena, David Torres, Wolf Singer, and Eugenio Rodriguez. Synchronization of neural activity across cortical areas correlates with conscious perception. Journal of neuroscience, 27(11))2888 28685, 2007.
[39] Atharva Mete, Haotian Xue, Albert Wilcox, Yongxin Chen, and Animesh Garg. Quest: Self- supervised skill abstractions for learning continuous control. arXiv preprint arXiv:2407.15840 2024.
[40] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909-918, 2019.
[41] John D Murray, Alberto Bernacchia, David J Freedman, Ranulfo Romo, Jonathan Wallis, Xinying Cai, Camillo Padoa- Padoa-Schioppa, Tatiana Pasternak, Hyojung Seo, Daeyeol Lee, et al. A hierarchy intrinsic timescales across primate cortex. Nature neuroscience, 17(12):1611-1663, 2014.
[42] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 2022.
[43] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07190 02.33000.77999 07190, 2023.
[44] Karl Pertsch. Oleh Rybkin, Jingyun Yang, Shenghao Zhou, Konstantinos Derpanis, Kostas Danilidis, Joseph Lim, and Andrew Jaegle. Keyframing the future: Keyframe discovery for visual prediction and planning. In Learning for Dynamics and Control, pages 969-979. PMLR, 2020.
[45] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-languago-action models. arXiv preprint arXiv:2501.09747, 2025.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[47] Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In Conference on Robot Learning, pages 683- 693. PMLR, 2023.
[48] Seungeun Rho, Laura Smith, Tianyu Li, Sergey Levine, Xue Bin Peng, and Sehoon Ha. Language guided skill discovery. In The Thirteenth International Conference on Learning Representations.
14

---

[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[50] Dominik Schmidt and Minqi Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations, 2024.
[51] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332-1344, PMLR, 2023.
[52] Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35:22955 22968, 2022.
[53] Rutav Shah, Roberto Martin-Martín, and Yuke Zhu. Mutex: Learning unified policies from multimodal task specifications. arXiv preprint arXiv:2309.1433 14320, 2023.
[54] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awart unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
[55] Shashank Sharma, Vinay Namboodiri, and Janina Hoffmann. Multi-resolution skill discovery for hierarchical reinforcement learning. In NeurlPS 2023 Workshop on Goal-Conditioned Reinforcement Learning. 2023.
[56] Lucy Xiaoyang Shi, Archit Sharma, Tony Z Zhao, and Chelsea Finn. Waypoint- based imitation learning for robotic manipulation. In Conference on Robot Learning, pages 2195-2209. PMLR, 2023.
[57] Wolf Singer. Consciousness and neuronal synchronization. The neurology of consciousness, pages 43 52-52,225 2011.
(58) Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation. In The Thirteenth International Conference on Learning Representations, 2025.
[59] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[60] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723-1736, PMLR, 2023.
[61] Weikang Wan, Yifeng Zhu, Rutav Shah, and Yuke Zhu. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 537-544. IEEE, 2024.
[62] Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive visual learning with heterogeneous pre-trained transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[63] Thilo Womelsdorf and Pascal Fries. The role of neuronal synchronization in selective attention. Current opinion in neurobiology, 17(2): 150. 2007.
[64] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint 1tt.::2312.111111. 2023.
[65] Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pages 3536-3555. PMLR, 2023.
(66] Jonathan Yang, Dorsa Sadigh, and Chelsea Finn, Polybot: Training one policy across robots while embracing variability. arXiv preprint arXiv:2307.03719, 2023.
15

---

[67] Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofrr Nachum. Chain of thought im- itation with procedure cloning. Advances in Neural Information Processing Systems, 35:36366- 36381, 2022.
[68] Seonghyeon Ye, Joel Jang. Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Kei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758 2024.
[69] Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia. Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, al. Learning manipulation by predicting interaction. arXiv preprint arXiv:2406.00430 2024.
[70] Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu. Zhaoshuo Li, Qianli Ma. Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of- ofttouhtt reasoning for vision-languago-action models. arXiv preprint arXiv:2503 220020, 2025.
[71] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In ICML Workshop on New Frontiers in Learning. Control, and Dynamical Systems, 2023.
[72] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang. Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d vision-language-action generative world model. In Forty-first International Conference on Machine Learning, 2024.
[73] Pei Zhou, Ruizhe Liu, Qian Luo, Fan Wang. Yibing Song, and Yanchao Yang. Autoegp: Closed- loop concept-guided policies from unlabeled demonstrations. In The Thirteenth International Conference on Learning Representations.
[74] Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, (10)4126-4133. 2022.
[75] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, Yifeng Zhu, and Kevin Lin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 2020.
A Implementation details
A.1 Manipulation concept discovery (Ours)
This section details the neural network architectures and training procedures employed in our manipulation concepts discovery framework (Sec. 3) implemented on the LIBERO benchmark.
Manipulation Concept Encoder (Sec. 3.1) The manipulation concepts encoder & (Eq. 1) first encodes the multi-modal observations at each time step of the input demonstration into an encoded vector. It then utilizes self-attention transformer process the sequence of encoded vectors into a sequence of manipulation concepts. For the observation encoding process, our experiments on LIBERO incorporate two vision observations: agent-view vision and eye-in-hand vision. The original images are tensors of shape 128 x 128 x 3. To enhance processing efficiency, we preprocess the images for each time step using the VAE encoder from stable diffusion [49], compressing each image into a tensor of shape 16 x 16 x 4, which is then flattened into a 1024-dimensional vector. In addition to the vision observations, each demonstration as the proprioceptive state observation. For these three observations at each time step, we employ three distinct 2-layer MLPs process each observation into a feature vector of the hidden size (256) used by the subsequent transformer. The encoded features from these observations are then summed to form 256-dimensional representation that encapsulates the sensing information from the three modalities.
include 9-dimensional robot state each time step of
hav MLP av (Iav-compeess) hgy MLP ev
AALLPeeJee--coooo (Iov-compress) hpcop MLP peop (Sprop)
(11)
h = fay + hey + lpprop
16

---

Here, Iar-compeess represents the 1024 4i2essimesiional compressed agent-view vision, and Lev-compress represents the -llensinnsi compressed eye-in-hand vision. Sprop denotes the 9-dimensional proprioceptive state observation. The output of the hidden layers from the three MI.Ps is 1024 dimensions. The h Eq. represents the encoded observation feature at each time step of a given demonstration T5. (h). h². : The next module in & is a 12-layer self-attention (MHA in Eq. 12) transformer, enabling each time step to aggregate information from every other time step in the input sequence. In our implementation, we do not input the entire demonstration; instead, the transformer processes fixed input sequence length of Tommest embedding, represented as a tensor of shape 60 x 256, is added to the input sequence to enhance temporal representation. The hidden feature dimension at each time step is 256, and each self- attention layer consists of 8 heads. Morcover, since spherical distance is utilized in Sec. 3.3, the output manipulation concepts normalized to have unit length with respect to the 2-norm:
60. A learnable temporal
I 1) f- Norma (IMHA]x (ht, h
1+1
(12)
The output manipulation concept sequence in Eq. represents the predicted manipulation concepts at time-steps& f, + 1....... (t+ + Teomiest 1) of the demonstration T.. During training, demonstrations with lengths shorter than T comeett are padded to Teomest by repeating the observations from the last time-step at the end of each demonstration. During inference, when & is used to label the demonstrations the original dataset, the manipulation concepts at each time step are designed to incorporate information from as many future time steps as possible. This approach aims to better capture motion pattern dynamics, aligning with prior works that generate the dynamics at the current time step based on information derived from the dynamics spanning the current to future time steps [68]. Specifically:
For each time-step & < Ti - T comttexu the corresponding manipulation concepts are derived when the input to Eq. 12 starts from this time-step and spans a length of T (hξ, hitt :.... h T
For each time step > T: Toontes,, the corresponding manipulation concepts are derived when the input to Eq. 12 begins at time step h - and spans length of T cuntexts ensuring that the final time step corresponds to the end of the demonstration: (h,7 (N.T:TTonentt Touren .ha T
.h.
If the original demonstration length is smaller than Teomexs, the manipulation concepts correspond to the input appended with repeated observations as described earlier.
However, do not firmly believe this is the optimal approach for labeling manipulation concepts. Further exploration of inference-time strategy design left for future work, as it is not a core focus of the manipulation concept discovery methodology presented.
Learning Multi-Modal Features and Correlations (Sec. 3.2) The Cross-Modal Correlation Network C (Eq. 3) shares similar structure with E (Eq. 1). First, it includes four 2-layer MLP encoders, analogous to the three encoders in Eq. 11, with an additional encoder for processing the manipulation concepts. Each of these four MLPs outputs a hidden feature of dimension 1024, which is then reduced to a 256-dimensional encoded feature. These encoded features are summed to represent the combined information from the three observations and the manipulation concepts. Second, it incorporates a 4-layer self-attention transformer to process the sequence of features (with the same fixed length Toomeest 60) produced by the four MLPs. Following this, three 3-layer MLP decoders map the transformer's 's output to the reconstructed observations at each time step. Unlike in Eq. 12, the transformer's output does not require normalization. Each decoder MLP has hidden layers with a dimension of 1024. As described in Eq. 3, for the three observations agent- camera vision, eye-in-hand camera vision, and proprioceptive state observation we randomly mask these modalities, ensuring that at least one modality is masked during each iteration. The 23 = 7 possible masking scenarios follow uniform distribution, with each scenario appearing with a probability of 7. For the sampled masks, all observations of the corresponding masked modalities in the input sequence are replaced with zero tensors. The loss is applied separately to the reconstruction of the three different observations. Specifically, L.2 loss is applied to the two vision observations, while LI loss is applied to the proprioceptive state observations. The loss weight Amm in Eq. 8 is set to 1.0.
Learning Hiatarciiraa Sub-goals (Sec. 3.3) The Multi-Horiizon Future Predictor F (Fq. 7)
shares a similar structure with C (Eq. 3). The key differences are:
17

---

F does not require a masking strategy.
The transformer in F is 4-layer causal self-attention transformer. Causal attention used because, in Eq. 7, the prediction is made from each current time step to certain future time steps. Therefore, for each time-step input in F, access information from subsequent time steps restricted.
To incorporate the granularity parameter E [0, 1], we discretize the continuous range into 1000 uniform bins {0.000, 0.001,... 0.999) and learn a corresponding VQ-VAE codebook [59] with 1000 entries, each represented block, the ferward layer receives the concatenation the attention output and the embedding corresponding to the sampled value.
256-dimensional embedding vector. In each transformer
The output predictions correspond to the observations at the time steps determined by the rules described in Sec. 3.3 (Eqs. 4 and 6). Still, the loss is applied separately to the reconstruction of the three types of observations. Specifically, 1.2 loss is used for the two vision observations, while L. loss is applied the proprioceptive state observations. The loss weight Amh in Eq. is set 1.0.
Training Details We train manipulation concept discovery process for 200,000 iterations with batch size of 512. Each item in the batch is segment of demonstration with fixed length of Toontext 60. The training process uses the Adam optimizer with weight decay of 0.001 and momentum parameters 81 0.9 and 32 0.95. The base learning rate is set to 0.001. Initially, the model is trained with a 100-iteration warmup phase, during which the learning rate increases linearly from 0.0001 to 0.001. After the warmup, the model is trained for the remaining iterations using cosine decay schedule, gradually reducing the learning rate back to 0.0001. This training setup is compatible with GPUs such the GeForce RTX 3090 4090. However, leverage the A800 GPU for improved efficiency, completing the training process in 1.5 days.
A.2 Enhancing Imitation Learning
action
mani. concept
Agent view
head
Transformer
Transforee Encoder
Eye in-hand View
Dacoder
layers
CNN PosEmmb
Joints
Task
Pos. emb. (fooed)
Var.
Observation
task
Linear
Liner
Lineer
Linear
Linear
FILM
Oct
+-9
++B
O
+
Oir
+0
Orr
+
conv
conv
head
Bottlle Neck
Concept
seq.
From
conv
conv
head
mani. Concept
Figure 6: Upper: Enhanced ACT (decoder part); lower: Enhanced Diffusion Policy
18

---

This section introduces the neural network architectures and training details used in the Enhancing Imitation Learning process (Sec. 3.4). focuses on modifying the original neural network policy to enable the prediction of manipulation concepts, thereby enhancing performance. Moreover, the implementation of the base policy follow [39].
ACT The pipeline (we focus on the CVAE decoder it is the only modified component) is shown in the upper part of Fig. 6. Following [39]. the transformer encoder in ACT's CVAE decoder is modified to incorporate task embeddings provided by CLIP. The transformer decoder in ACT's CVAE decoder is adapted to predict manipulation concepts. Specifically, the output of the I-tth layer in the transformer decoder is processed by additional decoding head, which is nearly identical to the one used for outputting action chunks, with the only modification being the output dimension. This decoding head outputs manipulation concept chunks corresponding to the same time steps as the action chunks, with its parameters adjusted to match the dimensionality of the manipulation concepts at each time step (256). Other training and testing settings follow [39]. Moreover, the transformer decoder in ACT's CVAE decoder, as implemented by [39], consists of 7 layers. During our experiments, we tested various combinations of L-th layers determine the optimal layer for processing by the manipulation concept decoding head. Our results indicate that L = 2 provides slightly better performance than other configurations. We present the ablation study on L and the weight Ame in Eq. 9 for ACT on LIBERO-90 tasks, as shown in Tab. 5 ! However, we believe this raises an interesting and challenging direction for future work: systematically investigating the rationale and insights behind the selection of L,, even beyond the context of our setting.
Table 5: Ablation study on the intermediate layer outputs (L) used as inputs to the manipulation concept decoder and the loss weight Ame in Eq. 9 for Enhancing Imitation Learning in ACT, evaluated on LIBERO-99 90 tasks.
ACT
Ame = 1.0
Amm = 0.1
Ame = 0.01
Ame 0.001
L = 2
74.8+0.8
70.6+0.8
69.0+0.1
68.7.0.5
L = 3
70.0+0.4
69.9+0.2
68. 68.8+1.0
68.7+0.6
L = 4
72.6+0.5
69.9+0.3
69.6+0.2
67.3+0.5
Diffusion Policy The pipeline is illustrated in the lower part of Fig. 6. Following [39], the convolution-based Diffusson Policy is modified to concatenate the noise level (k: and corresponding <x) embedding, observation, and task embedding as the conditional input to the diffusion model network, using the FiLM strategy. We further introduce an additional manipulation concept decoding up- pp-mmmllnn module, nearly identical to the one used for outputting action chunks, with the only modification being the output dimension, to decode intermediate outputs from the corresponding up- sampling layer of the diffusion model. This decoding head can be configured to process intermediate outputs to predict manipulation concept chunks corresponding to the time steps of the predicted (noise of) action chunk outputs. The figure illustrates the cases for T, 0 and T. = 1. During our experiments, we tested various combinations of I.-th layers to identify the optimal layer for processing by the manipulation concept decoding head. Our results suggest that I, achieves better performance than other configurations. We present the ablation study on L and the weight Ame in Eq. 9 for Diffusion Policy on LIBERO-90 tasks, as shown in Tab. Similar to ACT, we believe this topic needs further systematic study to uncover deeper insights. Other training and testing settings follow [39].
Table 6: Ablation study the intermediate layer outputs (L) used as inputs to the manipulation concept decoder and the loss weight Ame in Eq. 9 for Enhancing Imitation Learning in Diffussion Policy, evaluated on LIBERO-90 tasks.
DP
Ame = 1.0
Ame = 0.1
Ame 0.01
Ame = 0.001
L L=
0
83.5#0.8
78.9+0.4
78.7+0.3
75.6+0.6
L=1 80.0+0.4
89.6+0.6
82.0#0.2
79.9+0.1
'We provide sample rollouts (uupplemeettrryy/oooooot..tuc summary and gif logs of tist-time ACT and DP performance http://rrrrrroooottteeeeeeeeeeea in the supplementary materials.
19

---

Our future work includes a deeper study of modification strategies for various policies to adapt to the Enhancing Imitation Learning framework, following the methodology outlined in Sec. 3.4.
A.3 Manipulation concept discovery (Baselines)
InfoCon. Based the design of InfoCon [31], All the size of hidden features output by trans- formers and concept features is 256. The state encoder (also process video clips consisting of concatenated, compressed vision observations and proprioceptive states, outlined in Sec. A.I) uses a 12-layer transformer. The state reconstructor uses a 4-layer transformer. The goal-based policy uses a 4-layer transformer. The predictor for the generative goal uses a 4-layer transformer. For
used for discriminative goals, we use 2 hidden layers in the goal function. The number of concepts is fixed, the maximum number of 30 manipulation concepts for all the tasks. We employ the Adam optimizer, coupled with a warm-up cosine annealing scheduler same as Sec. A. The weight decay is always 1.0 x 10-3, We use batch size of 512 during training. We train our model for 200,000 iterations with a base learning rate of 1.0 x 10-3 on a single A800 GPU within 1.5 days.
XSkilI. Following the design of XSkill [65], we implement its skill discovery framework on LIBERO-90, focusing exclusively on the "robot" embodiment and the Skill Discovery component from the XSkill pipeline. To ensure comparable model capacity and support multi-modality, our implementation employs a 12-layer Transformer as the temporal skill encoder. This encoder processes video clips consisting of concatenated, compressed vision observations and proprioceptive states, as outlined in Sec. A. along with a trainable token to predict skill representations, which are subsequently used for skill prototype prediction. To augment the concatenated video clips containing multi-modality information, Gaussian noise with a = 1.0 10-3 3 is applied. This unified augmentation approach accommodates the nature of proprioceptive states, as standard image augmentation techniques are not directly suitable for robotic proprioception. The training process employs a batch size of 512 and learning rate of 1.0 x 10-3 for 200,000 iterations single A800 GPU within 1.5 days. DecisionNCE We fine-tune the DecisionNCE-T model hattps://githin
com/2toinf/ DecisionNCE) on our dataset, as it outperforms DecisionNCE-P in our analysis of the exper- imental results in [27]. We use two types of language annotations: (1) the original task descriptions (tccisionttask), and (2) detailed subtask labels derived by decomposing each task into meaningful subprocesses (Decision-moties motion)). To construct the latter, we manually segment each demonstration based on changes in the robot's proprioceptive state (e.g., movement direction, gripper open/close status). Segments corresponding to the same task are then assigned unified subtask labels across demonstrations, with remaining inconsistencies resolved through manual adjustment.
RPT. We modify the original RPT design [47] adapt it for our task of discovering manipulation concept latents in the LIBERO-90 setting. We employ a 16-layer self-attention transformer to process inputs consisting of 60 consecutive, interleaved agent-view and eye-in- in-d vision frames. Vision inputs are compressed using stable diffusion VAE encoder, similar to the method in Sec. A.I. The total sequence length processed by the transformer 60 3 180. Each modality is mapped to 256-dimensional embedding vector using an MLP, as defined in Eq. 11. The transformer's 's output then decoded to reconstruct the original inputs using 3-yeyrr MLP with 1024-dimensional hidden layers. We follow the masking strategy outlined in [47] to perform temporal MAE training for the transformer. To label manipulation concept latents using the trained transformer, we extract the intermediate output of the 12th layer when the input consists of the full observation without masking. Notice that we select the output at the proprioceptive state input positions of the transformer to represent the manipulation concept latent at each time-step. The labeling process follows the procedure introduced in Sec. A. For training, we use a batch size of 512 and a learning rate of 1.0 10 running for 200,000 iterations a single A800 GPU, which completes within days.
All. This is ablation version of our manipulation concept discovery method, focusing on the design for capturing multi-modal correlations (Sec. 3.2). Specifically, this baseline replaces the loss in Eq. 3 with a loss that does not use partial masking but instead always masks all modalities:
Can (t, T5) = C (0) = 0.) of Based on our reasoning and the experiment results show in Tab. 3, we think this method may not be good at learning correlation between different modalities.
Other settings follow Sec. A.
20

---

Next. This is an ablation version of our manipulation concept discovery method, focusing on the design for representing multi- hoaliio subgoals (Sec. 3.3). Specifically, this baseline replaces the loss in Eq. 7 with loss that always predicts the next adjacent time-step observation: Liest (4, TT)
[ 2 2 o II We observe that this setting is commonly used in recent works [7, 68). which learn representations based on adjacent time-step observations or observations separated by a fixed time horizon. We suggest that learning based on a fixed time horizon is conceptually similar to adjacent time-step settings, as the fixed time horizon can interpreted as unified time step. Our method differs by considering the temporal correlation across multiple variable horizons, which is also addressed by baseline methods like RPT. Other settings follow Sec. A.I.
CLIP. To ensure compatibility with other baselines, which have output dimension of 256, we select the ViT-B/32 CLIP model from the original source (https: pattps://gre ht///gttuut com/ openai/ CLIP). This model outputs a 512-dimension feature vector, the closest to 256 among the accessible CLIP models from this codebase when given an image.
DINOv2. To match the output dimension of 256 used by other baselines, we select the dinov2-small DINOv2 model from the source at https:/ https://uuggingfat manggingaace, co/ co/facebook/ = This model produces 384-dimensional feature vector when given image.
Note that NCC CLIP, and DINOv2 baselines use only vision (and language) information for concept discovery. We preserve their original modality structure rather than adapting them to include proprioceptive states, as this would deviate from their pretraining foundations.
A.4 Mutual information estimation
The estimation of mutual information is based on MINE [2], which uses batchwise samples drawn from joint distribution and employs neural network to estimate the mutual information. To extend this approach for estimating conditional mutual information (CMI), we reformulate CMI by decomposing it into mutual information terms, as shown below:
T(X : Y 2) = T(X Y) + T(XY: 2) T(X: 2) - T(Y Z),
(13)
where XY denotes the random variable sampled from the joint distribution of X and Y and is represented as the concatenation of their encoded vectors. The neural network in MINE has two layers, with the hidden layer size set to 1.5 times of the dimensions of the two random variables.
B
Pseudocode
Here we provide pseudocode for (i) Deriving subprocess from manipulation concept latents (Alg. 1). (ii) Manipulation concept disocovery training process of our method (Alg. 2).
C
More Study on Learned Manipulation Concepts
C.1 Additional Experiments on Enhanced Imitation Learning
Sampling Strategies In this part, we focus on methodology for deriving hierarchical structures from learned representations (Sec. 3.3). While we adopt threshold-based hierarchy derivation method (Eq. 4) as a proof of concept, we acknowledge that alternative derivation methodologics warrant further investigation (see Sec. D). For the threshold-based approach, we employ uniform sampling of the threshold e during training. This choice ensures full coverage of all possible hicrarchical structures, as we do not know priori which threshold values might be suboptimal. To validate this design choice, we conduct an ablation study comparing different sampling strategies for c in Eq. 7:
As shown in Tab. 7, uniform sampling currently achieves the best performance across both policy architectures. We hypothesize that while task-specific sampling strategies might excel on particular subsets, uniform sampling provides robust performance across diverse tasks due to its comprehensive coverage of the threshold space. Future work could explore adaptive sampling strategies tailored to specific task distributions.
21

---

Table 7: Sampling Strategies Ablation. We compare different sampling strategies for in Eq. Manipulation concepts are learned from LIBERO-99 and applied to policy learning on LIBERO- We report success rates (%).
Sampling Strategy
Description
ACT
DP
Uniform (Ours)
e~ (r-U(0,1)
74.8+0.8
89.6+0.6
Sparse
(..1, 11.0}
67.640.5
81.1+0.8
Biased
c (3313) 65.6+0.7
78.7+0.4
Learning Methodology Contribution We conduct an ablation study isolate the contributions of our two core learning methodologies: Capturing Multi-Modal Correlations (Sec. 3.2) and Represent- ing Multi-HHrizzon Sub-Goals (Sec. 3.3). Tab. compares three configurations: (1) Cross-modar only: learning with only cross-moda alignment objectives in Eq. 3. (2) Multi-horizon only: learning with multi-horizon sub-goal prediction in Eq. 7 but without cross-modal alignment, and (3) Full method: combining both cross-modal alignment and multi-horizon prediction.
Table 8: Methodology Contribution Ablation. We evaluate the contribution of each learning component by training manipulation concepts on LIBERO-90 and applying them to policy learning on LIBERO-90. We report success rates (%).
Method
ACT
DP
Cross-modal only
69.1=0.6
82.8+1.0
Multi-horizon only
71.6+0.4
80.5+0.5
Ours (Full method)
74.8+0.8
89.6+0.6
The results in Tab. 8 reveal that both components make substantial and complementary contributions to performance. We attribute this synergy to the distinct roles of each component: cross-modal aligmment grounds the understanding of correlations across different modalities, while multi-horizon prediction captures hierarchical temporal structure. Together, they enable the learning of manipulation concepts that are both correlationally coherent and temporally structured, leading to more robust policy learning.
Data Constraint Experiments We evaluate whether manipulation concepts can help mitigate the challenges of imitation learning under limited data. Specifically, we vary the amount of data available for training both the manipulation concept encoder (Eq. 1) and the enhanced imitation learning framework (Sec. 3.4) to assess their impact on policy success rates. We conduct experiments on LIBERO-90 tasks using the diffusion policy. As shown in Tab. 9, incorporating manipulation concepts consistently improves policy performance compared to settings without them, even under restricted data conditions. This demonstrates that learning and leveraging manipulation concepts can make imitation learning more data-efficient and effective.
Table 9: Performance under data constraints. Success rates of diffusion policies with and without manipulation concept enhancement, evaluated on LIBERO-90 (L90-90). In each setting, the number of demonstrations per task available for training both the manipulation concept encoder and the policy is limited indicated.
50 demos/task
25 demos/task
10 demos/task
Ours
89.6 + 0.6
77.6 + 0.5
61.2 1.1
Plain
75.1 0.6
70.1 # 0.3
59.1 0.9
Distance Metric We conduct an ablation study comparing spherical distance and cosine distance
1- cos (+) for dist(:,)) -) in Eq. 4. Tab. 10 reports the performance when concepts are learned and applied
22

---

to LIBERO-90 tasks. Further investigation into distance-threshold based subprocess derivation methods represents a promising direction for future work.
Table 10: Ablation study on distance metrics for concept learning on LIBERO-90. Spherical distance consistently outperforms cosine distance across both baseline methods.
Cosine Distance
Spherical Distance (Ours)
ACT
67.8+0.5
74.8+
DP
82.0+0.4
89.6#0.6
Sub-process Derivation We conduct an ablation study comparing two approaches for constraining manipulation concept latents within each sub-process in Eq. 4. Our proposed method enforces proximity among all concept latents throughout the sub-process ("Sequential Constraint"), while the baseline only constrains the distance between the initial and final concept latents ("Endpoint Constraint"). We evaluate both approaches on LIBERO-90, where concept discovery and policy enhancement are performed. Tab. 11 reports the task success rates when integrating the learned manipulation concepts with different policy architectures.
Table 11: Ablation study on sub-process derivation constraints. We compare enforcing proximity among all manipulation concept latents within each sub-process (Sequential Constraint) versus constraining only the initial and final latents (Endpoint Constraint). Results show average success rates (%) with standard errors across LIBERO-90 tasks.
Sequential Constraint
Endpoint Constraint
ACT
74.8=0.8
68.4+0.8
DP
89.6#0.6
79.8+0.5
Future Prediction Strategy Apart from the different sub-goal determination strategies we com- pared (Next and InfoCon in Sec. 4.1), evaluate two additional future prediction strategies.
Next-n. Unlike our sub-process derivation strategy (Eq. 4), this baseline encodes future observations varying time horizons by randomly sampling future timestep. Specifically: Lnext-o (t, T2)
En-. [F (o), 2f. θy) o o
Next-random. This strategy builds upon Next-n but differs in how future targets are selected. We first randomly segment training demonstrations into sub-processes for concept discovery. Then, for state at time-step I, the prediction target is randomly selected from among the end-states of subsequent sub-proces sub-processes. For example, if demonstration is segmented into sub-processes and time-step & is in the 2nd sub-process, the model will randomly predict one of the end-states from the 2nd, 3rd, 4th, or 5th sub-processes during concept discovery learning.
We evaluated diffusion policies enhanced by these strategies, with results presented in Tab. 12. The data demonstrates that our manipulation concepts yield better policy enhancement compared the alternative strategies. This highlights the importance of carefully designing which future observations to predict and validates the effectiveness of our self-supervised sub-goal derivation and learning method. Specifically, the performance decrease observed with Next-n and Next-random, despite their consideration of multi-horizon futures, likely stems from the fact that not future states effectively represent sub-goal completion. Intermediate movement states may be reached through multiple alternative trajectories that ultimately achieve the same sub-goal, thus providing limited information about the underlying task structure.
Usage of Manipulation Concept Encoder We investigate two strategies for leveraging the ma- nipulation concept encoder from Eq. 1 in downstream policy learning. The encoder serves as an intermediate module that extracts manipulation concept representations from demonstrations. We compare the following approaches: (1) Direct Conditioning: The trained encoder directly processes current observations to generate manipulation concepts, which are then concatenated with observa- tions as additional input features to the policy network. (2) Joint Prediction (Ours): The policy
23

---

Table 12: Comparison of Additional Future Prediction Strategies. Success rates of diffusion policies enhanced with manipulation concepts discovered using our method versus two alternative future prediction strategies on the LIBERO-90 benchmark.
L90-90
Ours
Next-n
Next-random
DP
89.6 0.6
83.040.3
82.8 0.4
network trained to jointly predict both future actions and future manipulation concepts from current observations, as described in Sec. 3.4. Tab. 13 presents the comparative results across two policy
architectures.
Table 13: Comparison of Manipulation Concept Usage Strategies.
Policy Direct Conditioning
Joint Prediction (Ours)
ACT
71.120.4
74.8=0.8
DP
79.3+0.9
89.6#0.6
The performance gap stems from a temporal alignment mismatch between concept representations and action predictions. In Direct Conditioning, the encoder extracts concepts from current or past observations, creating temporal lag: the policy receives historical concept information when planning future actions. In contrast, Joint Prediction enforces temporal coherence by training the policy to predict future manipulation concepts alongside future actions, ensuring that the predicted concepts align temporally with the planned action sequence.
This temporal alignment critical in multi-phase manipulation tasks. For example, consider pick-and-place scenario: immediately after grasping an object, the current observation encodes grasping-related dynamics. However, to execute the subsequent placement action, the policy requires placement-relevant information. Joint Prediction learns to anticipate these future task-phase concepts, providing the policy with forward-looking contextual information. Direct Conditioning, by contrast, conditions the policy backward-looking grasping concepts that offer limited guidance for placement planning.
While our results demonstrate the advantages of temporal alignment through joint prediction, we acknowledge that direct conditioning on historical concepts may benefit tasks requiring explicit long-horizon memory or reactive behaviors based on past states [14]. Future work will explore hybrid architectures that combine both strategies.
C.2 Alignment with Semantic Sub-Goals
We evaluate whether the manipulation concept latents learned by our method resemble human- interpretable semantics. Specifically, assess whether latents assigned to time steps of demonstra- tions (Sec. 3.1) exhibit higher pairwise similarity when those steps belong to sub-processes pursuing the same human-defined sub-goal.
To analyze the learned representations, we first group manipulation concept latents according to sub-goals. For instance, in the task "open the top drawer", latents from time steps where the robot reaches for the top drawer handle are categorized as "reach the top drawer". Latents from other demonstrations and tasks involving identical processes (reaching the top drawer) placed in the same category. We then quantify the similarity between two categories by calculating the average cosine similarity between their respective latents, defined in Eq. 10.
Fig. 9 shows results from analyzing demonstrations from three tasks:
Task #1: Open the top drawer of the cabinet and put the bowl in it;
Task #2: Close the bottom drawer of the cabinet and open the top drawer;
Task #3: Close the top drawer of the cabinet and put the black bowl top.
We selected these tasks because they clearly demonstrate overlapping subgoals across different tasks (c.g., Task #1 and Task #2 both include "opening the top drawer"). This enables testing whether the

---

latents capture similar subgoal semantics across different tasks an essential capability for cross-task learning efficiency (Sec. 1). Manipulation concept latents are grouped based on demaandeeiind sub-goals, with similarities between category pairs visualized as heatmaps. Three heatmaps are presented, each using different granularity of sub-goal annotation:
1. Top-Ist heatmap: Omits task-specific distinctions, merging similar manipulation processes across tasks into same category
2. Top-2nd heatmap: Further merges similar manipulation processes, disregarding distinctions like
"top drawer" versus "bottom drawer"
3. Top-3rd heatmap: Consolidates manipulation processes further, treating actions like bowl
transitions as the same concept regardless of context
In each heatmap, the entry at position (i.j) represents the average similarity (x (x10.0) between categories and J. For readability, only the top three similarity values in each row are displayed.
We emphasize that testing semantic capture at different "description granularity levels" is important because semantics naturally exist multiple levels abstraction, from highly specific details to broadly generalizable patterns. Finer- Finer-grained descriptions provide more precise details but limited generalization, while coarser-grained descriptions capture more general features applicable across diverse scenarios. For example, the general instruction "close the drawer" applies broadly to subpro- cesses both Task #2 and Task #3, whereas the more specific "close the top drawer" incorporates spatial features that make it applicable in Task #3 but not Task #2. Through this multi auuti-gramularity analysis, we evaluate whether our manipulation concept latents successfully capture both fine- eeeigra semantics needed for specific scenarios and coarse- graeeerrrndddddeee semantics that enable transfer across more scenarios.
What we observe is that the highest similarity values consistently appear along the diagonal in each heatmap in Fig. 9, so concept latents from the same category show higher similarity compared with different categories. This indicates that the learned latent clusters resemble clusters derived from human-interprettble sub-goal classifications, suggesting that our model captures meaningful semantic structure in the manipulation processes. Moreover, the patterns observed across the three heatmaps with different description granularities reveal that the latents encode semantics at multiple levels of abstraction. They capture both generalizable semantics applicable across tasks and scenes, while simultaneously preserving fine-grained scene-specific details.
Furthermore, Fig. 10(b) provides t-SNE visualization of manipulation concept latents from all 90 tasks in LIBERO-90. For each task, latents (=²) were extracted at every time step of demonstrations. In the plot, latents are color-coded by their originating tasks. We observe that clusters often contain latents from diverse tasks, as indicated by the mixed colors in each cluster. This further supports our finding that the learned latents generalize across tasks and capture shared semantic structures.
C.3 Motion Study
We evaluate whether the learned manipulation concept latents capture the robot's motion. Using Eq. 10, we calculate the average similarity (x 100.0) between movements based manipulation concept latents corresponding to specific gripper actions. Specifically, we collect latents for the following movements from task demonstrations in LIBERO-90:
Forward-backward motion: Latents for time-steps where the robot moves forward, backward, or remains still along the
axis.
Left-right motion: Latents for time-steps where the robot moves left, right, or remains still along the left-right axis.
Up-down motion: Latents for time-steps where the robot moves up, down, or remains still along the up-down axis.
Gripper state: Latents for time-steps where the gripper opens or closes.
It should be noted that t-SNE performs extreme dimensionality reduction, so these clusters may not perfectly reflect similarity in the high digeedimnnsional space. This visualization should therefore be considered as supplementary evidence.
25

---

Movements with velocities below 20% of the maximum observed velocity are classified as "still". Using these collected latents, we generate heatmaps (similar Fig. 9) to visualize the average cosine similarity across different movement directions and gripper states (Fig. 10(a)).
The heatmaps reveal that the highest cosine similarity values often appear along the diagonal. This demonstrates that latents corresponding to the same motion patterns exhibit greater similarity to each other than to those from different motion patterns, indicating that the latents effectively capture different movement directions and gripper states. However, we observe that forward-backwer motion captured with lower accuracy compared other dimensions. We hypothesize that incorporating additional informatiive modalities, such as depth maps, beyond the current proprioceptive states could improve the representation of motion along the of such modality incorporation to future work.
backward axis. We leave the exploration
C.4 Diversity& & Discrimination Study
We analyze the diversity and discriminability of learned manipulation concepts by comparing concept latents from our method (Sec. 3) and the baselines in Manipulation Concept Discovery Baselines (Sec. 4.1). Specifically, we cluster latents from these methods and examine the number of clusters under varying granularities. The number of clusters reflects concept diversity: more clusters indicate a wider variety of concepts. Clustering granularity determines whether clusters are fine-grained (fine granularity) general (coarse granularity). Additionally, small granularity perturbations test discriminability, as less discriminative latents lead to significant clustering changes under small granularity variations. For each method, We collect manipulation concept latents from 90 LIBERO- 90 tasks (one demonstration per task) and use DBSCAN to cluster them while varying the density parameter Eps, which controls clustering granularity. Fig. shows the cluster counts across different Eps values. From Fig. 11, our manipulation concept discovery method (Ours) demonstrates two key advantages: 1) At higher granularities (Eps > 0.2), Ours maintains higher number of clusters. 2) The decline in cluster count is relatively smooth and gradual, showing stability under small Eps changes. These results highlight the superior diversity and discriminability of our manipulation concept discovery method.
C.5 Multi-Level Hierarchical Structure
In Fig. 3, we present a visualization example of the Multi-Level Hierarchical Strueture described in Sec. 4.3. Additional visualization results are available in the supplementary materials under the directory supplementary/eet _atptttrrrvviiiii____. multi_h
C.6 Real Robot Experiments Details
Training Data. As shown in Fig. the training data for the "cleaning cup" task consists of demonstrations using mobile ALOHA [16] to place the cup from the table into the container. Each demonstration features a scene containing exactly one cup and one container. There are two pairings of color combinations: blue cups with green containers and yellow cups with pink containers. For each pairing, we collect 27 demonstrations with varied spatial arrangements.
Evaluation Setting. For evaluation, we test our model on six scenarios that introduce variations absent from the training data:
Novel Placements. Objects maintain the same color pairings as training but appear in previously unseen spatial arrangements.
Color Composition. We rearrange color pairings (blue cups with pink containers and yellow cups with green containers) to test generalization across color combinations.
Novel Objects. We introduce unseen objects, such as bamboo- woven containers, pink cups not present training, or cups initially placed on plates rather than directly on the table.
Obstacles. We position obstacles in front of cups to challenge visual perception. Barriers. We place a plate inside the container, requiring the robot to lift the cup high enough to clear this barrier when depositing it.
Grasping Together. We position two cups adjacent to one another, requiring the robot to grasp both simultaneously at their contact point and deposit them together in the container.
26

---

Current Observation
2
Multi-harizon Subgoal Prediction with Variable Coherence Thresholds
F
Figure 7: Multi-horizon goal prediction with learned manipulation concepts. Visualization of future states predicted by our Multi-Horizon Goal Predictor (MHGP, Eq. 7) when conditioned on the current observation, manipulation concept latent (2), and varying coherence thresholds (e). From left to right, as increases from 0 to I, predictions extend progressively further into the future, demonstrating how our manipulation concepts encode temporal abstraction multiple horizons. Note that predictions capture essential functional relationships (robot-object interactions) rather than pixel-perfect reconstructions, facilitating generalization across environments.
Manipulation Concept Discovery. The model architecture and hyperparameter configuration for manipulation concept discovery follow the methodology described in Sec. A.1. Since the dataset relatively small, we adapt smaller transformers: a 4-layer concept encoder (E, Eq. 1), 4-layer Cross-Modal Correlation Network (C, Eq. 3), and a 4-layer Multi-Horizon Future Predictor (F, Eq. 7). For data collected using mobile ALOHA [16], we incorporate the following modalities: three 640 x 480 resolution cameras (left-gripper, right-gripper, and upper-gripper) and 42-dimensional proprioception states (comprising 14-dimensional joint torque, position, and velocity measurements). All image data undergoes preprocessing as detailed in Sec. A.1.
Enhancing Imitation Learning. Please refer ACT section in Sec. A.2.
C.7 Multi-Horizon Goal Prediction Visualization
We provide visualization results of the Multi-Horizon Goal Prediction Visualization (Sec. 4.4) in Fig. 7 and supplementary materials under the directory suppleementaryyprediittooo Below are the details of the experiments:
Dataset. For our experiments, utilized the BridgeData dataset [60]. Since multi-view data is not universally available across all demonstrations, we selected two specific modalities: the robot's proprioceptive states (7DoF) and the third-person camera view. The camera images were preprocessed to 128 x 128 resolution following the procedure outlined in Sec. A.2.
Manipulation Concept Discovery. We implemented the model architecture and hyperparameter configuration detailed in Sec. A.1, adapting specifically to operate with the two modalities described in the Dataset section above.
C.8 Preliminary VLA Integration
We present a preliminary exploration of integrating manipulation concepts with vision-language- action models (VLAs). We build upon OpenVIA--OF [23], which fine-tunes OpenVL.A using pretrained parameters and a novel action adapter for downstream tasks. The action adapter processes hidden layer features from the original pretrained VLA model. Following this architecture, introduce an additional "concept adapter" that implements the method described in Sec. 3.4, enabling the integration of manipulation concepts into the VLA.
To evaluate the data efficiency gains from manipulation concepts, we fine-tune the enhanced VLA on 50% of the training data used for LIBERO- 10 tasks in the original VLenVLA-OP study [23]. We compare fine-tuning performance with and without manipulation concept integration. Fig. presents the results, where the x-axis indicates training epochs and the y-axis shows success rates for checkpoints each epoch. The solid lines labeled "best" represent the highest success rate achieved up to that epoch.
27

---

The results demonstrate that manipulation concepts improve data utilization. With only half the training data, the concept-enhanced approach consistently achieves higher success rates throughout training. Notably, the original OpenVI.A-OFT achieved 94.5% success with the full dataset [23], while our concept-enhanced model with half the data reaches comparable performance levels, indicating substantially improved data efficiency.
90.2
90
81.6
Success Rate (%)
80
70
60
50
With manipulation concepts
40
Without manipulation concepts
With manipulation concepts (best)
30
Without manipulation concepts (best)
20
10k
20k
30k
40k
50k
60k
70k
80k
90k
100k
110k
120k
130k
140k
150k
Training Epoch
Figure 8: Data efficiency comparison on LIBERO-10 10 tasks with 50% training data. Solid lines
show best performance up each epoch for models with and without manipulation concepts.
We hypothesize that this improvement stems from HiMaCon's 's ability to capture manipulation dynam- ics at multiple abstraction levels. The learned concepts provide explicit intermediate representations that bridge high-level task instructions and low-level control actions, thereby reducing the learning burden on VLAs by supplying structured manipulation knowledge rather than requiring learning of complex sensorimotor patterns from scratch. Further investigation of this integration will be pursued in future work.
D Limitations & Future works
Further Exploration of multi-modality. We propose enhancing robotic data collection with richer modalities and studying how these modalities can derive more effective manipulation concepts. While current robotics research primarily focuses on visual information, human manipulation relies on multiple sensory inputs, particularly tactile feedback to complement vision. This is especially crucial for robotic systems with limited tactile capabilities. Future work should investigate which modalities contribute most significantly to performance improvements and how to fully leverage their potential.
Further Exploration of multi-horizon sub-goal. Our work proposes methods derive sub- processes for achieving sub-goals across multiple horizons, though several improvements remain possible. Current methods inadequately capture relationships between different values of in Eq. failing to reflect the natural tree structure hierarchical sub-goals. Future research could explicitly derive tree structures [61, 74] where long-horizon sub-goals as parent nodes to short-horizon child nodes. Additionally, our cosine similarity approach for determining sub-goal correspondence could be refined with more sophisticated metrics.
Scaling up. Computational constraints have limited our exploration of how our method scales with larger datasets. We plan leverage pretrained multi-modal foundation models, adopting structures inspired by [7] and extending pretraining beyond robotics data as in [68]. We also aim to further investigate whether our manipulation concepts enhance advanced policies like Vision Vision-Language- Action models [4, 19, 23, 24].
28

---

Algorithm Derive Subprocess h(z;;e) e)
Input: manipulation concept vectors {={) t=1+ coherence parameter E 10.
T:
Initialize: End = []. 96 =
while gb S T do
9e = +
while true do
if 3u 19b. s.t. , 2 or T, then
break end if'
ge = 9c +1
end while End.append (196. 9c)) 9b 9e
end while
Return End
Algorithm 2 Manipulation Concept Discovery Training (one demonstration per batch)
Input: demonstrations т; E D, where = {(o of 2, ..... )
Initialize: Manipulation concept assignment encoder E(;
Initialize: Modality Correlation Learner C(=; Oz). Subgoal Learner F(;;O,) while true do for T, in D D do
(z) ... 2 - (lo)" ...,0; 0g M, (of ,..., 0 M.2).... (o !o ..... of M.T.); 03) 08))
while True do Randomly generate a tuple (mi, .2...., mM). where m, E (0,1)
if EM m, M then
break end if end while
(6 ................................ N
M. TE + (lot" mm, 21 my..... of M. m.s 2/)T= 0.)
2.m = Σ T.
1=1 ΣM clom o.m--1
e ~~ U()
End li(z!.... .... ,
T: (Alg. 1)
for to to do
min (1ge (ab, E Enn, > 13 (Ti})
end for
(o.
, ) +- FF ((o
of 2t
Mit at (1
y...
Lmh = Σ
Ti MM 1l0 0, m.
end for end while
29

---

reach the top drawer 0.04
0.01 0.02
open the top drawer #2
0.28
0.08
0.16
grasp the bowl #3
0.08
0.14
0.08
put the bowl in the drawer #4
0.07
0.07
0.20
reach bottom drawer #5
0.01
0.13
-0.01
close the battom drawer #6
0.16
0.33
0.10
close the top drawer #7
0.07
0.10
0.18
put the bowl on the cabinet #8
0.08
0.08
0.16
1
2
3
4
6
7
8
reach the drawer
0.03
-0.02
0.00
open the drawer
0.28
0.08
0.13
grasp the bowl
0.08
0.14
0.08
put the the drawer
0.07
0.07
0.20
close the drawer #5
0.13
0.05
0.21
put bowl on the cabinet #6
0.08
0.08
0.16
reach the drawer #1
0.03
-0.01 -0.03
pull / push the drawer #2
0.18
0.06
0.05
grasp the bowl
0.06
0.14
0.08
transition the bowl #4
0.05
0.08
0.11
Figure Average cosine similarity between pairs of sub-goal categories (defined by human semantics) computed using manipulation concept latents learned by our method (Sec.3). In each heatmap, the value at the i-th row and j-th column represents the average cosine similarity between latent vectors from the i-th and j-th categories. Three levels of labeling are provided across the heatmaps; please refer Sec. C.2 for details.
30

---

0:23
0.31
0.02
0.19
0.14
0.14
0:31
0.51
0.04
0:14
0.21
0.20
0.02
0.04
0.07
0.14
0.20
0.26
100
T-ebil
T-ebil 2-rigtt
2 badkward
5 S-lowarrd
B-iet
0.16
-0.10
3
-0.17
0.40
0.52
-0.10
0.72
-100
2
3
-30 1002
30 100
2:00s0
(a) Average cosine similarity between pairs (b) I-SNE Clustering of Manipulation Concept Latents correp- of movement categories (defined by hu- sonding to tasks. We perform t-SNE clustering the manipula- man semantics) computed using manipula- tion concepts cach time step. These concepts are generated by tion concept latents learned by our method our method (Sec. 3). Each sample is colored according to its task, (Sec.3).
representing one of 90 possible tasks as indicated by the colorbar.
Figure 10
Ours
Next
InfoCon
XSkill
TMAE
logl N(Cluster)
CLIP
DINOV2
4-
2
200
400
600
800
1000
Eps 1000.0
Figure 11: DBSCAN Clustering Analysis of Manipulation Concept Latents' Diversity and Discrimination. Clustering is performed manipulation concept latents generated by our method and the baseline methods described in Manipulation Concept Discovery Baselines (Sec. 4.1), across 90 tasks from the LIBERO-90 -90 dataset. The figure shows the (log) number of clusters obtained using DBSCAN for clustering density E [0, 1], with no points classified as noise.
31

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract gives a summary of our contribution on self-supervised learning of manipulation concepts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
32

---

Justification: We discuss limitations including improvements to hierarchy derivation, further work on scaling up, and modality balance.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We mainly make use of established theoretical frameworks (such as mutual information) for clarification and modeling of our method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details in the appendix and supplementary materials.
Guidelines:
33

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will open source all code and newly-created datasets upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
www.ammmissiioooiiiiiooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / ceaaps..c/uublic/uuii httppppuiiiggidssss/pppppeeeese/ /CodeSubmiss is asssonoooicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
34

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting//etails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are specified in the Experiments section and in the appendix and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: For the policy success rates we currently include standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to the details provided in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
35

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ //eurrr////riss././ cc cctpublic//eeeeeeccceccceceeeeee heroolic/Ethiccuuieeeii cechiihsscielllicc ines?
Answer: [Yes]
Justification: Current experiments and topics do not conflict with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Currently, the experiments are carried out in simulations and on robots in laboratories.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Currently, we have not encountered any safeguard issues.
Guidelines:
36

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have checked the sources we used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo//aaaaeoe com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We will release all new assets created (code/mmodels/dattsees)) upon accep-
tance.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
37

---

Justification:
We currently do not have crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We currently do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA].
Justification:
The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: htt:://eeni...eeeee cc/ cc//enffereencce/22255 for what should or should not be described.
38


Reviews:
[{"review_id": "OQitd9q3QY", "reviewer": "Reviewer_6hJG", "summary": "This paper introduces HiMaCon, a self-supervised framework designed to learn hierarchical concepts for robotic manipulation from unlabeled multi-modal data. The authors propose that learning abstract, hierarchically structured concepts can bridge the gap between low-level actions and high-level goals, enhancing policy robustness. The HiMaCon framework achieves this through two complementary mechanisms: \n\n(1) a cross-modal correlation network that learns invariant relationships between different sensory inputs (e.g., vision and proprioception), \n\n(2) and a multi-horizon predictor that organizes these concepts into a temporal hierarchy of short-term and long-term sub-goals. \n\nThe paper demonstrates through extensive experiments in both simulation and the real world that policies enhanced with these learned concepts significantly outperform standard imitation learning approaches, especially in challenging generalization tasks. The analysis also reveals that the discovered concepts are semantically meaningful and align with human-interpretable manipulation primitives, despite the absence of explicit labels during training.", "questions": "1. The sub-process derivation via the coherence threshold $\\varepsilon$ is a cornerstone of your method. Could you provide more intuition on how the uniform sampling of $\\varepsilon \\sim \\mathcal{U}(0,1)$ during training encourages the emergence of a meaningful and stable hierarchy? Did you experiment with alternative sampling strategies (e.g., discrete levels, a biased distribution) or analyze the sensitivity of the learned representations to this choice?\n2. In your real-world experiments, you hypothesize that concept enhancement provides a \"relational focus\" that allows the policy to ignore superficial features. This is a powerful claim. Can you elaborate on the mechanism by which the cross-modal correlation objective (Eq. 3) specifically encourages the model to learn abstract relations (like \"object inside container\") over concrete visual features (like object color or shape)? Is it because relational properties have more consistent correlations across modalities than surface features do?\n3. For real robot experiments, you only report success rates. Could you provide a more detailed analysis of failure modes? How does the concept-enhanced policy's behavior differ from the baseline in failed attempts?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "=== Strengths ===\n\n1. The combination of cross-modal correlation learning and multi-horizon sub-goal organization is interesting. The mask-and-predict strategy for capturing cross-modal patterns and the coherence-based clustering for temporal hierarchy are well-motivated.\n2. The paper provides extensive experiments across multiple settings (LIBERO-90, LIBERO-LONG, LIBERO-GOAL) and includes both simulation and real-world validation. The proposed method consistently outperforms baselines across different policy architectures (ACT and Diffusion Policy). The performance gains are particularly notable in transfer and generalization scenarios.\n3. The semantic alignment study showing that learned concepts correspond to human-interpretable sub-goals is valuable. The visualization of multi-horizon predictions provides good insights into what the model learns.\n\n=== Weaknesses ===\n\n1. The concept discovery phase involves training multiple transformer-based networks, which appears to be computationally intensive.\n2. The derivation of multi-horizon sub-processes relies on a coherence threshold $\\varepsilon$ applied to the cosine distance between latent concepts (Eq. 4). While sampling $\\varepsilon$ uniformly during training is a reasonable strategy, the underlying mechanism feels somewhat heuristic. The paper could be strengthened by discussing the sensitivity to this mechanism or exploring alternative, more structured approaches to discovering the hierarchy.", "was_revised": true, "final_justification": "The authors addressed my concerns well in the rebuttal period with more supportive experiments. I will keep my original rating and tend to accept it.", "rebuttal": "Thank you for your thoughtful and comprehensive review. We appreciate your recognition of the key technical contributions, particularly the novelty of combining cross-modal correlation learning with multi-horizon sub-goal organization. Your acknowledgment of our extensive experimental validation across multiple benchmarks and policy architectures, as well as the semantic interpretability analysis, is very encouraging. We're glad that you found the mask-and-predict strategy and coherence-based clustering well-motivated, and that you recognized the value of our semantic alignment study in demonstrating that HiMaCon learns meaningful manipulation primitives. Below we provide our responses to the weaknesses and questions you have raised.\n\n***\n\n# Re W1: Computation\n\nWe acknowledge the reviewer's concern about computational requirements and appreciate the opportunity to clarify this aspect of our method.\n\n**Reasonable Computational Requirements:** Our experiments demonstrate that the complete manipulation concept discovery pipeline operates under accessible computational constraints. As detailed in Sec.A.1, the concept discovery training can be executed on widely available hardware such as a single RTX 3090/4090 GPU (24GB memory), completing within 1.5 days. The architecture is deliberately modest, consisting of a 12-layer concept encoder $\\mathcal{E}$ and two 4-layer networks for cross-modal correlation $\\mathcal{C}$ (Eq.3) and multi-horizon prediction $\\mathcal{F}$ (Eq.7), totaling approximately 48M parameters across all three networks with 256 hidden dimensions.\n\n**Strong Return on Investment:** This represents a one-time computational investment. For example, once trained on LIBERO-90, the learned concepts transfer directly to LIBERO-LONG and LIBERO-GOAL without retraining (Sec.4.2), while consistently delivering performance improvements across benchmarks (Tab.1).\n\n**Future Optimization:** We acknowledge the importance of computational efficiency for larger-scale deployment. We are actively exploring strategies including foundation model initialization with efficient fine-tuning techniques like LoRA, and alternating training strategies between cross-modal correlation learning (Eq.3) and multi-horizon sub-goal learning (Eq.7) to reduce peak memory requirements.\n\n# Re W2 and Q1: Sampling Strategies\n\nWe thank the reviewer for their insightful questions about our hierarchical derivation mechanism and the sampling strategy of $\\epsilon$ during training.\n\n**Uniform Sampling for Hierarchical Representation Learning**\n\nOur approach implements a form of multi-scale temporal abstraction learning. Different $\\epsilon$ values correspond to different levels of temporal compression: low $\\epsilon$ values identify fine-grained manipulation primitives, while high $\\epsilon$ values discover coarse-grained subprocesses. By considering all possible $\\epsilon$ across this spectrum, we ensure that learned concepts can encode subgoals that are maximally informative about future state transitions across all temporal scales (Eq.5).\n\nThe uniform sampling of $\\epsilon\\sim\\mathcal{U}(0,1)$ is motivated by computational considerations. While considering all possible $\\epsilon$ values during each training iteration would be ideal, this approach is computationally prohibitive, leading us to employ uniform sampling as a practical approximation.\n\n**Why Uniform Distribution Rather Than Alternatives**\n\nSince different manipulation tasks naturally exhibit varying optimal temporal structures, uniform sampling provides equal representational capacity across all granularities. This prevents the model from biasing toward any particular temporal scale, which could lead to suboptimal hierarchy derivation for certain tasks.\n\n**Experimental Validation**\n\nWe conducted experiments on LIBERO-90 to validate our design choice, testing multiple alternative strategies:\n\n| Sampling Strategy | ACT | DP | Description |\n|-|-|-|-|\n| **Uniform (Ours)** | **74.8±0.8** | **89.6±0.6** | $\\epsilon\\sim\\mathcal{U}(0,1)$ |\n| Discrete | 67.6±0.5 | 81.1±0.8 | $\\epsilon\\sim${$0.1, 0.2, ..., 1.0$} |\n| Biased | 65.6±0.7 | 78.7±0.4 | $\\epsilon\\sim\\mathcal{U}(\\frac{1}{3},\\frac{2}{3})$ |\n\nWe will provide additional studies, experiments, and analysis of different selection strategies in the revised version, as we acknowledge that the sub-process derivation strategy is essential for the quality of learned concepts.\n\n# Re Q2: Mechanism for Relational Learning\n\nThank you for this insightful question about the mechanism underlying our cross-modal correlation objective. We explain how Eq.3 encourages learning abstract relations over superficial features through a principled information-theoretic argument.\n\n**The Fundamental Challenge: What Information Enables Cross-Modal Prediction?**\n\nConsider a robot placing an object in a container, observed through both a gripper-mounted camera (top view) and a third-person camera (front view). Our cross-modal correlation objective requires the model to predict one view from the other, conditioned on the manipulation concept latent. This creates a fundamental question: what type of information must the concept latent encode to successfully bridge between these different visual perspectives?\n\n**Why Surface Features Cannot Solve Cross-Modal Prediction**\n\nThe answer emerges from analyzing what information is already available versus what is missing. Both camera views already contain comprehensive appearance information—colors, textures, and object shapes are visible in both modalities. Therefore, if the concept latent redundantly encodes this already-available appearance information, it provides no additional predictive power for cross-view generation. Surface features like \"object A is red\" offer minimal benefit when predicting the front camera view from the top camera view, since color is already observable in the source view.\n\n**Why Relational Properties Are Essential**\n\nIn contrast, relational properties like spatial arrangements and containment relationships (\"object A is inside container B\") provide the geometric understanding necessary to successfully map between different visual perspectives. These abstract relationships remain consistent across viewpoints but require sophisticated understanding to bridge the visual gap between perspectives. From an information-theoretic standpoint, relational properties exhibit significantly higher mutual information across modalities than surface features, making them the viable solution for cross-modal prediction.\n\n**The Representational Bottleneck Effect**\n\nThis analysis reveals the core mechanism: our objective creates a \"representational bottleneck\" that acts as a natural filter. Since surface features cannot solve the cross-modal prediction task, the learning process is forced to discover and encode view-invariant relational structure. The prediction objective thus transforms the inherent cross-modal consistency of relational properties into learned representational structure that prioritizes relational understanding over superficial appearance.\n\n# Re Q3: Failure Modes\n\nThank you for this important question. We provide a failure mode analysis based on examination of failed attempts across our real robot experiments, revealing how manipulation concepts change robot behavior patterns.\n\n- **Subtask Awareness and Persistence:** Baseline failures frequently exhibit \"phantom grasping\"—proceeding to move toward containers without successfully grasping objects, indicating poor subtask completion awareness. In contrast, concept-enhanced failures demonstrate persistent re-grasping attempts when initial grasp attempts fail (1-4 times), maintaining task coherence by recognizing incomplete subgoals. While these attempts ultimately fail due to time constraints or object displacement, the policies show systematic task progression awareness rather than premature abandonment.\n\n- **Goal-Directed vs. Wandering Behavior:** In the **Barriers** condition, failure patterns diverge markedly. Baseline failures exhibit \"wandering\" behavior after grasping—moving the gripper aimlessly around containers without structured placement attempts, suggesting complete loss of task direction. Concept-enhanced failures maintain goal-directed behavior, successfully navigating barriers but failing due to physical constraints such as container tipping from elbow contact. Crucially, these policies continue executing placement subgoals rather than abandoning task progression.\n\n**Implications**: These patterns demonstrate that concept prediction training (Eq.9) provides policies with enhanced understanding of task progress and subgoal states. Concept enhancement transforms failure modes from random task abandonment to systematic, goal-directed behavior that maintains task coherence even when ultimately unsuccessful.\n\nWe will include these detailed behavioral comparisons in the revised version of the paper.\n\n***\n\nWe thank the reviewer again for these thoughtful questions and hope our clarifications reinforce the value and clarity of our contributions. Please feel free to let us know if you have more suggestions or questions."}, {"review_id": "2RvXOzQFx8", "reviewer": "Reviewer_WGug", "summary": "This paper proposes a self-supervised pipeline to discover 'concepts' from a dataset of demonstrations by cross-modal alignment and temporal clustering, and use these concepts to help imitation learning by adding an auxiliary goal of concept prediction.", "questions": "* Please refer to weaknesses.\n* The reviewer wonders how robust the pipeline is against visual disturbances, like background, lighting conditions, etc, and cross-embodiment generalization. The dataset used for concept discovery is quite similar to the test setting (all from Libero). Ideally, I think self-supervised training with any robotic dataset (even human manipulation videos from the internet) would help. Can the authors provide some intuition and, what would be even better, some preliminary empirical results?\n\nIf the authors can address the questions above, the reviewer will raise the score (possibly by a large margin).", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The idea of cross-modal alignment is natural and well motivated, and the experiment results are promising.\n\nWeaknesses:\n* There are some related works missing. For example, [1] has a very similar problem setting and method design; [2, 3] also learn concepts (object contact and geometrical relationships) from demonstrations. The author should consider comparing with [1] and adding all of the above to the literature review. \n* In  Section 3.3, the authors say they are learning 'sub-goals' and claim that this is helpful for later manipulation learning. However, it is not clear to me whether this help only comes from planning in the latent space, or planning in **the very** latent space which incorporates cross-modal alignment. The author should consider a finetuned version of DINOv2 or CLIP with their original loss function plus the multi-horizon sub-goal loss function proposed in Eq. 7. I believe this ablation will help readers understand the crucial role of the method proposed in Section 3.2.\n\n[1] Zhou, Pei, et al. \"AutoCGP: Closed-Loop Concept-Guided Policies from Unlabeled Demonstrations.\" The Thirteenth International Conference on Learning Representations.  \n[2] Mao, Jiayuan, et al. \"Learning reusable manipulation strategies.\" Conference on Robot Learning. PMLR, 2023.  \n[3] Liu, Yuyao, et al. \"One-Shot Manipulation Strategy Learning by Making Contact Analogies.\" ICRA, 2025.", "was_revised": true, "final_justification": "The authors addressed all my concerns during the rebuttal period: (1) some closely related works and baselines were missing; (2) an ablation to support their claim; (3) generalization and robustness of their method. Therefore, given the novelty and technical solidness of this work, I have raised my score from 3 to 5.", "rebuttal": "Thank you for the feedback and for recognizing the key contributions of our work. We appreciate that you found the cross-modal alignment approach natural and well-motivated, and we are encouraged by your assessment of our experimental results as promising. Your summary accurately captures our core methodology of discovering manipulation concepts through cross-modal alignment and temporal clustering, followed by their integration into imitation learning through auxiliary concept prediction. We are grateful for your constructive feedback on our work. Below we provide our responses to the weaknesses and questions you have raised.\n\n***\n\n# Re W1: Related Works\nWe sincerely thank the reviewer for highlighting these important related works. We agree these references significantly enrich the context of our research and will explicitly discuss and compare them in the revised version to clearly position our contributions.\n\n## Regarding AutoCGP [1]:\n\nWhile AutoCGP shares our goal of learning manipulation concepts from demonstrations, our approach differs fundamentally in three key aspects:\n\n**Multi-modal Concept Discovery:** AutoCGP primarily relies on proprioceptive states for concept extraction, whereas our method leverages comprehensive multi-modal observations (vision + proprioception). Our ablation study (Tab.2) demonstrates that incorporating multiple modalities consistently improves performance.\n\n**Hierarchical Temporal Structure:** Our coherence-based clustering approach (Eq.4) enables simultaneous discovery of concepts at multiple temporal scales within a unified framework. This allows policies to reason about both immediate actions and long-term goals simultaneously, whereas AutoCGP focuses on a single level of temporal abstraction through trajectory segmentation.\n\n**Policy Integration Strategy:** We integrate concepts through regularization applied to hidden representations (Eq.9), maintaining compatibility with diverse architectures while introducing only minimal additional model complexity. In contrast, AutoCGP explicitly conditions policy heads on predicted concepts, requiring an additional concept generation model that substantially increases both model capacity and computational requirements.\n\n**Experimental Validation:** To substantiate these technical differences, we conducted comparative experiments on LIBERO-90 using three configurations:\n\n- **AutoCGP (original):** Uses AutoCGP's proprioceptive-based concept discovery method and integration strategy.\n- **AutoCGP (HiMaCon):** Our multi-modal concept discovery method integrated using AutoCGP's concept conditioning strategy.\n- **Ours (DP):** Our complete framework using Diffusion Policy as the base policy architecture. We compare with Diffusion Policy (DP) since AutoCGP employs DP as its policy head.\n\n| Method | AutoCGP (original) | AutoCGP (HiMaCon) | Ours (DP) |\n|:-:|:-:|:-:|:-:|\n| Success Rate (%) | 81.9±0.3 | 83.5±0.8 | **89.6±0.6** |\n\nThe improvement when using our concepts with AutoCGP's integration method demonstrates the quality of our manipulation concepts, while our full method's superior performance validates both our concept discovery and integration strategies.\n\n## Regarding \\[2, 3]:\nWe appreciate the reviewer highlighting these valuable methods, which indeed address critical aspects of manipulation concept learning—particularly object contact behaviors and geometric relationships. Unlike these approaches, which typically require explicit labeling or prior knowledge about contact states, our approach achieves fully self-supervised discovery of hierarchical manipulation concepts directly from unlabeled multi-modal data, without any semantic annotations. Nonetheless, the demonstrated capabilities in \\[2] and \\[3] regarding few-shot learning scenarios inspire promising directions for future exploration. Extending our method to efficiently handle few-shot learning contexts represents an exciting avenue we intend to pursue in subsequent work.\n\nIn summary, we will expand the \"Related Work\" section of our manuscript to explicitly include detailed discussions and comparative analyses with these important related works \\[1, 2, 3]. This addition will clearly articulate how our methodology advances beyond existing approaches in terms of novelty, generality, and empirical performance, further enhancing the clarity and comprehensiveness of our contribution.\n\n# Re W2: Ablation to Isolate Contributions\nWe sincerely thank the reviewer for this insightful question that helps clarify the individual contributions of our framework's components. Following the reviewer's suggestion, we conducted additional experiments to isolate the contribution of our cross-modal correlation learning (Sec.3.2). We fine-tuned CLIP on LIBERO-90 data using the original text-image contrastive loss combined with our multi-horizon sub-goal loss (Eq.7), while excluding our cross-modal correlation methodology from Sec.3.2 (Eq.3). We also tested additional ablations to better understand component interactions:\n\n| Method | Cross-Modal (Eq.3) | Multi-Horizon (Eq.7) | ACT | DP |\n|:-:|:-:|:-:|:-:|:-:|\n| CLIP + Eq.7 (reviewer suggestion) | X | O | 66.4±0.5 | 81.6±0.6 |\n| Cross-modal only | O | X | 69.1±0.6 | 82.8±1.0 |\n| Multi-horizon only | X | O | 71.6±0.4 | 80.5±0.5 |\n| Ours (Full method) | O | O | **74.8±0.8** | **89.6±0.6** |\n\nThese results reveal several key insights:\n\n- **Cross-modal correlation learning provides substantial benefits**: The improvements over CLIP+Eq.7 and the multi-horizon-only ablation demonstrate that our mask-and-predict strategy for capturing cross-modal patterns (Sec.3.2) contributes meaningfully beyond hierarchical temporal learning alone. The planning will benefit from a latent space which incorporates cross-modal alignment.\n- **Both components are necessary and complementary**: Neither cross-modal learning alone nor multi-horizon prediction alone achieves full performance. This indicates that robust cross-modal correlations and multi-horizon subgoals have complementary effects, with their combination yielding superior performance compared to either component in isolation.\n\nWe thank the reviewer for this valuable suggestion and will include these comprehensive ablations in our revision.\n\n# Re Q1: About Weaknesses\nPlease refer to **Re W1: Related Works** and **Re W2: Ablation to Isolate Contributions**.\n\n# Re Q2: Robustness and Cross-Embodiment Generalization\nWe thank the reviewer for this insightful question about robustness and cross-domain generalization. We conducted additional experiments to directly address these concerns.\n\n## Visual Robustness Analysis\n\nWe evaluated robustness to visual disturbances using our Real-world Generalization protocol (Sec.4.4) with two additional test conditions: (1) **Background variation**: placing distracting objects and patterns behind the manipulation workspace, and (2) **Lighting variation**: testing under different illumination conditions (bright, dim, and flash lighting). Results demonstrate that policies enhanced with our manipulation concepts show significantly improved robustness:\n\n|   | Background | Lighting |\n|:-:|:-:|:-:|\n| w/o concept | 33.3% | 53.3% |\n| w/ concept | 53.3% | 66.7% |\n\n## Cross-Embodiment Concept Transfer\n\nWe tested whether concepts learned from human demonstrations can enhance robotic policies. We collected human manipulation videos of the cleaning-cup task using a camera. We trained our concept discovery pipeline on these human videos and applied the learned concepts to enhance robotic policies:\n\n|| Place | Color | Obj. | Obst. | Barr. | Multi | Background | Lighting |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| w/o concept | 53.3% | 46.7% | 40.0% | 20.0% | 0.0% | 0.0% | 33.3% | 53.3% |\n| w/ concept (human) | 66.7% | 60.0% | 46.7% | 33.3% | 13.3% | 13.3% | 40.0% | 60.0% |\n\nThe consistent improvements demonstrate that our self-supervised framework captures manipulation patterns that transcend embodiment differences. This supports the reviewer's intuition about training with diverse data sources.\n\nWe will include these results and video demonstrations in our revision.\n\n***\n\nWe deeply appreciate the reviewer's constructive comments and hope our responses provide clarity on the points raised. Please feel free to share any further concerns or suggestions you may have.\n\n[1] Zhou, Pei, et al. \"AutoCGP: Closed-Loop Concept-Guided Policies from Unlabeled Demonstrations.\" The Thirteenth International Conference on Learning Representations.\n\n[2] Mao, Jiayuan, et al. \"Learning reusable manipulation strategies.\" Conference on Robot Learning. PMLR, 2023.\n\n[3] Liu, Yuyao, et al. \"One-Shot Manipulation Strategy Learning by Making Contact Analogies.\" ICRA, 2025."}, {"review_id": "FUGeymAplA", "reviewer": "Reviewer_eio8", "summary": "This paper proposes a self-supervised framework to learn the hierarchical manipulation concepts. It operates through two mechanisms: 1) Cross-modal correlation learning to capture the invariant pattern across different sensory modalities, and 2) Multi-horizon sub-goal organization structures concepts hierarchically across temporal scales. The manipulation concepts can help imitation learning policy as an additional prediction goal. Experiments on simulation and real environment show the performance gain of the proposed method compared to multiple baselines.", "questions": "Please consider replying to the Weakness.", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The proposed algorithm is well-motivated. It aligns with human intuition and intuitively can help to improve the generalization ability of manipulation policies. The method is also well designed. The two mechanisms have their own clear goals separately, which are also complementary.\n\n2. The authors conduct extensive experiments in both simulation and real world to evaluate the proposed algorithm. The results show notable performance gain. Especially, I like the detailed analysis in Sec. 4.3 and 4.4 beyond pure quantitative numbers. These analysis greatly justify the algorithm design and help readers to understand the roles of each module.\n\n3. The authors provide many details including pseudo code for their algorithm implementation, which would benefit the potential reproduction. The visualizations and figures are also clear.\n\n\nWeaknesses:\n\n1. I think this idea of manipulation concepts shares great similarity with skill learning since they both extract some high-level and generalizable information. There are already many previous works for skill learning like [r1]. I believe it would be helpful to discuss with these related works.\n\n2. It is strange for me that the pretrained encoder is not re-used for the policy. Instead, the authors consider the manipulation concepts as extra regularization prediction goal. I hope the authors can give more explanation and comparison regarding this.\n\n3. For cross-modal correlation, it is more intuitive to directly apply corss-modal contrastive learning like CLIP rather than this masked prediction to achieve the goal. Maybe the authors can give some disucssion about it.\n\n\n\n[r1]: SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution", "was_revised": true, "final_justification": "The authors solve my concerns in the rebuttal period.  Overall, I think this paper is well supported by extensive experiments and has great reproducibility, so I will keep my original rating and tend to accept it.", "rebuttal": "Thank you for the thoughtful and comprehensive review. We greatly appreciate your recognition of our work's motivation and the intuitive alignment with human manipulation concepts. Your acknowledgment that our two-mechanism approach (cross-modal correlation learning and multi-horizon sub-goal organization) is well-designed with clear, complementary goals is particularly encouraging, as this dual structure represents a core contribution of our framework. We are also grateful for your positive feedback on our extensive experimental validation across both simulation and real-world environments, and especially your appreciation of the detailed analyses in Sec.4.3 and Sec.4.4. Finally, thank you for noting our efforts to ensure reproducibility through detailed implementation descriptions and pseudocode. We address your concerns and questions below:\n\n***\n\n# Re W1: Related Work of Skill Learning\nThank you for identifying this important connection to the skill learning literature. We acknowledge that manipulation concepts and skill learning both aim to extract generalizable, high-level information from demonstrations, and we will expand our related work section to provide a more comprehensive discussion of skill learning approaches, including SkillDiffuser [r1] and other relevant works [r2, r3].\n\nHere we outline several distinctions between our approach and skill learning methods:\n\n**Conceptual Level**\n\nWhile skill learning focuses on discovering **executable action primitives**, our manipulation concepts capture more abstract patterns that inform policy learning:\n\n- **Skills** are executable action sequences that can be directly invoked (e.g., \"grasp object\", \"move to location\").\n- **Manipulation Concepts** capture manipulation-relevant patterns across multiple abstraction levels—from fine-grained motor actions to high-level subgoals that may combine multiple skills to achieve completion.\n\n**Technical Approach**\n\n- **Representation Space**: Unlike SkillDiffuser [r1] which uses discrete skill representations that may limit discoverable patterns, our continuous concept latents (lines 116-118) provide flexible representational capacity without finite codebook constraints.\n- **Multi-Modal Integration**: Our framework explicitly models cross-modal correlations through mask-and-predict strategies (Eq.3), enabling robust generalization across sensory variations—a focus less emphasized in typical skill learning approaches.\n- **Integration Mechanism**: Rather than requiring explicit skill selection as in [r1] or specialized hierarchical planning frameworks, our concepts seamlessly integrate with existing policy architectures through joint prediction (Eq.9), simply adding a concept prediction head ($\\pi_z$) for regularization.\n\n\n# Re W2: Concept Encoder Usage\nThank you for this insightful question about our architectural design choices. We tested both approaches during development and can provide empirical evidence supporting our design decision.\n\n## Empirical Comparison\n\nWe compared two conditioning strategies on LIBERO-90 tasks:\n- **Direct Conditioning**: Using the pretrained concept encoder $\\mathcal{E}$ (Eq.1) to extract concepts from observations, then feeding these as additional input features to the policy\n- **Joint Prediction (Ours)**: Training the policy to predict future manipulation concepts alongside actions through joint optimization (Eq.9)\n\n| Method | Direct Conditioning | Joint Prediction (Ours) |\n|:-:|:-:|:-:|\n| ACT | 71.1±0.4 | 74.8±0.8 |\n| DP | 79.3±0.9 | 89.6±0.6 |\n\nOur joint prediction approach consistently outperforms direct conditioning.\n\n## Key Insight: Temporal Alignment\n\nWe believe the performance difference stems from a fundamental temporal alignment issue:\n\n- **Direct conditioning**: Uses manipulation concepts extracted from current/past observations as additional input features for predicting future actions, which creates a temporal misalignment\n- **Joint prediction**: Trains the policy to predict future manipulation concepts alongside future actions, creating temporal coherence between concept understanding and action planning\n\n**Concrete Example**: In a cup placement task, after grasping an object, the current manipulation concepts encode \"grasping dynamics.\" However, for planning the next placement action, the policy needs to understand \"placement dynamics.\" Our joint approach learns to predict placement-related concepts that directly inform upcoming actions, while direct conditioning provides historical grasping information that may be less relevant for placement planning.\n\nNevertheless, we acknowledge that direct conditioning on current/past manipulation concepts could prove valuable for tasks requiring explicit long-term memory [r4], and we plan to explore hybrid approaches in future work.\n\n# Re W3: Masked Prediction instead of Contrastive Learning\n\nThank you for this question about our choice of masked prediction over contrastive learning approaches like CLIP. We carefully considered both options and selected masked prediction primarily to avoid the inherent challenges and potential biases associated with negative sampling strategies in contrastive learning.\n\nSpecifically, manipulation involves recurring conceptual patterns (approach, contact, grasp, transport, release) that can appear across different demonstrations or multiple times within the same demonstration. This creates significant challenges for principled negative sampling without manual annotation or external supervision.\n\nIn contrast, our masked prediction approach elegantly sidesteps these challenges by providing self-supervised learning without requiring explicit similarity judgments. The model learns cross-modal correlations by reconstructing masked modalities using available context, naturally capturing the relationships essential for manipulation understanding. This approach eliminates the risk of conflicting supervision signals that could arise from improperly chosen negative samples. Therefore, we selected the mask-and-predict framework.\n\n***\nOnce again, we thank the reviewer for the detailed examination and hope our responses strengthen both the clarity and impact of our contributions. We invite any additional comments or questions you might have.\n\n[r1] Liang, Z., Mu, Y., Ma, H., Tomizuka, M., Ding, M., & Luo, P. (2024). Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16467-16476).\n\n[r2] Rho, S., Smith, L., Li, T., Levine, S., Peng, X. B., & Ha, S. Language Guided Skill Discovery. In The Thirteenth International Conference on Learning Representations.\n\n[r3] Chen, L., Bahl, S., & Pathak, D. (2023, December). Playfusion: Skill acquisition via diffusion from language-annotated play. In Conference on Robot Learning (pp. 2012-2029). PMLR.\n\n[r4] Fang, H., Grotz, M., Pumacay, W., Wang, Y. R., Fox, D., Krishna, R., & Duan, J. (2025). Sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation. arXiv preprint arXiv:2501.18564."}, {"review_id": "AmKZvvAKbz", "reviewer": "Reviewer_7UNt", "summary": "The paper proposes HiMaCon, a self-supervised framework for learning hierarchical manipulation concepts from unlabeled multi-modal robotic data. The method discovers latent representations that encapsulate both cross-modal correlations (e.g., between vision and proprioception) and multi-horizon temporal structure (short-term and long-term subgoals). These representations are then used to augment robotic manipulation policies, learnt from demonstrations using either a transformer based CVAE or a diffusion policy. Empirical evaluations on the LIBERO benchmark and real-world robot deployments demonstrate the method’s efficacy.", "questions": "- Why the choice of the spherical distance, as opposed to others, e.g. cosine similarity? Any reasons or intuition for this?\n- If I read equation 4 / algorithm 1 correct, the same sub-process is defined as any pair-wise distance between to elements of the sub-process should be below the threshold? Why not just enforce a distance between the start and (current) end in Algorithm 1 while loop? Is that not sufficient?\n- The z is used as an additional prediction signal for the policy to guide the learning. While this is clever and makes it easy to \"add-on\" to existing methods, it does mean that the policy network has to \"re-learn\" the structure captured by the Concept Encoder to correctly predict it. Did you try training a policy \"head\" on top of the Concept encoder directly, or an intermediate representation thereof?\n- In the future work and appendix you refer to and compare to VLA models, and it seems that you come close to their performance. You mention potentially combining it with VLA models. Do you think the improved performance of a VLA is orthogonal to what HiMaCon provides to the ACT/DP models, i.e. that HiMaCon will offer additional improvements on a VLA, or would a VLA rather capture similar manipulation concepts in their intermediate layers and hence perform similar already? I'd love to hear the author's view on this.", "limitations": "in appendix due to space constraints, but yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "*Strengths*\n- The method is novel and well-motivated from physiological evidence. Whereas I have seen skill discovery using mutual information between (discrete) latent and observations before (e.g. https://arxiv.org/abs/2211.13350), exploiting cross-modal correlations is nice and elegant. \n- I especially like the fact that it can be used as an \"add-on\" for existing methods, as evidenced by combinging it with both ACT and DP.\n- Strong benchmark results on LIBERO and in addition nice real-world experiments and interpretability results.\n\n*Weaknesses*\n- The approach needs access to aligned modality (i.e. RGB and proprioceptive) as well as action information of demonstrations to train on, which might be costly to obtain for different use cases.", "was_revised": true, "final_justification": "Solid paper, and interesting points brought forward by the authors during the discussion, which can only improve the work.", "rebuttal": "Thank you for the thoughtful review. We appreciate your recognition of HiMaCon's novelty, particularly your observation that exploiting cross-modal correlations is \"nice and elegant.\" We're pleased you found value in our approach's compatibility as an \"add-on\" for existing architectures like ACT and Diffusion Policy, which was a key design for practical applicability. Your appreciation of our comprehensive evaluation—including LIBERO benchmark results, real-world experiments, and interpretability analyses—is encouraging. We address your specific concerns below.\n\n***\n\n# Re W1: Multi-Modal Data Collection Cost\n\nWe acknowledge this is a valid practical consideration. However, we would like to highlight several points that address this concern:\n\n**Comparable Requirements to Standard Approaches**: Multi-modal data collection (vision + proprioception + actions) is standard in modern robotic platforms (e.g., ALOHA [1], UR, Franka) and datasets [2, 3]. Unlike methods requiring post-collection annotation (e.g., DecisionNCE baselines in Tab.1 require manual language labels), our framework is self-supervised, eliminating ongoing labeling costs.\n\n**Effectiveness with Reduced Sensor Requirements**: Our ablation studies (Tab.2) and no concept baseline (Plain in Tab.1) demonstrate that while additional modalities yield better performance, concepts learned from even single modalities provide policy improvements.\n\n**Validation with Cost-Effective Data Collection**: To further address cost concerns, we conducted an additional experiment on policies enhanced by concepts derived from human manipulation demonstrations collected via simple video recording for the real-world cleaning-cup task (Sec.4.4). This more accessible approach still enhances policy performance across scenarios:\n\n| | Place | Color | Obj. | Obst. | Barr. | Multi |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| w/o concept | 53.3% | 46.7% | 40.0% | 20.0% | 0.0% | 0.0% |\n| w/ concept (human) | 66.7% | 60.0% | 46.7% | 33.3% | 13.3% | 13.3% |\n\nThese results demonstrate practical deployment potential under cost limitations while maintaining performance improvements.\n\n# Re Q1: Spherical Distance\n\nThank you for this important question about our choice of spherical distance.\n\n**Empirical Validation**: We conducted ablation experiments comparing spherical distance with cosine similarity (cosine distance $\\frac{1-\\cos\\theta}{2}$) as the coherence metric in Eq.4. The results demonstrate clear performance advantages for spherical distance:\n\n| Method | Cosine Similarity | Spherical Distance (Ours) |\n|:-:|:-:|:-:|\n| DP | 82.0±0.4 | 89.6±0.6 |\n| ACT | 67.8±0.5 | 74.8±0.8 |\n\n**Justification**: This empirical advantage aligns with the theoretical considerations that motivated our choice. The arccos transformation in spherical distance provides non-linear scaling that emphasizes differences between vectors with high cosine similarity, amplifying subtle distinctions in concepts that cosine similarity might miss.\n\n# Re Q2: Sub-process Segmentation Criterion\n\nThank you for this insightful algorithmic question about our sub-process derivation strategy.\n\n## Empirical Validation\nWe tested the proposed start-to-end strategy on LIBERO-90 and found consistent performance degradation compared to our pairwise approach:\n\n| Method | Our Approach | Start-to-End Strategy |\n| :-: | :-: | :-: |\n| ACT | 74.8±0.8 | 68.4±0.8 |\n| DP  | 89.6±0.6 | 79.8±0.5 |\n\n## Core Issue: Asymmetric Coherence\nThe start-to-end approach enforces a \"star pattern\" where all concepts must be similar to the initial concept, but concepts later in the sequence may be quite different from each other. This can inappropriately group semantically distinct manipulation phases that happen to share similarity with an initial state.\n\n**Example**: Consider this manipulation sequence:\n```\n[reach forward] → [reach upward] → [grasp cup] → [reach downward] → [place plate]\n```\n\nIf the coherence threshold allows \"reach forward\" to be similar to both \"reach upward\" and \"reach downward\" (due to shared movement characteristics), but \"reach upward\" and \"reach downward\" are dissimilar to each other, then:\n\n- **Start-to-end method**: Groups the entire sequence into one sub-process because each step is similar enough to the initial \"reach forward\"\n- **Our pairwise method**: Splits into `[reach forward → reach upward → grasp cup]` and `[reach downward → place plate]` because it detects the dissimilarity between upward and downward reaching\n\n**Why This Matters**: This asymmetric grouping can merge distinct manipulation sub-goals (e.g., \"acquire object from top\" vs. \"place object at bottom\"), thereby missing the discovery of meaningful hierarchical manipulation concepts.\n\nWe acknowledge this merits further investigation and plan to explore additional sub-process derivation methodologies in future work (Sec.D).\n\n# Re Q3: Alternative Concept Integration Strategies\nThank you for this insightful question about our architectural design choices.\n\n## Empirical Comparison\nWe compared using concept encoder features as direct input conditioning versus our joint prediction approach on LIBERO-90 tasks:\n\n| Method | Direct Conditioning | Joint Prediction (Ours) |\n|:-:|:-:|:-:|\n| ACT | 71.1±0.4 | 74.8±0.8 |\n| DP | 79.3±0.9 | 89.6±0.6 |\n\nOur joint prediction approach consistently outperforms direct conditioning across both policy architectures.\n\n## Key Insight: Temporal Alignment\nThe performance difference stems from a fundamental temporal alignment issue:\n- **Direct conditioning**: Uses concepts extracted from current/past observations as additional input features for predicting future actions, creating temporal misalignment.\n- **Joint prediction**: Learns to predict future concepts alongside future actions, creating temporal coherence between concept understanding and action planning.\n\n**Concrete Example**: In a cup placement task, after grasping an object, current concepts encode \"grasping dynamics.\" However, for planning the next placement action, the policy needs to understand \"placement dynamics.\" Our joint approach learns to predict placement-related concepts that directly inform upcoming actions, while direct conditioning provides historical grasping information that may be less relevant for placement planning.\n\nNevertheless, direct conditioning on current/past manipulation concepts could prove valuable for tasks requiring explicit long-term memory [4], and we plan to explore hybrid approaches in future work.\n\n# Re Q4: HiMaCon and VLAs\n\nThank you for this thoughtful question about the relationship between HiMaCon and VLA models. Our manipulation concepts offer **complementary benefits** to VLAs, particularly in training data efficiency.\n\n**Empirical Evidence**: We tested this hypothesis by integrating HiMaCon with OpenVLA-7B using the OpenVLA-OFT protocol [5] and the method in Sec.3.4. We fine-tuned on **half** of the LIBERO-10 demonstration data (for fair comparison, HiMaCon discovered concepts on the same LIBERO-10 data):\n\n| Method | w/o Concept | w/ Concept |\n|:-:|:-:|:-:|\n| Success Rate (%) | 81.0 | 90.2 |\n\nThe improved performance under data-constrained conditions demonstrates that manipulation concepts provide valuable structural inductive biases for data-efficient policy learning.\n\n**Core Hypothesis**: HiMaCon captures fine-grained manipulation dynamics at multiple abstraction levels. Our concepts provide explicit intermediate representations that bridge high-level instructions and low-level actions. This reduces VLAs' learning burden by providing structured manipulation knowledge rather than requiring them to learn complex patterns from scratch.\n\n**Addressing Orthogonality**: With sufficiently large-scale training data, VLAs might internally develop similar manipulation representations simply using action prediction loss. However, robotic data collection faces fundamental scalability challenges. Our approach provides manipulation concepts, which may help VLAs derive similar representations with less data.\n\n**Additional Practical Advantages**: Our method can also extract concepts from non-robotic demonstration data (e.g., human videos) as demonstrated in **Re W1**, enabling diverse data sources to enhance VLA training.\n\n**Future Investigation**: We plan to analyze VLAs at different training scales to understand when explicit concept integration maintains advantages. We will also explore integration strategies for VLAs and concepts discovered by HiMaCon. While theoretical orthogonality remains an open question, we see clear practical value in manipulation concepts for data-efficient robotic learning.\n\n# Re Paper Formatting Concerns\n\nThank you for this helpful formatting suggestion. We will implement the recommended side-by-side layout for Tab.3, Fig.3, and other similar cases.\n\n***\n\nWe sincerely appreciate the reviewer's feedback and hope our clarifications enhance understanding of our work's significance and presentation. Please don't hesitate to share any further thoughts or concerns.\n\n[1] Fu, Z., Zhao, T. Z., & Finn, C. (2024). Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117.\n\n[2] Walke, H. R., Black, K., Zhao, T. Z., Vuong, Q., Zheng, C., Hansen-Estruch, P., ... & Levine, S. (2023, December). Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (pp. 1723-1736). PMLR.\n\n[3] Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., ... & Finn, C. (2024). Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945.\n\n[4] Fang, H., Grotz, M., Pumacay, W., Wang, Y. R., Fox, D., Krishna, R., & Duan, J. (2025). sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation. arXiv preprint arXiv:2501.18564.\n\n[5] Kim, M. J., Finn, C., & Liang, P. (2025). Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645."}]

Author rebuttal:

Meta-review: All reviewers acknowledge the novelty of the proposed method and its superior performance compared to the DP and ACT baselines. However, the paper does not compare against numerous existing works that also predict video frames or other forms of subgoals, nor against several methods that achieve significantly stronger performance on the LIBERO benchmark—such as Flower-VLA (https://openreview.net/pdf?id=ifo8oWSLSq).
Decision: Accept (poster)