STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Authors: Hao Li
Qi Lv
Rui Shao
Xiang Deng
Yinchuan Li
Jianye HAO
Liqiang Nie
Venue: ICML 2025 spotlightposter (2025)
Keywords: Embodied AI
Robotics
Skill Learning
Primary area: deep_learning
TL;DR: We propose STAR,  a framework that advances both skill learning and composition to complete complex behaviors.

Abstract: Transforming complex actions into discrete skill abstractions has demonstrated strong potential for robotic manipulation.Existing approaches mainly leverage latent variable models, e.g., VQ-VAE, to learn skill abstractions through learned vectors (codebooks), while they suffer from codebook collapse and modeling the causal relationship between learned skills. To address these limitations, we present **S**kill **T**raining with **A**ugmented **R**otation (**STAR**), a framework that advances both skill learning and composition to complete complex behaviors. Specifically, to prevent codebook collapse, we devise rotation-augmented residual skill quantization (RaRSQ).It encodes relative angles between encoder outputs into the gradient flow by rotation-based gradient mechanism. Points within the same skill code are forced to be either pushed apart or pulled closer together depending on gradient directions.Further, to capture the casual relationship between skills, we present causal skill transformer (CST) which explicitly models dependencies between skill representations through an autoregressive mechanism for coherent action generation.Extensive experiments demonstrate the superiority of STAR on both LIBERO benchmark and realworld tasks, with around 12% improvement over the baselines.

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented
Vector Quantization
Hao Li Qi
Rui Shao
Xiang Deng 11 Yinchuan Li
Jianye Hao
Liqiang Nie
STAR. gá wig.ttttthh thub.io
Abstract
Decokler
Transforming complex actions into discrete skill abstractions has demonstrated strong potential for robotic manipulation. Existing approaches mainly leverage latent variable models, e.g., VQ-VAE, to learn skill abstractions through learned vectors (codebooks), while they suffer from codebook collapse and modcling the causal relationship be- tween learned skills. To address these limita- tions, we present Skill Training with Augmented Rotation (STAR). a framework that advances both skill learning and composition to complete com- plex behaviors. Specifically, to prevent codebook collapse, we devise -aggeet- residual skill quantization (RaRSQ). It encodes relative angles between encoder outputs into the gradi- ent flow by rotation-based gradient mechanism. Points within the same skill code are forced to be either pushed apart or pulled closer together depending on gradient directions. Further, to cap- ture the causal relationship between skills, we present causal skill transformer (CST) which ex- plicitly models dependencies between skill repre- sentations through an autoregressive mechanism for coherent action generation. Extensive exper- iments demonstrate the superiority of STAR on both LIBERO benchmark and realworld tasks, with around 12% improvement over the baselines.
pants
Uptat
Figure I. Comparison between naive VQ and RaRSQ approach in the skill learning process. Top: Overview of skill quantization framework. Bottom: Visualization of gradient flow and codebook updates across three stages (hefore update, during gradient flow, and after update), where RaRSQ maintains geometric relationships between embeddings, leading to more diverse skills.
et al., 2016: Zhu al., 2018). Individual manipulation tasks already pose significant challenges like multimodal action distributions (Mandlekar et al., 2021), while these chal- lenges are substantially amplified in the multitask setting. This results in a highly entangled action space where char- acteristics of different tasks interact and overlap, making challenging learn complex manipulation behaviors (Lv et al., 2024).
An intuitive approach to alleviate this problem is to learn structured representations of manipulation behaviors by de- composing complex actions into simpler, reusable skill ab- stractions (Fu et al., 2024; Sharma et al., 2019). This hi- erarchical framework reflects the compositional structure inherent manipulation tasks, enabling the systematic de- composition of complex behaviors into sequences of skill abstractions. Recent studies (Ju et al., 2024; Wu et al., 2024) have demonstrated promising results using latent variable models (LVM) to discretize continuous action spaces into learned skills. These methods enable more efficient repre- sentation and composition of complex behaviors.
1. Introduction
The challenge of modeling multitask visuomotor policy has long been central problem in robotic manipulation (Levine
Cooresppnndnng Author 'School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen) 2 Huawei Noah's Ark Lab. Correspondence to: Shao <shaorui www..euuucnn...... Xiang Deng <ddengxiang @ hit..edu.cn.....
Proceedings of the = International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmentted Vector Quantization
However, while this discretization paradigm transforms con- tinuous actions into skills and provides structured represen- tation for complex behaviors, existing LVM-based methods face two critical limitations in skill learning and composi- tion (Garg et al., 2022; Lee et al., 2024). First, techniques like VQ- suffer from codebook collapse (Mentzer et al., 2023; Roy et al., 2018). Most codebook vectors remain unused during training, with only small subset being fre- quently utilized for encoding diverse manipulation skills. This severely limits the capacity to capture the rich variety of robot behaviors. We argue that this limitation stems from the straight-thruuhh gradient estimator (STE) in VQ-VAE (Van Den Oord et al., 2017). During training, as illustrated in the middle column of Fig. 1, STE assigns identical gradients to all encoder embeddings (hollow circles) that are quantized to the same codebook vector (orange solid circles). This oversimplified gradient assignment ignores the inherent ge- ometric relationships between different embeddings within the same partition (regions separated by curved blue deci- sion boundaries), leading to suboptimal codebook updates and eventual collapse of the representation space.
(CST) which explicitly models dependencies between skill representations through autoregressive mechanism for coherent action generation. By leveraging the hierarchical nature of residual quantization, CST sequentially predicts skill codes from coarse to fine levels and incorporates an offset prediction mechanism from BeT (Shafiullah et al., 2022) to bridge the gap between discrete skills and contin- uous control. This combination of hierarchical prediction and continuous refinement enables precise control through- out extended sequences, making it particularly effective for long- hong-horizon manipulation tasks. To summarize, our main contributions are as follows:
A rotation- -uggrettd residual skill quantization (RaRSQ) mechanism that maintains diverse skill representations by encoding relative angular relationships in gradient up- dates, while achieving precise skill abstraction through hierarchical residual encoding.
A causal skill transformer (CST) that models skill depen- dencies through autoregressive prediction and enhances action precision via action refinement.
Second, existing approaches struggle with effective skill composition, particularly in complex, long-horizon tasks that require precise coordination of multiple skills (Mete et al., 2024). While some methods adopt residual quanti- zation (RQ) (Zeghidour et al., 2021) to decompose skills into multiple levels for more precise representation, they fail to model the dependencies between different skill abstrac- tions (Lee et al., 2024). This makes it difficult to generate coherent and temporally consistent actions for multi-stage manipulation tasks, where skills need to be carefully se- quenced and composed.
Comprehensive experimental validation across multiple benchmarks and real-world tasks, demonstrating substan- tial improvements in both skill learning efficiency and task performance.
2. Related Work
Multi-task Imitation Learning. Multi-task robotic learn- ing has been approached through various methods including supervised pre-training (Sun al., 2023; Wu et al., 2023; Li et al.) and large-scale demonstration learning (Vuong et al., 2023; Brohan et al., 2023; Li et al., 2025). Recent ad- vances have explored the use of generative models, with frameworks like diffusion models (Chi et al., 2023; Ze et al., 2024; Lv et al., 2025) and transformer-based ap- proaches (Pertsch et al., 2025; Bharadhwaj et al., 2024) showing promising results in handling multimodal action distributions. The Behavior Transformer (BeT) (Shafiul- lah al., 2022) demonstrated that policies operating in discretized action spaces can effectively model diverse be- haviors, and introduced an offset prediction mechanism to handle the discretization-inuueed precision loss. Action Chunking Transformer (ACT) (Zhao et al., 2023) further addressed temporal correlations by predicting action chunks. Unlike existing methods, STAR advances this line of work by introducing novel mechanisms for learning structured skill representations while preserving the geometric rela- tionships inherent in continuous action sequences.
To address these fundamental challenges, we propose Skill Training with Augmented Rotation (STAR), a novel frame- work that advances both skill learning and composition for robot manipulation. Our key insight is that encoding ge- ometric relationships between action sequences into the residual quantization process is crucial for learning diverse and reusable skills, rather than relying on the oversimplified gradient assignment of ttriigh-throuuh estimation. Specifi- cally, (1) to prevent codebook collapse, we devise rotation- augmented residual skill quantization (RaRSQ), which com- bines multi-level residual encoding with edtation-base gra- dient mechanisms. The residual structure progressively cap- tures skills at different abstraction levels, while the rotation- augmented gradient flow enables points within the same skill code to be either pushed apart or pulled closer together based on their geometric relationships. Compared to naive VQ-VAE where similar embeddings are forced to have iden- tical gradients, Fig. 1 demonstrates that RaRSQ prevents em- beddings from collapsing the same codebook vector, lead- ing to more diverse skill representations; and (2) for effee- tive skill composition, we present causal skill transformer
Robotic Manipulation in Learned Latent Spaces. Latent Variable Models (LVMs) have emerged as powerful tools for learning structured representations in robotics (Yang
2

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
al., 2024; Luo et al., 2023; Bharadhwaj al., 2023), particularly for offline imitation learning and skill abstrac- tions. Among them, methods like LAPA (Ye et al., 2024), IGOR (Chen et al., 2024), leverage internet-scale human videos to learn transferable manipulation skills. Several works have explored discrete latent spaces for skill rep- resentation. PRISE (Zheng et al., 2024) employs BPE tokenization for temporal abstraction but struggles ef- fectively encode varied action distributions across tasks.
Algorithm Rotation-auugmented Residual Skill Quantiza- tion (RaRsQ)
Require: action sequence a codebooks {C,)
I: z + (ncoder(aue+r))
2: Tu f
3: for d
do
4:
Quantize current residual
5: kig < arg ming ||ra- edd.a;
6:
I/ Compute rotation matrix that aligns rá-1 to @(d.ka)
7:
Ra < ContputeRottiionnra.... ed.,g)
TAP (Jiang et al., 2022) and H-GAP (Jiang et al., 2023) use self-supervised autoencoders for skill learning but rely heav- ily on state-based model predictive control, limiting their real-world applicability. QueST (Mete et al., 2024) learns discrete latent skills with temporal dependencies, but strug- gles to learn diverse skill representations. VQ-BeT (Lee al., 2024) shares our motivation of using discrete latent skills for transformer-basee based policies. but their standard quan- tization approach suffers from codebook collapse and fails to capture temporal dependencies between skills. Unlike existing approaches, STAR addresses both the representa- tional and temporal challenges through a two-stages frame- work that combines ootation-augmenter quantization for preventing codebook collapse with explicit modeling of skill dependencies for coherent behavior generation.
8:
Apply rotation with stop- gradient to preserve geometric
structure
9:
qd < sg ||x ill
R ra-1
10:
// Updatt residual for next level
11: Id f- - qd
12: end for
13: 2 + Σb qd
14: à < Decoder(2)
15: return reconstructed action a, codes (ka 1
commitment losses:
=
|| - -él12 + Blle - sg()))13
(4)
where sg(-) denotes the stop-gradient operator and / is a hyperparameter scaling the commitment loss for multi-stage residual learning stability (Lee et al., 2022).
3. Method
3.1. Preliminaries
Due to the non-differentiabbiity of the quantization operation Q(-), the trriiht-through estimator (STE) is employed for backpropagation by simply copying gradients from è to e through setting dē de/ = I. This residual approach enables more precise approximation than standard VQ- VQ-VAE with codebook size K and depth D, it can effectively represent KD distinct quantization outputs while maintaining better computational efficiency and training stability.
Residual VQ-VAE and STE. VQ-VAE with residual quan- tization transforms continuous data into hierarchical dis- crete representations through multi-stage quantization pro- cess (Adiban et al., 2022). It consists of three key com- ponents: encoder E, decoder D, and multiple code- books C, Given an input x E IR", the encoder & first maps it to a continuous latent code e E RT. Then, e are quantized using D codebooks, where each codebook C. = (e ((,k)) contains K learnable vectors.
Rotation Trick. To address the limitations of STE in pre- serving geometric relationships during gradient propagation, recent work(Fifty et al., 2024) proposes the rotation trick that transforms encoder outputs codebook vectors via rotation and rescaling. For encoder output and codebook vector q, it computes:
Starting with the initial residual ro e, residual quan- tization iteratively performs nearest neighbor lookup and residual computation:
ky = Q(ra-1:Ca) = arg meinge(l....... |[r-- - e(.,))
(1)
q = |q||
(2)
R
(5)
where ki is the selected code index at depth d, and ra is the remaining residual to be quantized by subsequent code- books. The final quantized representation é is obtained by summing the selected code vectors:
where R the rotation matrix that aligns e with q. Dur- ing backpropagation, the rotation transformation preserves relative angles between gradients and vectors, enabling dif- ferent points within the same quantization region to receive varying gradient updates based on their geometric relation- ships. This mechanism helps prevent codebook collapse and maintain diverse vector representations by encouraging appropriate exploration of the latent space.
Σ (d.kk))
(3)
The decoder D then reconstructs the input: x = D(è). The model trained with a combination of reconstruction and
3

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Quantizer
Skall Learning
Encoder
Decoder
Rotation Quannierr
Robnion Unantizer
Ground-truth action
Image Encoder
Offset Head
Casual Transformer
Loss
CST Training
Observations
State Encoder
Skäll Predictor
Text Encoder
Decoder
Proprioception
Instruction:
"upen drawer" "!
Skill Abstraction
Ground-truth
Encoder
Action Sequence
Figure 2. Overview of the STAR framework. Top:
The RaRSQ module encodes continuous action sequences into hierarchical discrete residual quantization. Bottom: The CST module processes multimodal inputs (visual observations,
skills through proprioceptive states, and language instructions) to generate actions through autoregressive skill prediction and action refinement.
3.2. STAR
d == (1, DJ. quantize and rotate the residual:
3.2.1. OVERVIEW
ka arg ming
||a-1
(d,,c)
l1
(6)
|
In this section, we present the Skill Training with Aug- mented Rotation (STAR) framework, which employs Rotation-augmented Residual Skill Quantization (RaRSQ) and Causal Skill Transformer (CST) to enhance skill leamn ing and composition in robotie manipulation. The frame- work of STAR shown in Fig. 2. STAR adopts two-stage training strategy: first training RaRSQ to learn skill abstrac- tions, then fixing RaRSQ train CST for skill composition.
q sg
|ra-11
|l
R
rd-1
(7)
ra rd-1
qd
(8)
where Cd,k, represents the ka-th vector in codebook Ca. and sg[:] denotes the stop-gradient operator. The rotation matrix R is computed as:
R. -- 2rará + 2q
3.2.2. ROTATION-AUGMENTD RESIDUAL SKILL QUANTIZATION
where
qu (d.ka) /llaaallllllll - (
(10)
To learn expressive and reusable skill representations from continuous action sequences, we propose Rotation- augmented Residual Skill Quantization (RaRSQ). Our ap- proach addresses key limitations of standard VQ-VAE: codebook collapse and inefficient skill representation. By integrating rotation transformations with residual quanti- zation, RaRSQ preserves geometric relationships between action sequences while enabling hierarchical skill encoding.
P--1
qd
F2
(11)
|
qull
The final skill representation is obtained by summing the rotated quantized vectors:
qd
(12)
During backpropagation, gradients flow through the rotation matrices:
Skill Encoding Process. Given an action sequence A we first encode it into a latent vector Φ( ((a..rT) through encoder network (see Algorithm 1). RaRSQ then discretizes through an iterative process as follows:
02
||
R.
(13)
Dr
|
This rotation-based gradient mechanism enables different points within the same quantization region to receive vary- ing updates based on their geometric relationships, effec- tively preventing codebook collapse.
Starting with the initial residual ro = z, for each depth

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Theoretical Benefits. Our formulation offers three key advantages:
hierarchical process where each skill depends on previous choices:
1. Improved Skill Diversity: The rotation-basee gradient mechanism prevents codebook collapse by preserving ge- ometric relationships between actions and corresponding skills, Our rotation transformation enables varied up- dates based on relative angular relationships, preventing skills from collapsing small subset of codes.
D
P(k kplo:-htt Aets T) II P(kalkica88/ (18)
where ked represents all previously predicted skill codes, and ka E {1,..., }} denotes the index selected from the d- th codebook. For each depth d, a prediction head Ça outputs categorical distribution over the K possible codebook indices, enabling the model to select appropriate skills at each level of abstraction. This autoregressive formulation is crucial as it captures the natural dependency structure in our residual skill space coarse movement primitives must be selected before fine-grained adjustments.
2. Hierarchical Skill Structure: The residual quantization naturally decomposes actions into a hierarchy, where 11 captures coarse primitives while subsequent codes encode finer details, matching the inherent structure of manipulation tasks.
The combination of residual quantization and rotation-augmented gradi- ents enables representing K2 distinet skills with only K' codes per level, while maintaining low quantization errors compared to naive VQ approaches.
Enhanced Representation Capacity:
Action Refinement. While the predicted codes can be di- rectly decoded to actions using the decoder in RaRSQ, dis- cretization of the continuous action space inevitably leads to some loss of fidelity (Shafiullah al., 2022). Following BeT, we introduce a refinement mechanism through an off- set prediction head to bridge this gap. Specifically, we add an offset head Gref to predict continuous refinements to the discretized actions. The final action is computed as:
Training Objective. We train RaRSQ using a combination of reconstruction and commitment losses:
Lrecon + Lcommit
(14)
Lrecon = ||ar:r - (2)
(15)
||ca.kal ll
Ccommit =3
>11 sg 1a- d=1
| R R
11
(16)
â 1 L Raed + (g:)
(19)
where & is the decoder, sg[-] denotes mrop-gradient, and / a weighting coefficient. The reconstruction loss Lreeon ensures accurate action reconstruction, while the commit- ment loss Lcommit encourages the residuals to stay close to their corresponding rotated and scaled skill abstractions, maintaining geometric relationships during quantization.
where v is the RaRSQ decoder from Section 3.2.2, ed, is the selected codebook vector at depth d, and R, is the corresponding rotation matrix.
Training Objective. We optimize our framework using a combination of skill prediction and refinement losses:
D L C= Σ log P(kglked 8:) + Alla: à
3.2.3. CAUSAL SKILI. TRANSFORMER
(20)
To effectively compose learned skills for sequential manipu- lation tasks, we propose the Causal Skill Transformer (CST) that combines autoregressive skill prediction with adaptive refinement. Our framework explicitly models the hierar- chical dependencies between skills while enabling precise action generation through refinement.
where Kū are the ground truth codes from RaRSQ encoding of the expert action a, and is a trade-off coefficient.
3.2.4. INFERENCE PROCESS
At inference time, framework generates actions through an efficient two-stage process that balances exploration and precision. Given the current observation context Oz-k: and task instruction E, we perform:
Input Representation. Given a sequence of observations Ou-hut = {ix ). where is and ps represent visual and proprioceptive inputs at timestep kt and a task instruc- tion т, we first encode the multimodal inputs using:h; = [fvis (ix);
Hierarchical Skill Selection. We first sample skill codes (k), kp) autoregressively using nucleus sampling with temperature 7 and threshold p:
where fvis and fpeop are vision and proprioceptive encoders respectively. The transformer- based policy Ty then processes these encodings along with the task embedding to generate contextual features:
ha NucleusSanplee(P(kkkee. (a.. p.).
(21)
g: [ h.)..)
(17)
Action Generation and Execution. The sampled skill codes are mapped to their corresponding codebook vec- tors and combined with predicted refinements to generate
Hierarchical Skill Prediction. Building on our residual quantization framework, CST models skill selection as a
5

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Method
LIBERO-Object
LIBERO-Spatial
LIBERO-Goal
LIBERO-Long
LIBERO-90
Overall
Octo!
85.7
+:0.9
78.9 +1.0
84.6 +0.9
51.1
+1.3
75.1 =0.6
OpenVLA!
88.4 10.8
84.7
+0.9
79.2 #1.0
53.7
#1.3
76.5 =0.6
ResNet-T
78.9 11.4
75.7 11.9
$2.7 12.4
45.0 111
83.9 11.5
67.3 =099
Diffusion Policy
62.6 12.8
69.5 11.
10.5
$1.2
75.3
10.7
62.6 0066
ACT
78.8 +1.2
82.0 +0.5
66.1 +1.6
44.0 +0.5
63.4
+:588
66.8 =1.1
VQ-BeT
90.3 + 1.5
88.7 +:20
61.3 +1.0
59.7 +0022
84.2
+:03
76.8 005
QueST
90.0 111
84.5 +.0.
76.7
10.9
69.1 1100
87.4
1:0.4
81.5 0.6
Ours
98.3 0.2
95.5 +0.6
95.0 +:0.7
88.5 +003
90.8
#:0.2
93.6
=0.1
Table Overall Performance. The results of the baselines marked with are cited from their original papers. Bold indicate the highest score. We report the and standard deviation of the normalized score with three random seeds.
actions:
94
90.6
88.2
actth 1/
ed Coffset (g:)
(22)
88.4
87.7
Mean Success Rate
The system executes the generated action sequence and up- dates observations before re-planning. This rolling horizon approach allows our framework to adapt to environment dynamics while maintaining behavioral consistency through the learned skill space.
86
4. Experiment
4.1. Setup and Baselines
VQ-BeT
QuesT
Policy
We evaluate STAR on two comprehensive manipulation benchmarks: LIBERO (130 tasks across five suites) and MetaWorld MT50 (50 distinct manipulation tasks), plus two real-world long-horizon tasks. Success Rate (SR) is mea- sured over 50 episodes per task with three random seeds. Detailed descriptions are in Appendix A. We compare STAR against three categories of state-of-the-arr methods: (1) Dis- crete LVM approaches, (2) End-to-end imitation learn- ing, (3) Large-scale VLA models. Detailed descriptions of these baselines can be found in Appendix A.5.1.
Figure 3. Performance comparison on the MetaWorld MT50 bench- mark. STAR achieves consistently superior performance (92.7%) compared to baseline methods across 50 manipulation tasks.
collapse issue discrete latent approaches like VQ-BeT and QueST. Notably, STAR even outperforms large-scale models like Octo and Open despite their access to significantly more training data.
To further validate the capability of our approach across different manipulation scenarios, we evaluate STAR on the Meta World MT50 benchmark. As demonstrated in Fig. 3, the strong performance extends to the MetaWorke MT50 benchmark, where STAR achieves 92.7% average success rate across all 50 tasks, outperforming existing methods with a margin of 2.1 11%55%%% The consistent improvements across both manipulation benchmarks demonstrate the ef- fectiveness of STAR as general framework for learning diverse robot skills.
4.2. Overall Performance
As shown in Table 1, STAR significantly outperforms all baselines across different LIBERO task suites, achiev- ing 93.6% overall success rate and surpassing the previ- ous state-of-the-art QueST by 12.1% % (81.5%). The perfor- mance improvement is particularly pronounced on LIBERO- Long tasks (88.5% vs. 69.1%, 1%, +19.4%) and complex manipulation scenarios like LIBERO-Objer Object (98.3% vs. 90.0%, +8.3%). Compared with other baselines, STAR demonstrates consistent improvements across all task cate- gories. For basic manipulation tasks in LIBERO-Object and LIBERO-Spatial, our method achieves 7.2%-12.6%% higher success rates. The improvement margins expand signifi- cantly to 18. 18.3%-33.999 9% on more challenging LIBERO-Goal and LIBERO-Long tasks.
4.3. Analysis of Learned Skill Diversity
To evaluate the effectiveness of STAR in preventing code- book collapse, we conduct quantitative analysis of the learned skill representations. Fig. 4 reveals several find- ings that demonstrate the superiority of our approach over naive VQ-VAE First, STAR achieves complete codebook
This larger gap on complex tasks stems from the codebook
6

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Method
LIBERO-Object
LIBERO-Spatie
LIBERO-Goal
LIBERO-Loon
LIBERO-90
Overall
Ours
98.3 +002
95.5 +0.6
95.0 #0.7
88.5
+0.3
90.8 +0.2
93.6 #001
w/o AR
95.3
+0.5
94,3 +0.6
88.1 +0.2
83.3
+1.1
86.4 +0.2
89.5 +0.1
w/o Rotation
93.7 +0.2
94.6 +0.3
91.3 +0.3
85.7
+0.8
89.7 40.2
91.0 10.3
w/o Rotation and AR
93.3 +0.4
91.9 +0.8
86.9 +0.2
81.5 #1.1
85.5 #2.0
87.8 #0.4
Table 2. Ablation Results.
"w/o AR/Rotation" represents removing the module of auto regressive and rotation matrix, respectively. Best
repressnt
results marked bold.
Codebook Usage Distribution
gradient in RaRSQ, using standard straight-through estima- tion; (3) w/o Rotation and AR: Removes both components.
7,36
RARSQ Nolvo vo VAE
14-
5.2%
12.0%
10.9%
The results demonstrate that both components are essential for strong performance. First, removing the autoregressive prediction (w/o AR) significantly impacts the ability to cap- ture skill dependencies, leading a substantial performance drop (89.5% 93.6% overall). This degradation is partic- ularly severe tasks requiring precise skill sequencing, such as LIBERO-Goal (-6.9%) and LIBERO-Long (-5.2%). Without autoregressive prediction, the model struggles to maintain temporal coherence between selected skills, re- sulting in fragmented or inconsistent behavior sequences. The rotation- augmented gradient also proves crucial for effective skill learning. Removing this component (w/o Rotation) leads to degraded performance (91.0% overall), with the impact most pronounced on LIBERO-Object (- 4.6%). This aligns with our theoretical analysis without augmented gradients, the model suffers from code- book collapse and fails to maintain diverse skill representa- tions. The performance decline is especially noticeable in tasks requiring varied manipulation skills, where having rich repertoire of distinct skills is essential.
12
1.2%
5.0%
10-
2.11%
Code Index
25.5%
6.80%
1.5%
1.8%
3.4%
13.5%
20:15
10.7%
5:00k
10
Usage Rabe (%)
Figure 4. Analysis of codebook utilization patterns. Comparison between RaRSQ (light blue) and naive VQ-VAE (dark blue) ap- proaches shows RaRSQ achieves complete codebook utilization across all 16 codes, while naive VQ-VAE exhibits severe collapse with only 7 active codes. The percentage values indicate the usage frequency of each codebook index during training.
utilization with all 16 codes being actively engaged in skill representation, while naive VQ-VAR exhibits severe col- lapse, utilizing only 43.8% of its codebook capacity (7 out of 16 codes). This comprehensive utilization indicates that RaRSQ successfully maintains diverse skill abstractions throughout the learning process. Second, beyond mere uti- lization, RaRSQ demonstrates significantly more balanced skill distribution across its codebook. The mean utilization frequency per code is 6.25%, approaching the theoretical optimal uniform distribution. In contrast, VQ-VAE shows a highly skewed distribution with 14.29% mean utilization per active code, indicating overreliance on limited subset of representations. RaRSQ crucially maintains this healthy variation across its entire codebook rather than concentrat- ing it in a small subset of active codes.
When both components are removed (w/o Rotation and AR), we observe the most severe performance degradation 37.8% overall). This synergistic effect demonstrates how our two-stage approach first learning diverse skills through iration-augmented quantization, then capturing their causal re relationships through autoregressive prediction is crucial for effective skill learning and composition. The substan- tial performance gap (5.8% lower than STAR) validates our design principle of combining geometric structure preser- vation for skill diversity with explicit temporal dependency modeling for skill composition.
Sequential Stages
Overall
Method
Open
+Place
+Close
Success
4.4. Ablation Study
VQ-BET
4/10
3/10
1/10
1/10
QueST
3/10
1/10
0/10
Q/10
To evaluate the contribution of each key component in STAR, we conduct ablation studies by removing critical modules. The results are shown in Table 2. We compare the following variants: (1) w/o AR: Removes the autoregressive prediction in CST, directly predicting all skill codes indepen- dently; (2) w/o Rotation: Removes the rotation- augmented
Ours
6/10
4/10
3/10
3/10
Table 3. Performance on sequential manipulation task (drawer op- eration). Results show successful trials out of 10 attempts for each stage and overall task completion.
7

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmenned Vector Quantization
Instruction: Pick the cube into the plate and pick the toy into the box.
start
move
down
pick
up
place
c
move
pick
up
move
place
Instruction: Open the drawer, place the toy the drawer and then close it.
start
move
grasp
open
relax
move
pick
up
move
place
move
push
Figure 5. Visualization of two real-worle manipulation tasks with key execution stages. Each frame shows critical step in the manipulation sequence.
Task Completion
Overall
complete task success compared to 10% for VQ-BeT and 0% for QueST. The performance degradation from open- ing (60%) to complete execution (30%) reveals the com- pounding difficulty of maintaining precise control through extended sequences, though the degradation of STAR is notably less severe than the baselines.
Method
Cube-Plate
Toy-Box
Success
VQ-BET
5/10
3/10
3/10
QueST
4/10
4/10
Ours
8/10
6/10
6/10
Table 4. Performance on sequential manipulation task (sequential object placement). Results show successful trials out of 10 attempts for each subtask and overall completion.
As show in Table 4, the sequential object placement results further demonstrate the effectiveness of STAR in handling varied manipulation skills. STAR achieves 60% success rate for complete task execution, significantly outperforming VQ-BeT and QueST. The performance difference between initial cube placement and the more challenging toy place- ment aligns with task complexity, the second placement requires more precise control given the confined space of the box. These results, visualized in Fig. 5, demonstrate that the improvements of STAR in skill learning and composition are effective in real-world scenarios.
4.5. STAR on Real-World Robots
To validate the effectiveness of STAR beyond simulation, evaluate our approach
challenging real-world ma- nipulation tasks: sequential object placement task ("Pick the cube into the plate and pick the toy into the box") and structured drawer manipulation sequence ("Open the drawer, place the toy in the drawer and then close it"). These tasks mirror the complexity of LIBERO-Long tasks while intro- ducing real-world challenges like lighting variations and physical dynamics.
5. Conclusion
In this paper, we presented STAR, framework for learn ing and composing diverse robot skills through rotation- augmented vector quantization. Through RaRSQ and CST, our approach effectively prevents codebook collapse while enabling precise skill composition. Extensive experiments demonstrate the strong performance of STAR across manip-
For the drawer manipulation task, shown in Table 3, the results highlight key advantage of the hierarchical skill decomposition. While all methods can initiate basic ac- tions like drawer opening (60% success rate for STAR), performance degrades through complex sequences. STAR maintains higher success rates across stages, achieving 30%
8

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
ulation benchmarks, significantly outperforming state-of- the-art methods in both simulation and real-world settings.
ence on Robotics and Automation (ICRA), pp- 4798.
IEEE, 2024.
Limitations. While STAR demonstrates strong perfor- mance across benchmarks, our approach requires pre- defined codehook sizes and quantization depths, which must be manually tuned for different task domains. Further, as an imitation learning approach, STAR depends on the qual- ity of available expert demonstrations, which may limit its applicability when such data is scarce.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., Florence, P., Fu, C., Arenas, M. G., Gopalakrishnan, K., Han, K., Hausman, K., Herzog, A., Hsu, J., Ichter, B., Irpan, A., Joshi, N., Julian. R., Kalashnikov, D., Kuang, Y.. Leal, I., Lee, L., Lee. T.-W. E., Levine, S., Lu, Y., Michalewski, H., Mordatch, I., Pertsch, K., Rao, K., Rey- mann, K.. Ryoo, M., Salazar, G., Sanketi. P., Sermanet, P., Singh, J., Singh, A., Soricut, R., Tran, H., Vanhoucke, V. Vuong, Q.. Wahid, A., Welker, S., Wohlhart, P., Wu, J., Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, models transfer web
Acknowledgements
We would like to thank all co-authors for their efforts and the reviewers for their constructive comments. This work is supported by National Natural Science Foundation of China (Grant No. 62306090), National Natural Science Founda- tion of China (Grant No. 62406092), National Natural Sci- ence Foundation of China (Grant No. U24B20175), Natural Science Foundation of Guangdong Province of China (Grant No. 2024A15 15010147,, Guangdong Basic and Applied Basic Research Foundation (Grant No. 2025A151501016% Shenzhen Science and Technology Program (Grant No. Shenzhen Science and Tech- nology Program (Grant No. KJZD20240903 10001 7022), and Research on Efficient Exploration and
B. Rt-2: Vision- Vision-language-actie knowledge to robotic control, 2023.
Chen, X., Guo, J., He, T., Zhang, C., Zhang, P., Yang, D. C., Zhao, L.,, and Bian, J. Igor: image-gode representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:241 1.00755, 2024.
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273668, 2023.
KQTID2024022
of APP Agents & Embodied Intelligent Cerebellum Control Model and Collaborative Feedback Training Project (Grant No. TC20240403047).
Fifty, C.,, Junkins, R. G., Duan, D., Iger, A., Liu, J. W., Amid, E., Thrun, S., and Ré, C. Restructuring vec- tor quantization with the rotation trick. arXiv preprint urXiv:2410.06422 2024.
Impact Statement
Fu, H., Sharma, P., Stengel-Eskin, E., Konidaris, G., Roux, N. L., Côté, M.-A., and Yuan, X. Language-guide skill learning with temporal variational inference. arXiv preprint arXiv:2402.16355 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Garg, D., Vaidyanath, S., Kim, K., Song, J., and Ermon, S. Lisa: Learning interpretable skill abstractions from language. Advances in Neural Information Processing Systems, 35:21711-21724 2022.
References
Adiban, M., Siniscalchi, M., Stefanov, K., and Salvi, G. Hierarchical residual learning based vector quantized variational autoencorder for image reconstruction and generation. In 33rd British Machine Vision Conference, 2022.
Jiang, Z., Zhang, T., Janner, M., Li, Y., Rocktäschel, T., Grefenstette, E., and Tian, Y. Efficient planning in a com- pact latent action space. arXiv preprint arXiv:2208. 1029/, 2022.
Bharadhwaj, H., Gupta, A., and Tulsiani, S. Visual affor- dance prediction for guiding robot exploration. 2023 IEEE International Conference on Robotics and Automa- tion (ICRA), pp. 3029-3036. IEEE, 2023.
Jiang, Z., Xu, Y., Wagener, N., Luo, Y., Janner, M., Grefen- stette, E., Rocktäschel, T., and Tian, Y. H-gap: Hu- manoid control with a generalist planner. arXiv preprint ar arXiv:2312.02682, 2023.
Ju, Z., Yang, C., Sun, F., Wang, H., and Qiao, Y. Rethink- ing mutual information for language conditioned skill discovery on imitation learning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 34, pp. 301-309, 2024.
Bharadhwaj. H., Vakil, J.. Sharma, M., Gupta, A., Tulsiani, S., and Kumar, V. Roboagent: Generalization and effi- ciency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Confer-

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmentedd Vector Quantization
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.09222 2024.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. 2023.
Mete, A., Xue, H., Wilcox, A., Chen, Y., and Garg, A. Quest: Self-supervised skill abstractions for learning continuous control. arXiv preprint arXiv:2407.15880 2024.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-. Autore- gressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1523-11532, 2022.
Pertsch, K. Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q. Mees, O., Finn, C., and Levine, S. Fast: Efficient action tokenization for vision- iisiio-language-ctionn models. arXiv preprini urXiv:2501.0974 2025.
Lee, S., Wang, Y., Etukuru, H., Kim, H. J., Shafiullah, N. M. M., and Pinto, L. Behavior generation with latent actions. arXiv preprint arX1v:2403.03181. 2024.
Roy, A., Vaswani, A., Neelakantan, A., and Parmar, N. The- ory and experiments on vector quantized autoencoders. arXiv preprint arXiv: 1066, 2018.
Levine, S., Finn, C., Darrell, T., and Abbeel, End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40,, 2016.
1063,
Shafiullah, N. M., Cui, Z., Altanzaya, A. A., and Pinto, Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35: 22955-22968, 2022.
Li, Z., Xie, Y., Shao, R., Chen, G., Jiang. D., and Nie, L. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery skills. arXiv preprint 197.01990. 2019.
Li, Z., Xie, Y., Shao, R., Chen, G., Jiang, D., and Nie, L. Optimus-2: Multimodal minecraft agent with goal- observation action conditioned policy. arXiv preprint arXiv:2502. 19902, 2025.
Sun, Y., Ma, S., Madaan, R., Bonatti, R., Huang, F, and Kapoor, A. Smart: Self-supervised multi-task pretraining with control transformers. arXiv preprint arXiv:2301.098/1 2023.
Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q. Zhu, Y., and Stone, P. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
Team, O. M.. Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
Luo, J., Dong, P., Wu, J., Kumar, A., Geng, X., and Levine, S. Action-quantized offline reinforcement learning for robotic skill learning. In Conference on Robot Learning, pp. 1348-1361. PMLR, 2023.
Van Den Oord, A., Vinyals, O., al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Vuong, Q., Levine, S., Walke, H. R., Pertsch, K., Singh, A., Doshi, R., Xu, C., Luo, J., Tan, L., Shah, D., Finn, C., Du, M. Kim, M. J., Khazatsky, A., Yang, J. H., Zhao, T. 7.,, Goldberg, K., Hoque, R., Chen, L. Y., Adebola, S., Sukhatme, G. S., Salhotra, G., Dass, S., Pinto, L., Cui, Z. J., Haldar, S., Rai, A., Shafiullah, N. M. M., Zhu, Y., Zhu, Y., Nasiriany, S., Song, S., Chi, C., Pan, C., Burgard, W., Mees, O., Huang, C., Pathak, D., Bahl, S., Mendonca,
Lv, Q., Li, H., Deng, X., Shao, R., Wang. M. Y., and Nie, L. Rohomp2: a robotic multimodal iinception-planniing framework with multimodal large language models. In Proceedings of the 41 International Conference on Ma- chine Learning, pp. 33558-33553 33574, 2024.
Lv, Q., Li, H., Deng, X., Shao, R., Li, Y., Hao, J., Gao, L., Wang, M. Y., and Nie, L. Spatial-temporal graph diffusion policy with kinematic modeling for bimanual robotic manipulation. arXiv preprint arXiv:2503.1077 2025.
R., Zhou, G., Srirama, M. K., Dasari, S., Lu, C., Fang, H.- S., Fang, H., Christensen, H. I., Tomizuka, M., Zhan, W., Ding, M., Xu, C., Zhu, X., Tian, R., Lee, Y., Sadigh, D., Cui, Y., Belkhale, S., Sundaresan, P., Darrell, T., Malik, J., Radosavovic, I., Bohg. J., Srinivasan, K., Wang, X., Hansen, N., Wu, Y.-H., Yan, G., Su, H., Gu, J., Li, X., Suenderhauf, N., Rana, K., Burgess-Limerick, B., Ceola, F., Kawaharazuka, K., Kanazawa, N., Matsushima, T., Matsuo, Y., Iwasawa, Y., Furuta, H., Oh, J., Harada, T.
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L.. Savarese, S., Zhu, Y., and Martín-Martin R. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint ar 2rxiv:22 108 100.0328, 2021.
10

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Osa, T., Tang, Y., Kroemer, O., Sharma, M., Zhang, K. L., Kim, B., Cho, Y., Han, J., Kim, J., Lim, J. J., Johns, E., Palo, N. D., Stulp, F., Raffin, A., Bustamante, S., Silvério, J.. Padalkar. A., Peters, J., Schölkopf, B., Büchler, D., Schneider, J., Guist, S., Wu, J., Tian, S., Shi, H., Li, Y., Wang. Y., Zhang, M., Amor, H. B., Zhou, Y., Majd, K., Ott, L., Schiavi, G., Martín-Martín, R., Shah, R., Bisk, Y., Bingham, T., Yu, T., Jain, V., Xiao, T., Hausman, K., Chan, C., Herzog, A., Xu, Z., Kirmani, S., Vanhoucke, V., Julian, R., Lee, L., Ding, T., Chebotar, Y., Tan, J., Liang, J., Mordatch, I., Rao, K., Lu, Y., Gopalakrishnan, K., Welker, S., Joshi, N. Devin, C. M., Irpan, A., Moore, S., Wahid, A., Wu, J.. Chen, X., Wohlhart, P., Bewley, A., Zhou, W., Leal, I., Kalashnikov, D., Sanketi, R., Fu, C., Xu, Y., Xu, S., brian ichter, Hsu, J., Xu, Brohan, A., Sermanet, P., Heess, N., Ahn, M., Rafailov, R., Pooley, A., Byrne, K., Davchev, T., Oslund, K., Schaal, S., Jain, A., Go, K., Xia, F., Tompson, J., Armstrong, T., and Driess, D. Open x-embodiment: Robotic learning datasets and RT-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 2023. URL httpt:://////.. htttps::/ppee.eev..eee wwt. forum? id id=zraBtFgxTO.
Zheng, R., Cheng, C.-A., Daumé III, H., Huang, F,, and Kolobov, A. Prise: Learning temporal action abstrac- tions sequence compression problem. arXiv preprint arXiv:2402. 10450, 2024.
Zhu, Y., Wang. Z., Merel, J., Rusu, A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kramár, J., Hadsell, R., de Freitas, N., et al. Reinforcement and imitation learning for di- verse visuomotor skills. arXiv preprint arXiv 1xxx::802.09966 2018
Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. Unleashing scage video generative pre-training for visual robot manipula- tion. arXiv preprint arXiv:2312.13133 2023.
Wu, K., Zhu, Y., Li, J., Wen, J., Liu, N., Xu, Z., Qiu, Q., and Tang, J. Discrete policy: Learning disentangled action space for multi-task robotic manipulation. arXiv preprint arXiv:2409. 187077 2024.
Yang, C., Liconti, D., and Katzschmann, R. K. Vq-ace: Efficient policy search for dexterous robotic manipu- lation via action chunking embedding. arXiv preprint arXiv:2411.0355 2024.
Ye. S., Jang. J., Jeon, B., Joo, S., Yang, J., Peng, B., Mandlekar, A., Tan, R., Chao, Y.W.,, Lin, B. Y., al. Latent action pretraining from videos. arXiv preprint 0rXiv:2410.11758 2024.
Ze, Y., Zhang, G., Zhang, K., Hu, C., Wang, M., and Xu, H. diffusion policy. arXiv preprint 4rX1v.2403.3355, 2024.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Zhao, T. Z.,, Kumar, V., Levine, S., and Finn, C. Learn- ing fine-graine bimanual manipulation with low-cost hardware, arXiv preprint arXiv:2304. 2023.

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
A. Experimental and Dataset
Put the yellow and white mug in the microwave and close it demo
turn on the stove and put the moka pot it demo
put both the alphabet soup and the cream cheese box the basket demoput the moka pot on demo
Figure The visualization simulated tasks, including
A.1. Benchmark Environments
We evaluate STAR across multiple comprehensive benchmarks spanning both simulated and real-world environments assess its effectiveness in diverse manipulation scenarios.
A.1.1. LIBERO BENCIMARK
LIBERO is a comprehensive benchmark for language-conddtioned manipulation tasks that evaluates different aspects of robotic manipulation capabilities through five distinct task suites:
LIBERR-Spatial contains 10 tasks focusing on identical objects different spatial layouts, testing the understanding of spatial relationships.
LIBERO-Object comprises tasks with consistent layouts but varying objects, evaluating the ability to generalize across different object types.
LIBERO-Goal features 10 tasks sharing the same object categories and spatial layouts but with different goals, assessing the capability to learn diverse task-oriented behaviors.
LIBERO-Long presents 10 challenging long-horizon tasks involving diverse object categories and layouts, testing temporal reasoning and sequential manipulation skills.
LIBERO-90 encompasses 90 tasks with extremely diverse object categories, layouts, and task goals, providing comprehensive evaluation platform.
For our experiments, we utilize 50 expert demonstrations per task from the author- provided dataset. These demonstrations capture diverse manipulation strategies and initial conditions, enabling robust policy learning.
12

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
A. 1.2. METAWORLD MT50
Meta World MT50 is challenging multi-task benchmark consisting of 50 distinct manipulation tasks performed by Sawyer robotic arm in simulation. The tasks range from basic object manipulation (e.g., pushing, pulling) complex tool usage (e.g., using hammer, operating door lock). Each task requires precise control to achieve specific goals, such as moving objects to target locations or manipulating articulated objects to desired configurations. The MT50 setting specifically evaluates the ability to simultaneously learn and master all 50 tasks, making it a rigorous test for skill learning and composition.
For training, we collect 100 demonstrations per task using the scripted policies provided in the official Meta World codebase. These policies ensure consistent and optimal task execution, providing reliable basis for learning manipulation skills.
A.2. Real-World Setup
To validate the effectiveness of our approach in real-workd scenarios, we conducted experiments using the Cobot Agilex ALOHA robot, a dual-arm manipulator platform. For our experiments, we utilized a single arm to perform two challenging sequential manipulation tasks designed test the ability handle complex, multi-stage operations.Foo each task, collected 45 demonstrations through human teleoperation.
A.2.1. SEQUENTIAL OBJECT PLACEMENT TASK
The first task required the robot to "Pick the cube into the plate and pick the toy into the box." This task was designed to evaluate the ability to perform sequential pick-and-place operations involving multiple objects and target locations. Specifically, the robot needed to:
Successfully grasp and place cube onto a plate.
Transition to a second object (a toy) and place it into a box.
A.2.2. DRAWER MANIPULATION TASK
The second task involved more complex sequence: "Open the drawer, place the toy in the drawer, and then close it." This task was designed assess the capability to handle confined spaces and sequential manipulation steps. The robot needed to:
Open drawer.
Place toy inside the drawer.
Close drawer.
A.3. Evaluation Protocol
To ensure rigorous evaluation of our approach, we adopted comprehensive evaluation protocol across both simulated and real-world tasks.
1. SIMULATED BENCHMARKS
For the LIBERO and MetaWorld MT50 benchmarks, we evaluated performance using the Success Rate (SR) metric, calculated over 50 episodes per task with three random seeds. This protocol allowed assess the robustness and consistency of our approach across different task configurations.
A.3.2. REAL-WORLD TASKS
For the real-world tasks, we conducted 10 trials per task and reported both the overall success rate and the stage-wise wise completion rate. Specifically:
For the sequential object placement task, we tracked the successful completion of both the cube-to ta-betto--lte and toy-to-box placements.
13

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
For the drawer manipulation task, we measured success rates across three sequential stages: drawer opening, toy placement, and drawer closing.
A trial considered successful only if all stages of the task were completed in the correct sequence. This evaluation protocol allowed to identify potential bottlenecks in the manipulation sequence and assess the overall robustness of the learned policies.
A.4. Implementation Details
provide comprchensive implementation details of our framework, including the architecture configurations and training settings for both RaRSQ and CST modules, as well as the optimization process.
For the Rotation-augmenned Residual Skill Quantization (RaRSQ) module in our proposed STAR framework, we adopt a single-layer MIP as the encoder with a hidden dimension of 128. The decoder implemented as a transformer with 4 attention heads, decoder layers, and a hidden dimension of 128. We set the codebook size K = 16 and the quantization depth D 2, with each skill abstraction spanning timesteps. During training, the encoder and decoder are jointly optimized, while the codebook vectors are updated through the rotation-augmented gradient mechanism.
For the Causal Skill Transformer (CST), we utilize a ResNet-18 model trained from scratch as the visual encoder and pre-trained CLIP-base model as the language encoder. The proprioception encoder is implemented as single-layer ML. with a hidden dimension of 128. The transformer decoder consists of 6 layers, 6 attention heads, and an embedding dimension of 384. We set the start token dimension to 16, the beam size to 5, and the temperature to 1.0 for sampling. The observation window is fixed at 10 timesteps. During training, the visual and language encoders are frozen, and only the weights of the transformer decoder and the offset prediction head are updated.
We train the entire framework using the AdamW optimizer with a cosine decay learning schedule. For RaRSQ module, we use batch size of 1024, learning rate of 5.5e-5, and train for 100 epochs. For CST module, we use a batch size of 512, learning rate of 8e-4, and train for 500 epochs. Both modules use warmup step of 10 epochs and weight decay of 1e-6. The loss weights for the first codebook prediction, second codebook prediction, and offset head prediction are set to 2.0, 1.0, and 20.0, respectively. The models are implemented in Py PoTrhc and trained on a server with 8 Nvidia RTX L40S 48GB GPUs, with all models easily fitting on a single GPU.
A.5. Baseline Implementation
1. BASELINE DESCRIPTION
We systematically evaluate STAR against state-of-the-art methods three major categories:
Discrete LVM approaches: (1) VQ-BeT (Lee et al., 2024) combines VQ-VAE for discrete latent space learning with a transformer for latent code prediction; (2) QueST (Mete et al., 2024) leverages Finite-State Quantization for discrete latent space construction and employs a causal transformer for action sequence modeling.
End-to-end imitation learning: (1) ResNet-T (Liu et al., 2024) uses ResNet-18 with FiLM for observation and task instruction encoding, followed by transformer and GMM output layer for action prediction; (2) Diffusion Policy(Chi et al., 2023) implements a UNet-based architecture that maps Gaussian noise to action trajectories through a learned denoising diffusion process; (3) ACT (Zhao et al., 2023) proposes a wansformer-basee CVAE that generates temporally extended action sequences by decomposing behaviors into action chunks.
Large-scale VLA models: (1) Octo (Team et al., 2024) features large-scale transformer policy trained on 800K robotic demonstrations with siffussionbbsed action generation; (2) Open (Kim et al., 2024) presents a 7B 7B-parameter vision-language-action model that integrates Llama 2 with DINOv2 and SigLIP visual features.
A.5.2. BASELINE IMPLEMENTATION
Following prior works, we compare our method with several representative approaches in robotic manipulation. To ensure fair comparison, these baselines are implemented with consistent input modalities and encoders.
The architecture consists of transformer with 6 layers and hidden dimension of 256. The observation encoder incorporates ResNet-18 features combined with spatial attention mechanisms, maintaining a temporal context window of 10 timesteps for sequential prediction.
ResNet-T.
14

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmentedd Vector Quantization
ACT. The architecture incorporates 8 cross-attention and 4 self-attention layers, processing visual inputs through ResNet-18 backhone and language inputs via CLIP text encoder. The continuous action space is discretized into 256 bins per dimension, with model optimization performed using AdamW and cosine learning rate decay.
Diffussin Policy. The backbone follows U-Net design with channel dimensions [256, 512, 1024]. For the LIBERO benchmark, the prediction and execution horizons set to T = 32 and Ta 16 respectively, while the MetaWorld setup uses T = 16 and Ta 8. The policy maintains single-step observation history during execution.
VQ-BeT. The framework employs a single-layer MLP encoder (dimension 128) and residual vector quantization module with approximately 1024 codes. The sequence processing utilizes an observation window of 10 timesteps and action window size = 5, as longer sequences (T = 32) lead to information loss through excessive compression.
The model utilizes a transformer decoder with 6 layers and 6 attention heads (embedding dimension 384). The action generation process incorporates a vocabulary size of 1000 and equivalent start token dimension. At inference time, beam search is applied with width 5 and temperature 1.0, operating on action blocks of 8 timesteps.
Que
For all baselines, multi-modal observations are processed through concatenation and dimension-specific projections, with hyperparameters following their respective original implementations.
B. Extensive Ablation Studies
Method
LIBERO-Object
LIBERO-Spatial
LIBERO-Goal
LIBERO-Long
LIBERO-90
Overall
Ours w/o Action Refinement
98.3 +0.2
95.5 -0.6
95.0
10.7
88.5
20.3
90.8 +0.2
93.6 10.1
87.7 +1.7
76.8
228
54.4
10.5
37.6
#1.
38.4
10.3
59.0 10.8
Table Ablation Results. "w/o Action Refinement" represents removing the module action refinement. Best results are marked in bold.
B.1. Ablation Study on Action Refinement
To investigate the importance of action refinement in our framework, we conduct ablation experiments by removing the refinement module while keeping all other components unchanged. As shown in Table 5, removing action refinement leads to substantial performance degradation across all task suites.
The impact is particularly pronounced on more complex tasks that require precise manipulation. For LIBERO-Long tasks, removing action refinement causes a dramatic drop in performance from 88.5% to 37.6% (-50.9%). Similarly, for LIBERO-Goal tasks, the success rate decreases from 95.0% to 54.4% (-40.6%). This significant performance gap highlights that discrete skill codes alone, while effective for capturing high-level behavior patterss, are insufficient for achieving the precision required in complex manipulation tasks.
Even for relatively simpler tasks in LIBERO-Object and LIBERO-Spatial we observe notable performance decreases of 10.6% and 18.7% respectively. This suggests that action refinement plays a crucial role in bridging the gap between discrete skill abstractions and continuous control requirements, even for basic manipulation tasks.
The impact becomes more severe as task complexity increases, as evidenced by the larger performance drops in LIBERO-90 (-52.4%) and LIBERO-Long (-50.9%). This trend can be attributed to two factors:
1. Complex tasks often require more precise adjustments to handle varying object positions and environmental conditions, which cannot be fully captured by discrete skill codes alone.
2. Longer manipulation sequences accumulate small precision errors from discrete quantization, making the refinement mechanism increasingly important for maintaining task success over extended horizons.
Overall, the ablation results demonstrate that action refinement an essential component of our framework, contributing to a 34.6% improvement in average performance across all tasks. This validates our design choice of combining discrete skill abstractions with continuous refinement to enable both structured behavior representation and precise control execution.
15

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmennte Vector Quantization
C. Extensive Analysis
(a) Specialized Pattern
(b) Bimodal Pattern
(kg 3)
(k3 =
0.30
0.30
0.25
0.25
Probability
0.20
Probability
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
4
8
12
4
8
12
Second-level Code (kg)
Second-level Code (ks)
(c) Distributed Pattern
(k1 22)
0.150
0.125
Probabilty
0.100
0.075
0.050
0.025
0.000
8
12
Second-level Code (kx)
Figure 7. Visualization of representative skill dependency patterns. (a) Specialized pattern showing strong preference for specific second- level codes (k)=3). (b) Bimodal pattern indicating branching skill relationships (k)=7). (c) Distributed pattern more uniform dependencics (k)=2).
C.1. Evaluation of Skill Composition
We analyze the conditional probability distribution P(kglkı) between first and second codebook indices understand the hierarchical dependencies in our skill abstraction approach. Fig. 7 visualizes three representative patterns that demonstrate how our framework decomposes complex behaviors.
As shown in Fig. 7(a), when 4 = 3, we observe a highly concentrated distribution with a prominent peak at hı (P(k2 k = 3) 0.286), indicating this first-level skill code consistently pairs with specific second-level codes. Fig. 7(b) shows a bi-modal distribution for k 7, suggesting this primitive skill branches into two distinct paths. contrast, Fig. 7(c) illustrates more uniform distribution for h = 2, indicating more flexible skill combinations.
These diverse dependency patterns validate our hierarchical skill decomposition design, demonstrating that RaRSQ effectively captures different aspects of manipulation behaviors at multiple abstraction levels. This structured representation enables CST to generate coherent action sequences for accurate control.
C.2. Codebook Relationship
The relationship between consecutive codebook layers in our RaRSQ approach reveals important insights into how robot skills are hierarchically represented and composed. Fig. visualizes the conditional probability distribution P(k, kj) between indices from the first and second codebooks, demonstrating several key patterns that validate our architectural choices.
16

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
Skill Composition Matrix
0.117
0.184
0.020
0.003
0.152
0.086
0.038
0.038
0.018 0.061 0.101 0.094
0.037
0.009
0.002
0.040
0.055
0.132
0.051
0.003
0.064
0.107
0.015
0.105
0.032
0.074
0.061
0.123
0.011
0.032
0.047
0.088
0.079
0.045
0.024
0.002
0.286
0.057
0.041
0.085
0.040
0.014
0.037
0.030
0.070
0.091
0.061
0.047
0.071
0.114
0.135
0.017
0.000
0.071
0,005
0.012
0.032
0.010
0.142
0.188
0.051
0.018
0.033
0.000
0.171
0.20
0.051
0.157
0.070
0.006
0.155
0.040
0.038
0.024
0.071
0.120
0.019
0.055
0.015
0.042
0.004
0.131
0.037
First Codebook
0.024
0.202
0.030
0.003
0.075
0.101
0.068
0.056
0.008
0.207
0.034
0.114
0.014
0.010
0.004
0.049
0.15
0.051
0.044
0.004
0.054
0.060
0.011
0.062
0.087
0.005
0.206
0.024
0.167
0.063
0.033
0.091
0.038
0.060
0.118
0.087
0.014
0.023
0.016
0.042
0.085
0,014
0.097
0.023
0.225
0.033
0.007
0.109
0.047
0.075
-0.10
11
0.092
0.101
0.025
0.003
0.064
0,003
0.039
0.034
0.030
0.187
0.051
0.092
0.043
0.063
0.004
0.168
12
00.055
0.147
0.026
0.002
0.095
0.018
0.009
0.200
0.109
0.096
0.008
0.019
0.006
0.136
0.002
0.103
00.5
13
0..16
0.152
0.006
0.012
0.086
0.062
0.043
0.043
0.024
0.128
0.185
0.058
0.022
0.026
0,001
0.036
0.062
0.116
-0.016
0.033
0.129
0.001
0.164
0.084
0.019
0.097
0.205
0.082
0.015
0.060
0.005
0.033
0.005
0.051
0.00
0
1
2
3
4
5
10
11
12
13
14
15
Second Codebook
Figure Visualization conditional probability distribution P(k2(k)) between first-level (ki) and second-level (Aa) codebook indices.
The heatmap exhibits distinct non-uniform distributions across different first-level codes, indicating strong statistical dependencies between the codebook levels. For instance, when in observe particularly high probability of P(k2 3 k| = 3) 0.286, suggesting this second-level code frequently serves refinement for the third first-level primitive skill.
We observe several interesting patterns the codebook relationships:
1. Specialized Connections: Some first-level codes show strong preferences for specific second-level codes. For example,
1 7 exhibits high probabilities with 2/ and k2 9 (P(k2 1|kr 7) 0.202 and P(ke 9/k] = 7) 0.207), indicating these pairs may capture complementary aspects of certain manipulation skills.
2. Distributed Patterns: Other first-level codes (e.g., k = 2) demonstrate more uniform distributions across second-level codes, suggesting these represent more general behaviors requiring diverse refinements.
3. Balanced Utilization: The moderate maximum conditional probability (0.286) and the presence of both strong (dark red) and weak (light yellow) connections indicate the model learns meaningful skill hierarchies while avoiding over-specialization.
These patterns validate several key design choices in our framework:
The clear statistical dependencies between codebook levels support our use of autoregressive prediction in CST
17

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
The balanced utilization patterns demonstrate the effectiveness of RaRSQ in preventing codebook collapse
The hierarchical structure revealed by the conditional probabilities confirms effective decomposition of complex behaviors into primitive skills and their refinements
This analysis provides quantitative evidence for both the effectiveness of our hierarchical skill encoding and the importance of modeling dependencies between codebook levels for robust skill composition.
C.3. Skill Quantization Loss
To further validate the effectiveness of our rotation-augmentte residual skill quantization (RaRSQ) approach, we analyze the quantization loss across different LIBERO task suites. The quantization loss measures the LI distance between the encoder outputs and their quantized representations, serving as a direct indicator of how well the codebook captures the underlying skill space.
Fig. 9 shows the evolution of quantization loss during training for both standard VQ-VAE and our RaRSQ approach. Several key observations emerge from this comparison:
Initial Convergence: Both approaches start with similar loss values (around 0.001) across all task suites, indicating comparable initialization conditions. However, RaRSQ demonstrates consistently lower initial losses (e.g.. 0.001 16 vs 0.00121 for LIBERO-Object at step 100), suggesting better initialization the codebook structure.
Training Dynamics: Standard VQ-VAE exhibits a concerning pattern where the quantization loss increases substan- tially during training before partial recovery. For instance, in HIBERO-Spatiall the loss peaks at 0.0105 around step 5300 before settling at 0.0094. This pattern is indicative of codebook collapse, where the model struggles to maintain diverse skill representations.
Stability: In contrast, RaRSQ shows remarkably stable training dynamics. While it experiences minor initial increases in loss (e.g., peaking at 0.004 for AIBRO--Spatia around step 600), these increases are both smaller in magnitude and shorter in duration compared to standard VQ-VAE.
Final Performance: RaRSQ achieves significantly lower final quantization losses across all task suites. The improve- ments particularly pronouneed for complex tasks LIBERO- Long shows final loss of 0.00025 with RaRSQ compared to 0.0076 with standard VQ-VAE, representing 30x reduction in quantization error.
The loss patterns strongly correlate with our main experimental findings. The lower and more stable quantization losses of RaRSQ directly translate to improved task performance, particularly for complex manipulation sequences where precise skill representation is crucial. The brief initial increase in loss likely represents an exploration phase where the model discovers and refines its skill abstractions, while the subsequent rapid convergence low loss values indicates successful potation-augment gradient mechanism.
preservation of geometric relationships through
Notably, the reduction in quantization error is most significant for LIBERO-Long (97% reduction) and LIBERO-90 (92% reduction), precisely the task suites where our method shows the largest performance improvements. This suggests that the ability of RaRSQ to maintain low quantization error is particularly beneficial for complex, long- morizon tasks that require precise and diverse skill representations.
18

---

STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization
(a) Libero Object
0.012
Original VQ-VAE
0.010
VQ-VAE
0.008
LOSS
0.004
0.000
23
46
70
93
116
Steps (k)
(b) Libero Spatial
0.010
Loss
0.006
0.002
0.000
Steps (k) (c) Libero Goal
0.012
0.010
0.008
0.006
0.004
0.002
19
79
Steps (k)
(d) Libero Long
0.012
0.010
0.008
LOSS
0.004
0.000
52
104
156
208
260
Steps (k) (e) Libero
0.010
Loss
0.006
0.002
0.000
209
629
Steps (k)
Figure 9, skill quantization loss
19


Reviews:
[{"review_id": "mHDBIG8mCF", "reviewer": "Reviewer_G1Cu", "summary": "The paper proposes to improve prior latent discrete policies by preventing codebook collapse, improving the codebook utilization and proposes to use an autoregressive poiicy to chain together the various discrete skills. To achieve better codebook utilization (and preventing code collapse), the paper proposes to use rotation-augmented residual skill quantization. Essentially, this strategy introduces a hierarchical, coarse-to-fine latent vector codebook where the latents are encoded at different depths. Finally, a causal transformer predicts these latent vectors (again in a coarse to fine fashion) and decodes actions for the downstream policy. The proposed training strategy is tested on LIBERO and Meta-World, and shows superior performance than prior SOTA methods as well as ablative versions of the proposed method which do not use the rotation-augmented codebook strategy and the auto-regressive decoding process.\n\n## update after rebuttal\nI have read all reviewer's review and author rebuttal. I think the author responses make sense and address most concerns raised by other reviewers. Hence, I will vote for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer G1Cu for the thorough and positive assessment of our work. We appreciate your recognition of our key contributions in improving codebook utilization through rotation-augmented residual skill quantization and implementing autoregressive decoding for effective skill composition.\n\nIf you have any further questions, we would be more than happy to address them.", "claims_and_evidence": "Yes, in my reading, the claims are: a) that the proposed strategy leads to better use of codebooks (which is shown in section 4.3), and that leads to overall better performance than prior latent variable models (shown in Table-1) and the proposed changes help (shown in ablations of Table-2)", "methods_and_evaluation_criteria": "Yes -- the proposed method is well motivated and works on improving the codebook utilization, and the benchmark datasets are reasonable (i.e. they are standard) and make sense for the problem.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes -- the experiments are sound and show that the proposed method is better than SOTA and that the proposed changes help improve performance.", "relation_to_broader_scientific_literature": "The key contribution is to improve the performance of latent variable based policies by designing a strategy to improve codebook utilization and thus finally the downstream performance on manipulation tasks.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "In general, the paper is well-written and easy to follow.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cLIhLNA0fx", "reviewer": "Reviewer_Lv7s", "summary": "The paper investigates robot skill abstraction for manipulation tasks and introduces STAR—a framework for learning discrete robot skill representations. STAR comprises two main components: Rotation-Augmented Residual Skill Quantization (RaRSQ), which mitigates codebook collapse in VQ-VAE-based methods using rotation-based gradients, and a Causal Skill Transformer (CST) that explicitly models dependencies between hierarchical skill representations via an autoregressive mechanism. Experiments on the LIBERO and MetaWorld MT50 benchmarks, as well as real-world tasks, demonstrate that STAR outperforms several baselines in terms of success rate.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments and positive assessment of our work. Below we address the specific questions and concerns:\n## R1: A broader discussion on recent progress in robot learning—especially work on vision-language-action (VLA) transformers like π0—would provide valuable context regarding performance, efficiency, and scalability.\nIndeed, we have already compared our method with several advanced VLA methods in our manuscript, including OpenVLA (7B) and Octo, as shown in Table. 1. Further, we appreciate and follow your good advice to include π0 in our evaluation. The results on LIBERO benchmark are summarized below:\n| |**Size**|**Pretrain**|**LIBERO-Long**|**LIBERO-Spatial**|**LIBERO-Goal**|**LIBERO-Object**|**Avg**|\n|-|-|-|-|-|-|-|-|\n|π0|3.3B|True|85.2|96.8|95.8|98.8|94.15|\n|Ours|16.2M|False|88.5|95.5|95.0|98.3|94.30|\n\nDespite using significantly fewer parameters (200× smaller) and no pretraining, our approach achieves higher average performance (94.3% vs. 94.15%). Most notably, we observe the largest improvement on LIBERO-Long tasks (+3.3%), which aligns with our method's focus on addressing challenges in long-horizon tasks through effective skill composition.\n\nVLA methods typically follow a pretraining-finetuning paradigm, with performance dependent on pretraining data and architecture (e.g., π0 > OpenVLA). Our work shows lightweight models trained from scratch can match or exceed VLA methods requiring extensive pretraining. Future work could integrate our approach with large language models to further advance robot learning capabilities.\n\n## Q1: What is the unique contribution of STAR compared to prior works in VQ-VAE, rotation-based augmentation, and VLA transformers?\nWe would like to further clarify that our key contributions lie in designing robot-specific residual VQ-VAE that effectively decomposes complex robot behaviors into discrete skills, addressing fundamental challenges in robot skill learning.\n### (1) Robot-Specific Solution to Codebook Collapse in Skill Learning\nRather than a straightforward application of rotation tricks to VQ-VAE, RaRSQ is specifically designed for robotic skill space with hierarchical structure. While residual VQ-VAE offers larger representation space, this makes codebook collapse more problematic. Our integration of rotation-based gradients within the residual framework preserves geometric relationships throughout hierarchical action quantization, effectively capturing both coarse primitives (move, pick) and fine-grained adjustments required for complex manipulation.\n### (2) Explicit Modeling of Hierarchical Skill Dependencies\nOur CST differs fundamentally from previous VLA transformers by explicitly modeling conditional dependencies between different skill abstraction levels. Unlike approaches that predict actions directly or treat skills independently, CST captures structured relationships between coarse and fine-grained behaviors. As shown in Fig. 7, it reveals distinct skill dependency patterns that validate this approach. For example, given a first-level code, we found strong preference for specific second-level codes, showing how coarse skills constrain fine-grained behaviors in ways prior approaches cannot capture.\n\nAs shown in Table. 2, removing the autoregressive component significantly reduces performance, , especially in LIBERO-Goal (-6.9%) and LIBERO-Long (-5.2%). In addition, removing both components creates a 5.8% drop over all tasks, exceeding the sum of individual effects, demonstrating the synergistic relationship between skill representation and composition.\n## Q2: Can you confirm if the use of r_d in both eq(8) and eq(11) is correct, or if it is a typographical error?\nThanks for pointing out the typo. $r_d$ in eq(11) should be $\\hat{r_d}$, and we will correct it in the final revision.\n## Q3: Could you provide more details or video on the real-world robot experiments as well as completion time or speed?\nWe have recorded comprehensive videos showing the complete execution sequences for both tasks (drawer manipulation and sequential object placement). Since ICML rebuttal guidelines explicitly support anonymous links for supplementary figures and tables but don't specifically videos, complete videos will be included on our future project page.\n\nRegarding completion time, STAR demonstrates favorable completion times, averaging 29.6 seconds for the sequential object placement task and 37.7 seconds for the drawer manipulation task. This efficiency is attributable to our hierarchical skill abstraction approach, where a single inference step produces an action chunk (consisting of eight atomic actions in our implementation). This reduces the number of required inference steps and consequently decreases overall execution time. We will incorporate these timing metrics in the final version to provide a more comprehensive evaluation.", "claims_and_evidence": "The paper makes two key claims. First, the proposed RaRSQ method helps mitigate codebook collapse in VQ-VAE-based skill abstraction (quantization) by leveraging rotation-based residual skill abstraction. Second, the proposed causal skill transformer explicitly models dependencies between skill representations through an autoregressive mechanism, making it effective for complex, long-horizon manipulation tasks. To validate these claims, the authors conducted experiments on the LIBERO and MetaWorld MT50 benchmarks, as well as real-world experiments. Results of experiments demonstrate superior performance of the method and thus support the claim.", "methods_and_evaluation_criteria": "STAR leverages rotation-based gradients in its RaRSQ component and uses an autoregressive transformer to capture causal relationships between skills. The evaluation is conducted on established benchmarks (LIBERO and MetaWorld MT50) and validated with real-world robot manipulation tasks. The metrics focus on success rates, and the results indicate that STAR achieves higher performance than competing methods.", "theoretical_claims": "The paper does not present significant theoretical contributions.", "experimental_designs_or_analyses": "Experiments were conducted on the LIBERO and MetaWorld MT50 benchmarks, as well as on real-world tasks. The quantitative results demonstrate superior performance compared to several baselines, supporting the paper’s claims. Additionally, the authors performed ablation studies to assess the effectiveness of each variant of the proposed method, further reinforcing their findings.", "relation_to_broader_scientific_literature": "The work relates to VQ-VAE-based discrete representation learning, causal modeling, and robot learning.", "essential_references_not_discussed": "A broader discussion on recent progress in robot learning—especially work on vision-language-action (VLA) transformers like π0 [1]—would provide valuable context regarding performance, efficiency, and scalability.\n\n[1] Kevin Black, et al.,”π0: A Vision-Language-Action Flow Model for General Robot Control”, https://www.physicalintelligence.company/blog/pi0.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper addresses a critical and practical problem in robot learning: robot skill abstraction.\n2. The proposed method is reasonable and effective, utilizing RaRSQ to prevent codebook collapse and CST to model dependencies between skills.\n3. The thorough experiments demonstrate the algorithm’s effectiveness, and real-world experiments further support the paper’s claims.\n3. Overall, the paper is well-written and easy to follow.\n\nWeaknesses:\n1. My main concern is that this approach primarily builds on existing methods. For example, RaRSQ is essentially a combination of VQ-VAE and a rotation trick, while CST integrates a VLA Transformer with autoregressive prediction. This makes it difficult to pinpoint the paper’s unique contribution.", "other_comments_or_suggestions": "1. Both eq(8) and eq(11) denote r_d. Please clarify whether this is a typo.\n2. Including a video demo for the real-world manipulation tasks would be highly beneficial, as metrics beyond success rate—such as completion time—are also critical.", "questions_for_authors": "1. What is the unique contribution of STAR compared to prior works in VQ-VAE, rotation-based augmentation, and VLA transformers? It is important to highlight the differences. \n2. Can you confirm if the use of r_d in both eq(8) and eq(11) is correct, or if it is a typographical error?\n3. Could you provide more details or video on the real-world robot experiments as well as completion time or speed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TcKmHcz0XI", "reviewer": "Reviewer_6m5A", "summary": "This paper proposes STAR, a novel framework for learning diverse robot manipulation skills through skill quantization and causal modeling. STAR consists of two key components: RaRSQ, which enhances residual skill quantization with a rotation-based gradient mechanism to mitigate codebook collapse, and CST, a transformer-based model explicitly capturing temporal dependencies between discrete skills. The authors demonstrate that STAR significantly improves performance in multi-task imitation learning and complex, long-horizon manipulation tasks on standard benchmarks, achieving state-of-the-art results and highlighting its effectiveness in accurately composing complex action sequences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our extensive experiments and the practical effectiveness of our approach. We address your questions below:\n## Q1：Have the authors considered comparing their method against QueST equipped with an offset prediction to isolate the contributions of the proposed combination more clearly?\nWe appreciate and follow your good advice by adding additional experiments that compare STAR against QueST across various configurations:\n|Method|Finetune decoder|Offset|LIBERO-Object|LIBERO-Spatial|LIBERO-Goal|LIBERO-Long|Avg|\n|-|-|-|-|-|-|-|-|\n|QueST (Original)|✅|❌|90.0|84.5|76.7|69.1|80.1|\n|QueST(freeze decoder)|❌|❌|78.1|63|56|24|55.3|\n|QueST(fine-tune decoder + offset)|✅|✅|85.8|73.9|65.9|64.7|72.6|\n|Ours|❌|✅|**98.3**|**95.5**|**95.0**|**88.5**|**94.3**|\n\nThe results demonstrate that QueST with offset prediction (72.6%) significantly underperforms compared to our STAR method (94.3%)\n\nWe introduce offset head because our decoder is frozen during stage-1 training. This design means predicted actions can only reconstruct stage-0 actions, which undergo lossy compression through quantization, making fine-grained operations difficult. Therefore, we use an additional offset head to compensate for this gap. \n\nIn contrast, quest has already fine-tuned the decoder to learn fine-grained operations, which serves a similar purpose to our approach but through a different mechanism. Further adding an offset head does not yield significant gains and may instead conflict with the decoder's output.\n\nTo further verify the importance of learning fine-grained operations, we conducted an ablation study with QueST using a frozen decoder in stage-1. Performance dropped from 80.1% to 55.3%, consistent with our findings in Table 5 where removing the offset head significantly reduces our method's performance.\n\nIn summary, both approaches require mechanisms for learning fine-grained operations, but the superior performance of STAR comes from our novel contributions (RaRSQ and CST) rather than the offset head alone. The ablation studies in our original Table. 2 further support this conclusion by isolating the contributions of each proposed component.\n## Q2：It is currently unclear which component is primarily responsible for the performance gains. Could you further clarify to highlight the contribution of each component more clearly?\nOur framework consists of two components: (1)rotation-augmented residual skill quantization (RaRSQ), which addresses codebook collapse in robotic skill learning, and (2)causal skill transformer (CST) , which captures causal relationship between skills through autoregressive prediction. We would like to further clarify the contribution of each component:\n### **RaRSQ: Enhanced Skill Diversity and Representation**\nRaRSQ directly addresses the fundamental limitation of naive residual VQ-VAE - codebook collapse. As shown in Fig. 4, residual VQ-VAE utilizes only 43.8% codes, severely limiting the robot's ability to express diverse actions. In contrast, RaRSQ achieves 100% codebook utilization with balanced distribution.\n\nThis enhanced skill diversity translates to performance gains through precise skill decomposition. Complex manipulation tasks inherently require fine-grained action representation. With full codebook utilization, RaRSQ can distinguish between subtly different manipulation skills (e.g., picking different objects, precise positioning) that would otherwise be mapped to the same code in collapsed codebooks. This enhances the model's capability to recognize and execute task-specific manipulation patterns.\n### **CST's Contribution to Skill Composition**\nCST's autoregressive design comes from analyzing the learned hierarchical dependencies between skills. The conditional probability analysis in Fig. 8 reveals causal relationships between first and second-level skills -  some first-level codes show strong preferences for specific second-level codes. CST models the causal dependencies between skills, which is crucial for generating coherent action sequences. As show in Table. 2, when removing CST, we observe a significant performance drop, especially for LIBERO-Goal (-6.9%) and LIBERO-Long (-5.2%), indicating that modeling these dependencies is particularly important for complex manipulation tasks.\n## W1：While the integration of existing techniques is practically effective, the proposed method rely on adaptations and combinations of established ideas, which somewhat limits the theoretical novelty.\nOur key contributions lie in designing a robot-specific residual VQ-VAE that effectively decomposes complex robot behaviors into discrete skills, addressing fundamental challenges in robot skill learning. RaRSQ directly addresses codebook collapse in robotic residual skill quantization, while CST explicitly models the causal dependencies between different skill abstraction levels, revealing structured relationships between coarse and fine-grained behaviors.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical section is presented in the paper.", "experimental_designs_or_analyses": "I find the experimental designs reasonable and sufficiently aligned with the paper’s objectives.", "relation_to_broader_scientific_literature": "This paper builds upon recent advances in residual quantization methods and latent variable models (e.g., VQ-VAE, Rotation Trick) for robot skill learning, addressing common limitations such as codebook collapse and insufficient temporal skill modeling by introducing a novel combination of rotation-based quantization (RaRSQ) and autoregressive causal skill modeling (CST).", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper effectively integrates existing techniques (Residual Quantization, Rotation Trick, and causal transformer architectures) into a  practically effective framework.\n\nThe experimental evaluation is extensive, utilizing multiple benchmarks (LIBERO, MetaWorld).\n\nWeaknesses:\n\nWhile the integration of existing techniques is practically effective, the proposed methodologies heavily rely on adaptations and combinations of previously established ideas, which somewhat limits the theoretical novelty.\n\nSpecifically, the proposed approach significantly depends on well-established techniques (residual quantization and the rotational trick), and autoregressive skill prediction has already been extensively explored in QueST. My primary concern is that the notable performance improvement observed in this paper may arise more from offset prediction, as indicated in Table 5, rather than the novel combination itself.", "other_comments_or_suggestions": "To better highlight the strengths of your proposed approach, it would be beneficial to compare the performance explicitly against QueST with an additional offset prediction mechanism. This would clarify whether the observed performance improvements truly stem from the proposed combination or primarily from the offset predictor component.", "questions_for_authors": "Have the authors considered comparing their method explicitly against QueST equipped with an offset prediction mechanism to isolate the contributions of the proposed combination more clearly?\n\nIt is currently unclear which component of proposed method is primarily responsible for the observed performance gains. Could you further clarify to highlight the contribution of each component more clearly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel VQ-VAE for action sequences that builds a hierarchical coarse-to-fine latent vector codebook where the latents are encoded at different depths, by considering rotation-augmented residual skill quantization. The paper motivates the proposed method  as a way to mitigate mode collapse in vanilla VQ-VAEs. A causal transformer predicts the sequence of latents per skill (action sequence). They show in LIBERO improved results over alternatives. The paper addresses an important problem of discretizing action sequences, and proposes a working solution.  All reviewers recommend acceptance.
Decision: Accept (spotlight poster)