MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning
Authors: Chenxing Lin
Xinhui Gao
Haipeng Zhang
Xinran Li
Haitao Wang
Songzhu Mei
Chenglu Wen
Weiquan Liu
Siqi Shen
Cheng Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Reinforcement Learning; Auto-Regressive; Multi-Scale; Long-horizon
Primary area: reinforcement learning
TL;DR: A multi-scale auto-regressive generation method for offline RL

Abstract: Generative models have gained significant traction in offline reinforcement learning (RL) due to their ability to model complex trajectory distributions. However, existing generation-based approaches still struggle with long-horizon tasks characterized by sparse rewards. Some hierarchical generation methods have been developed to mitigate this issue by decomposing the original problem into shorter-horizon subproblems using one policy and generating detailed actions with another. While effective, these methods often overlook the multi-scale temporal structure inherent in trajectories, resulting in suboptimal performance. To overcome these limitations, we propose MAGE, a Multi-scale Autoregressive GEneration-based offline RL method. MAGE incorporates a condition-guided multi-scale autoencoder to learn hierarchical trajectory representations, along with a multi-scale transformer that autoregressively generates trajectory representations from coarse to fine temporal scales. MAGE effectively captures temporal dependencies of trajectories at multiple resolutions. Additionally, a condition-guided decoder is employed to exert precise control over short-term behaviors. Extensive experiments on five offline RL benchmarks against fifteen baseline algorithms show that MAGE successfully integrates multi-scale trajectory modeling with conditional guidance, generating coherent and controllable trajectories in long-horizon sparse-reward settings. The source code is available at https://github.com/xmu-rl-3dv/MAGE.

Published as a conference paper at ICLR 2026
MAGE: MULTI-SCALE AUTOREGRESSIVE GENERATION FOR OFFLINE REINFORCEMENT LEARNING
Chenxing Lin Xinhui G Haipen Zhang Xinran Li Haitao Wang Songzhu Mei Chenglu Wen Weiquan Liu Siqi Shen Cheng Wang "Fujann Key Laboratory Urban Intelligent Sensing and Computing, School of Informatics, Xiamen University (XMU), China
KKey Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China 'Hong Kong University of Science and Technology
"Scooo of Computer, National University of Defense Technology, China "College of Computer Engincering, Jimei University, China
(1incx1 123, xi xinhuigao zhanghaipeng hai taowang)) @stu. xmu edu. en, (siqishen, cwang, cl wen, wql i u) @xmu. edu, cn, s2 me i @nudt. edu. cn, xinran. 110 connect. ust. hk
ABSTRACT
Generative models have gained significant traction in offline reinforcement learning (RL) due their ability model complex trajectory distributions. However, exist- ing interaationbbsede approaches still struggle with long-horizon tasks characterized by sparse rewards. Some hierarchical generation methods have been developed to mitigate this issue by decomposing the original problem into shorter-horizon subproblems using one policy and generating detailed actions with another. While effective, these methods often overlook the multi-scale temporal structure inherent in trajectories, resulting in suboptimal performance. To overcome these limitations, we propose MAGE, Multi-scale Autoregressive GEneration-ban brasrd offline RI. method. MAGE incorporates a condition-guii guudeed multi-scale autoencoder to learn hierarchical trajectory representations, along with multi-scale transformer that autoregressively generates trajectory representations from coarse to fine temporal scales. MAGE effectively captures temporal dependencies of trajectories at mul- tiple resolutions. Additionally, a mandition-gudded decoder is employed to exert precise control over short-term behaviors. Extensive experiments five offline RL benchmarks against fifteen baseline algorithms show that MAGE successfully integrates multi-scale trajectory modeling with conditional guidance, generating coherent and controllable trajectories in long-horizon sparse-reward settings. The wttpg/hhwb.ooommm----//-/--
source code is available
INTRODUCTION
In offline reinforcement learning (RL) (Lange et al., 2012), agents trained solely from previously collected datasets without further interaction with environments, which makes it attractive for multiple real-world applications, such as robotics (Gupta et al., 2019; Deng et al., 2025; Shen et al., 2023) and clinical medicine (Lee et al., 2025). However, relying on fixed collected datasets to learn a policy faces several challenges, including distributional shift and overestimation bias (Levine al., 2020; Kumar et al., 2019).
Existing offline RL methods generally fall into four main categories: (1) Generation- based approaches, which view policy learning as conditional trajectory generation (Janner et al., 2021; Chen et al., 2021; Janner al., 2022: Wang et al., 2023: Ajay et al., 2023; Li et al., 2025); (2) Regularization- based approaches, which aim to prevent policy deviation by adding constraints relative to the behavior policy (Fujimoto et al., 2019; Kumar et al., 2019; Fujimoto & Gu, 2021); (3) Constraint- based approaches, which assign pessimistic values out-of-distribution actions to suppress their selection (Yang et al., 2021; Kumar et al., 2020); and (4) Model-based approaches, which utilize
"Equal contribution Corresponning author

---

Published as a conference paper at ICLR 2026
(a) Decision Transformer
(b) Decision Diffisser
(,,,, be (8p (4)))
Multiple Temporal scale
Dataset
(e) MAGE
Figure 1: Schematic Illustration of Generation Offline RL Methods. (a) Decision Transformer follows a step-by-step, autoregressive generation process.
(b) Decision Diffuser utilizes an all-at-once, denoising-bant based generation approach. (c) MAGE operates in top-down manner, first establishing a macroscopic outline of a trajectory and then progressively refining it with microscopic details.
a learned environment model for planning (Yu et al., 2021; Williams et al., 2016). Our proposed method falls under the generation-based paradigm.
Generation-based offline RL methods, such as Diffusion-QL. (Wang et al., 2023), Decision Dif- fuser (Ajay al., 2023) and DoF (Li al., 2025) have shown competitive performance and high trajectory diversity. benefiting from the strong representational capacity of generative models like diffusion processes (Ho al., 2020; Song et al., 2021) to capture multi-modal distributions. Despite these advantages, such methods struggle in long-horizon sparse-reward tasks, which are prevalent applications such as robotic manipulation and strategic planning. such settings, delayed feedback and complex temporal dependencies pose significant challenges for reliable policy learning (Tang et al., 2022; Villaflor et al., 2022; Andersen et al., 2018).
in
The inferior performance of generation-based offline RL in long-horizon tasks stems primarily from inadequate modeling of multi-scale temporal dependencies, particularly long-range information. While transformers (Chen et al., 2021; Janner et al., 2021) widely used, their unidirectional autoregressive nature limits bidirectional understanding of the global context. Diffusion models (Wang al., 2023; Ajay et al., 2023), although generally achieving stronger results, exhibit local generation bias (Lu et al., 2025), often producing trajectories that are locally plausible but lack global coherence over extended horizons.
promising direction is to use hierarchical generation methods (HGM), which convert long-horizon tasks into shorter-horizon subproblems. Existing approaches (Ma et al., 2024; Li et al., 2023; Chen et al., 2024) typically adopt fixed two-layer hierarchy, with each level governed by a distinct high level and detailed
policy. For example, HDMI (Li et al., 2023) generates sub-goals trajectories at low level. This rigid structure not only limits the ability to capture multi-scale temporal abstractions, but also introduces significant optimization challenges. The need to jointly optimize two interdependent policies within a fixed hierarchy could lead to training efficiency issues.
To address the challenges long-hore -orizon and sparse- reward tasks, we propose MAGE, novel Multi- scale Auto-regressive Generation model for offline RI. MAGE generates trajectories in top-down coarse-to-fie manner. produces a long-term, coarse-grain -raiined trajectory representation, and then this initial sketch is progressively refined through iterative rounds of auto-regressive generation, with each step yielding a finer-grained representation. Finally, actions are determined based on the resulting multi-scale trajectory representations. Figure I provides schematic overview of this generative process.
MAGE comprises two components: multi-scale autoencoder and multi-scale transformer. The autoencoder encodes a trajectory into a hierarchy of latent representations according to a predefined scale schedule, constituting of token maps from coarse-to-fine temporal resolutions. Coarse-
2

---

Published as a conference paper at ICLR 2026
scale tokens capture long-term dependencies, while fine-scale tokens encapsulate short-term details, The multi-scale transformer autoregressively generates these token maps sequentially, with each finer-scale token map conditioned coarser-scale token maps generated in the previous step. This coarse-to- generation scheme enables the model to capture both the global trajectory structure and local temporal dynamics, resulting in highly coherent trajectories. For finer-grained control, condition cnnditiin-guided -guided adapter module is integrated into the decoder, modulating internal representations based on specified conditions precisely steer the generated trajectories.
Extensive evaluations on five offline RL benchmarks show that MAGE achieves state-of-the-art performance, particularly in long-horizon tasks with sparse rewards, while remaining competitive in dense-reward settings. Systematic ablations confirm the critical role of multi-scale temporal modeling and conditional guidance. Additionally, MAGE exhibits fast inference speeds, providing an efficient and practical solution for complex sequential decision-making
BACKGROUND
2.1 AUTO-REGRESSIVE MODELS
As predominant autoregressive architecture, the Transformer (Vaswani et al., 2017) generates sequences by predicting each token x; solely based on its predecessors Tei. The probability of sequence under this model is defined by:
T p(11, T2,.... PT) = II p1,, .2, 1=1
The Visual Autoregressive (VAR) model (Tian et al., 2024) introduces a hierarchical approach to autoregressive data generation. Central to VAR is hierarchical autoregressive likelihood, which operates across multiple spatial scales rather than individual tokens. The joint probability of generating the complete set of token maps B (b1, b2..b...... ... bx)
is formulated as:
K
p(h, b2 .., bx)
II A=
p(ha (b,, b2, .... bk-1)
Here, each by denotes a token map of dimensions h& x wk, and the sequence ordered according increasing spatial resolution (i.e., hy-1 hy, Wk+1 > wx). The generation of is conditioned on all previously generated coarser-scale maps bęk-
2.2 VECTOR QUANTIZED VARIATIONAL AUTOENCODER
The Vector Quantized Variational Autoencoder (VQ-VAE) (Van Den Oord et al., 2017) extends the standard Variational Autoencoder (VAE) (Kingma Welling, 2014) by incorporating tokens (discrete latent representations) through vector quantization. The model comprises three key components: an encoder Eep that maps an input E to a continuous latent vector 2. F.r(x,, learnable codebook {ex km containing embedding vectors in R and a decoder Do that reconstructs the input from the quantized codes å = Do (zg)) The continuous output ze is discretized by replacing it with the nearest codebook entry:
Zq = Quantize = Ck, = min ||=e eyl12-
This quantization step enables the learning of tokens, which well suited for autoregressive modeling.
3 MAGE: MULTI-SCALE AUTO-REGRESSIVE DECISION MAKING
Generation- offline RL models have demonstrated competitive advantage in decision-make makiin tasks with complex trajectory distributions. However, they struggle with long-horizon tasks that have sparse rewards. Their lack of long-term awareness and multi-scale modeling for temporal abstraction
3

---

Published as a conference paper at ICLR 2026
often produces trajectories that are locally coherent but globally inconsistent, hindering effective decision-making
Our key observation is that, for effective long- horizon tasks, it is important generate trajectories multiple temporal scales, capturing both long-term and short-term information. We propose MAGE, a Multi-scale Autoregressive GEneration method for offiine RL. MAGE consists of two major modules: multi-scale trajectory autoencoder (Section 3.1) and multi-scale condition-guni guiddd auto-regressive generator (Section 3.2).
3.1 MULTI-SCALE TRAJECTORY AUTOENCODER
The Multi-scale Trajectory Autoencoder (MTAE) incorporates a multi-scale quantization architecture to capture hierarchical dependencies in long-horizon trajectories. represents trajectory as sequence of state and return-to-go (RTG) pairs: т (Ro, 80), (R1, ..)...... (Rr, 8T). where R; denotes the cumulative future reward from timestep onward, and 8, is the corresponding state. To enable autoregressive modeling, MTAE tokenizes the trajectory into discrete representations. This is achieved through a top-down encoding process that maps into a multi-scale token map M (m1. M2..-... mx)- Each token map my [V] is sequence of 4 tokens, where each token t E [V] is an integer from a vocabulary size V. The token map my encapsulates temporal information the k-th scale of the trajectory, with my capturing the coarsest, global-level structure and m containing the finest-grained details.
The encoding and decoding processes of MTAE depicted in Algorithm and 2. E(). 2(-). and D(-) denote the encoder, quantizer, and decoder, respectively. MTAE employs a similar architecture VQVAE (Van Den Oord al., 2017). Besides, shared codebook C is utilized across all scales to ensure that all tokens have the same size and the same vocabulary. The scale-up and scale-down operators are implemented through linear projection. For trajectory modeling, we empirically find that modeling (R, s) rather than other alternatives leads to the highest performance, as shown in Section 4.4.
Algorithm Multi-scale Encoding
Algorithm Multi-scale Decoding
Require: 7 {(S, Ro),..., (ST, Rr)); Require: temporal scales [ codebook C;
Require:
multi-scale token maps M;
Require: temporal scales [4x k codebook C:
1: ʃ = ET, Ro), .M
1: for l: ... K do
2: for k 1, K do
2: me = queue_popy M);
3:
my = Q(Scale down(f.44) (.)); M queue_ queue_pushh Baii__uushMMMMM m²);
Z Lookup(C, my);
4:
4:
= seale_up(zx: lx);
5:
ZF Lookup(C. mx);
end for
6:
24 Scale_uph 2k, 1k)
6: Z = (21..... 2x)
7:
f = f - zki
f D(Z, Ro);
8: end for
Ensure: reconstructed trajectory ř;
Ensure: multi-scale token maps M;
3.2 MULTI-SCALE CONDITIONAL GUIDE AUTOREGRESSIVE GENERATION
MAGE uses multi-scal temporal information as guidance to generate tokens. The token map my is generated based on all previous token maps m, i < k and (sp, Ro). After generating the token maps, an action a determined, which executed by the agent.
In MAGF, multi-scale conditional guide transformer is tasked with autoregressively predicting the sequence codebook maps (m,, m2...., mg). The generative process described as follows.
K
pmm,, m......m. my So, Ro) II k=1
p(m mmck, 80; Ro)-
At each scale k. the input to the transformer consists of sp. Rg. and the token maps from the previous scale m-ck- This hierarchical conditioning approach makes the predicted token map my close Ro
4

---

Published as a conference paper at ICLR 2026
Finetune Decoder with Adeeter
Deooii BI-A
Dendee
Deoiie Bloc
Decoder
Medligee Tier State
Invers Dymamik
L..
Rig
m0x
m.
Rv
Causal Transformer
Encoder
(a)
(b)
Figure 2: MAGE Overview: (a) Multi-scale Representation: hicrarchical quantization of trajectories across
scales for global-local structure modcling. (b) cundition-guides Decision-makin makiing: autoregressive latent prediction with conditional refinement for consistent trajectory generation.
and across different temporal scales. The transformer outputs categorical distribution over the token map at each scale. trained using a cross-entropy loss against the ground-truth token maps.
K Pv m log ma.k
LCE
(5)
where ma,i denotes the one-hot encoding of the ground-truth integer at position i and scale k and m,g is the predicted categorical distribution. The latent representation Z (=1.-.. ,Ex) for the generate trajectory i is obtained alongside (m1,..., ,mk) through look up in codebook C.
Determining Action MAGE adopts a latent inverse dynamics model / to determine the action a to be executed from the generated trajectory. Given the aggregated latent representation Z encoding the generated trajectory, determines the action as
2k); Lim=
aol12
ag is the action taken in timestep 0. The objective Lin is designed encourage the latent variable to preserve camamics-consistee information at the finest temporal scale for the most recent timestep. As evidenced by the ablation study in Appendix B.5, utilizing this latent representation Z leads to superior performance compared to the use of the fully generated trajectory.
Candition-Guider Refinement MAGE generates trajectories starting from the current state sq- However, we identified a challenge: the cross-entropy loss Leg alone does not guarantee that the first state of the generated trajectory + exactly matches sp, potentially leading trajectories that diverge from the intended condition. This issue is compounded by the information loss inherent in the quantization of latent variables 2. To correct for these deviations. MAGE incorporates an additional condition guidedd refinement loss, implemented as a mean squared error between the decoded initial state-return pair and the true initial condition (So, Ro).
(7) D' is an augmented decoder with a parameter-efficien refinement module. This decoder maps the latent codes Z back trajectory space, where D' (Z(Z, Ro)o denotes the generated initial condition ($p, Ro). To ensure this output strictly matches the true initial condition (sq, Rg). the conditional loss Loond is applied during training. This loss term guides the auto-regressive process to yield conditionally coherent trajectories. The necessity of Lcond is validated in Appendix Figure 4, where its removal leads to a deviation at the trajectory outset.
Lcond | (Z, Rolo (so, Ro)
4 EVALUATION
Comprehensive evaluations against 15 baselines across 5 benchmarks demonstrate MAGE's strong performance in long-horizon sparse-reward tasks, along with competitiveness in dense-reward settings.
5

---

Published as a conference paper at ICLR 2026
(a) Decision Trranffomee
(b) Decision Diffuser
(c) Hierarchical Diffuser
(d) Ours
Figure 3: A Maze Game: The dark grid cells walls. The trajectories plotted in red. Lighter color represents earlier timesteps. The circles shown in (c) are subgoals.
Ablations validate the necessity of multi-scale modeling and compare trajectory schemes, while results confirm high inference efficiency. Due to space limitations, please refer to Appendix for details.
4.1 EXPERIMENTAL SETUP
Environment We evaluate MAGE widely-used benchmarks covering long-horizon tasks-such as dexterous manipulation (Adroit (Rajeswaran et al., 2018)), sequential tasks (Franka Kitchen (Gupta al., 2019)), and navigation (Maze2D, Multi2D, AntMaze (Fu et al., 2020) as well as locomotion tasks with dense rewards (MuJoCo (Todorov et al., 2012)).
Baselines, Our method is compared with broad set of 15 representative baselines covering different families of offline RL approaches.
Non-generation methods: Behavior Cloning(BC) (Bain & Sammut, 1995), Conservative Q- Learning(CQL) (Kumar et al., 2020), and Implicit Q-learning(IQL) (Kostrikov et al., 2022) learn policies directly from offline data via behavior cloning or policy regularization, while Model Predictive Path IMtPg (Williams al., 2016) uses environment dynamics for model-based trajectory optimization.
Generation-based methods: Decision Transformer(DT) (Chen et al., 2021), Trajectory Trans- former(TT) (Janner al., 2021) and TAP (Zhang et al., 2023) generate trajectories using Trans- formers. Diffuser (Janner et al., 2022), Decision Diffuser(DD) (Ajay et al., 2023), and RGG (Lee al., 2023) generate trajectories using diffusion, while Diffusion-QL. (Wang et al., 2023) gener- ates actions through diffusion.
Hierarchical generation methods: ADT (Ma et al., 2024) two-level transformer-bast method. HDMI (Li et al., 2023) and HD (Chen et al., 2024) are two-level diffusion- based methods. CARP (Gong et al., 2025) is a coarse-to-fine autoregressive modeling method that generates action sequences.
To ensure fair and meaningful comparison, for each algorithm, we report its performance for each environment reported in the official paper. If such results are unavailable, we report the performance by running the algorithm for some environments.
For all experiments, the average score (with standard error) is utilized to measure the performance of all algorithms. For each environment, results averaged over 5 random training seeds. In Adroit and Kitchen, each seed tested 20 times, while in Maze2D, Multi2D, and AntMaze, each seed is tested 100 times due to the stochasticity of the environment.
4.2 AN ILLUSTRATIVE EXAMPLE
Figure 3 illustrates maze navigation task in which the agent must first collect the silver coin, then the gold coin, and finally reach the goal. The agent receives rewards for collecting coins and reaching the goal. The results indicate that Decision Transformer (DT) fails reach the goal, Decision Diffuser (DD) reaches the goal but fails to discover the gold coin, and Hierarchical Diffuser (HD) produces trajectories that cross walls. MAGE obtain all the coins and reach the goal. This example

---

Published as a conference paper at ICLR 2026
Table 1: The Average Scores for the Adroit Scenarios.
Scenario
IQL
ADT
CARP
DD
D-QL
HD
MAGE
Expert
128.0:9.2
116.3a1.2
113.3:12.1
112.7219.8
107.617.6
112.628.1
109.548.0
121.4114.3
147.8:4.9
Pea
Human
78.428.2
67.625.4
70.1=16.1
62.3+21.4
64.129.0
66.0s8.3
66.228.8
47.6+14.9
137.1
Cloned
83.428.1
64,4=14
35.9=13.1
12.5=15.2
47.749.2
49.3+8.0
48.3+8.9
13.949.7
108.4
Expert
106.620.3
104.810.3
105.140.1
98.444.7
87.0.0.8
93.740.8
85,910.9
105.940.6
106.8
Door
Human
3.2
4.420.8
7.5+2.3
5.024.6
6.9=1.2
8.0+1.2
7.111.1
0.240.0
16.520.9
Cloned
3.0=1.7
7.6+3.2
1.81133
0.0=0.0
9.0=1.6
10.6#1.7
9.321.6
4,5+3.6
20.5+2.5
Expert
128.620.3
117.4a6.6
127.4a0.4
127.5a0.6
106.7=1.8
114.8&1.7
1.7
1268&111
131,7
Hammer Human
1.7a0.8
1.240.1
1.8.0.2
0.920.3
1.040.1
1.310.1
1.240.1
0.9.0.3
10.4+1.2
Cloned 1.520.6
1.80.5
2.120.5
0.9a0.2
0.9g0.1
1.120.1
1.040.1
0,940.2
13.2:4..7
Expert
106.1+4.0
104.240.4
106.4#1.4
71.0=8.3
87.5+2.8
95.2+2.8
91.342.6
62.1
109.6
Relncate Human
0.1
0.1
0.1+0.1
0.0a0.0
0.240.1
0.240.1
0,140.1
0.0+0.0
0.3
Cloned
0.020.0
0.0a0.0
0.020.0
-0.240.0
-0.240.0
-0.240.0
-0.1
-0.240.0
0.020.0
Mean(w/o Expert)
21.4
18.4
14.9
10.2
16.2
17.0
16.6
8.5
38.3
Mean(all settings)
53.4
44.3
66.9
Table 2: The Average Scores for the Franka Kitchen Scenarios.
Scenario
IQL
DT
ADT
CARP
DD
DQL
HDMI
HD
MAGE
Partial
59.7=8.3
31.4=19.5
64.245.1
32.5+2.6
65 05.0+2.8
60.5+6.9
73.3+1.4
91.3+3.2
Kitchen
Mixed
53.2+1.6
25.8+5.0
69.243.3
30.0+2.2
57.0+2.5
62.6+5.1
69.241.8
71.742.5
86.343.3
Average
56.5
28.6
66.7
31.3
61.0
61.6
72.5
88.8
qualitatively demonstrates the ability of MAGE in such a long- hong-horizon task with sparse rewards. Please refer Appendix B.4 for further details.
4.3 COMPARISON STUDY
We evaluate MAGE against 15 offline RL algorithms on different sets of environments. The results demonstrate MAGE's compelling ability in long-horizon tasks with sparse rewards. Moreover, MAGE is also competitive in tasks with dense rewards.
4.3.1 ADROIT: DEXTEROUS MANIPULATION ENVIRONMENTS
The key difficulty of the Adroit environment (Rajeswaran et al., 2018) lies in its sparse reward signals and the requirement for long-horizon high-dimensional, fine-grained control. As shown in Table 1, MAGE achieves significant improvements on the Pen, Door, and Hammer tasks, with particularly strong performance on Pen, where substantially outperforms other methods. The results demonstrate that MAGE maintains consistent advantages in Adroit, despite the challenges of sparse rewards and high-dimensional control.
IQL shows limited performance, as it learns value functions, making it suffer from the deadly triad issues (Sutton& Barto, 2018). While DD improves performance through modeling trajectories, its single-step, non-holistic process lacks global perspective, leading to poor performance. CARP does not model the relationship between generated action sequences and rewards, leading to weaker performance than MAGE. Hierarchical RL methods, such as ADT and HD, do not fully model the multi-scale temporal information in trajectories, leading to suboptimal performance.
MAGE employs trajectory modeling with multi-scale temporal guidance and RTG for high- trajectory generation. MAGE achieves the best performance for such horizon nigh-dimensional. fine-grained control and sparse reward tasks.
7

---

Published as a conference paper at ICLR 2026
Table 3: The Average Scores for the Antmaze Scenarios.
Scenario
BC
CQL
IQL
ADT
DD
D-QL
MAGE
U-maze
47.244.0
37.243.7
70.6=3.7
51.750.4
83.013.1
49.243.1
66.248.6
94.024.9
95.242.2
Diverse Medium
0.840.8
67.243.5
61.7=6.1
0.0+0.0
83,4+19
4.0+2.8
78.6=10.3
88.748.1
98.2+1.3
Large
0.0+0.0
20.5
27.627.8
0.0+0.0
65.414.9
0.0+0.0
56 66.6
83.625.8
84.623.6
U-maze
55.244.1
92.7+1.9
83.3+4.5
$7.0+9.8
83.8+2.3
73.1+2.5
93.4+3.4
72.242.0
92.242.7
Play
Medium
0.0+0.0
65.7#11.6
64.624.9
0.0+0.0
82.0+1.7
8.024.3
16.6=10.8
42.0+1.9
92.0+2.7
Large
0.0+0.0
20.7+7.2
42.5=6.5
0.0+0.0
71.0
0.0+0.0
46.4+8.3
54.7+2.0
75.8+4.3
Average
17.2
50.7
58.4
18.1
78.1
22.4
69.6
72.5
89.7
Table 4: The Average Scores for the Maze2D and Multi2D Scenarios.
Scenario
CQL
IQL
DT
ADT
CARP
DD
HDMI
HD
MAGE
U-maze
8.916.1
42.1=0.5
31.0+21.3
60.5=2.0
26.243.9
116.242.7
120.1+25
128.413.6
145.4:3.2
Maze2D Medium
86.129.6
34.8=2.7
8.224.4
109.426.2
65.8+2.4
122.342.1
121.216
135.623.0
155.0:3.3
Large
23.7 23.7236.7
61,743.5
2,340.9
155.4:10.4
0.742.0
125.9a1.6
128.642.9
155.842.5
159.4
Single-task Average
33.6
46.2
13.8
108.4
30.9
121.5
123.5
139.9
153.3
U-maze
25.455.8
13.5+3.0
15.6=2.4
66.9+5.2
82,543.4
128.242.1
131,3+18
144.1
150.4=1.8
Multi2D>
Medium
8.343.9
8.343.4
6.3+1.6
48.742.6
129.742.7
131.6#1.9
140.2+1.6
147.743.1
Large
8.2+4.2
5.2=1.4
5,1=1.4
159.
32,2+3.2
130,5+4.2
135.4+2.5
165,5+0.5
166.8+3.6
Multi-task Average
14.0
9.0
9.0
111.6
54.5
129.5
132.8
149.9
155.0
4.3.2 FRANKA KITCHEN: COMPOSITIONAL ENVIRONMENTS
In the Franka Kitchen environments (Gupta et al., 2019), success depends not only on reaching individual sub-goals but also on executing them in the correct order, which makes naive trajectory generation prone to errors. By leveraging multi-scale trajectory generation, MAGE captures both the global task structure and local sub-goal details, providing coherent and fine-grained decision-making. MAGE demonstrates superior performance, surpassing all competing algorithms by considerable margin, as detailed in Table 2.
4.3.3 MAZE NAVIGATION ENVIRONMENTS, ANTMAZE, MAZE2I), AND MUL.TI2D
In the AntMaze, Maze2D, and Multi2D scenarios (Fu et al., 2020), a robot must navigate through maze-like structures to reach distant goal location. Mazes of different sizes (U-shaped, medium, and large) are evaluated. In the AntMaze tasks, the proposed MAGE outperforms the baselines on 5 out of 6 datasets, shown in Table 3. MAGE performs the best all datasets for the Maze2D and Multi2D scenarios, as shown in Table 4. The results demonstrate that our method can effectively handle the long-hon hong-horizon navigation tasks.
Beyond sparse-reward long-horizon tasks, MAGE also demonstrates strong performance in environ- ments with dense rewards, detailed in Appendix B.3 for Gym locomotion tasks. Notably, MAGE achieves top performance in 7 out of 9 tasks, confirming its general competitiveness across different reward structures.
4.4 ABLATION STUDY
To explore the effectiveness of design components, we conduct ablation studies regarding temporal scale, generation scheme, and conditional guidance on the Adroit scenarios: Pen-Expert and Door- Cloned. Moreover, evaluate the inference time of MAGE.
Importance of multiple temporal seales (K), We analyze the impact of varying the number of temporal scales K. Results in Table 5 show that performance generally improves as K increases up to 8, confirming that modeling multiple temporal scales beneficial. However, beyond this point (e.g., for Door-Clonede performance declines. This suggests that while incorporating finer- grainee information helps up to a certain point, excessive granularity (K 2 8) may introduce noise or unnecessary complexity without further gains. The optimal K is thus task-dependent.
8

---

Published as a conference paper at ICLR 2026
Table 5: Performance of varying the number of temporal scales K
Scenario
ADT
HD
2
8
10
Pen-Ex Pen-Expert
113.3+12.1
121.4+14.3
123,549.1
127.5+5.2
134.247.7
139.5+5.7
147.8+4,9
149.949.2
Door-Cloned
1.
4.5+3.6
5.2+1.8
6.0+2.1
10.7+2.3
14.0=2.7
20.5+2.5
17.0+2.7
Table 6: Different trajectory sequence generation schemes.
Scenario
ADT
HD
A
A+CQL
R,S,A A
Ours
Pen-Expert
113.3+12.1
121.4+14.3
127.743.3
127.1=13.0
127.6+4.6
124.9=7.8 17.2+3.0
147.8+4.9
Door-Cloned
1.8+1.3
4.5+3.6
11.9+2.7
9.1+2.3
4.9+1.0
20.5+2.5
Table 7: Ablation results of condition and constraint loss on the Adroit scenarios
Scenario
ADT
HD
Kin
A in mas1
A in Coand
Ours
Pen-Expert
113.3 12.1
121.4+14.3
140.3+9.1
139.5+6.7
139.947.2
147.844.9
Door-Cloned
1.8 1.3
4.523.6
12.3+1.9
16.3+2.6
17.142.5
20,5+2.5
Table 8: Average inference time (ms) Adroit environments.
Methed
Ours
DT
TT
ADT
DD
HD
Time (ms)
27.30=0.69
6.49=0.11
12863 19.33
7.81
1339.16=11.37
1480.21#25,18
Comparison of Trajectory Sequence Generation Schemes. We benehmark various sequence modeling strategies to identify the most effective scheme for trajectory generation. As shown in Table these include modeling states only (S), actions only (A), actions with CQL regularization (A+CQL), and the joint modeling of returns, states, and actions (R, S, A). Our approach, which models returns and states (R, S), achieves the best performance. This result suggests that the (R, S) scheme optimally balances the capture of high-level outcome intent (via returns) with detailed environmental dynamics (via states), whereas incorporating actions adds unnecessary complexity that hinders performance.
Role of RTG- based Conditioning. To quantify the importance of return-to-go (RTG) guidance, we systematically ablate its use in three key parts of MAGE: the autoencoder (K in D), the multi-scale transformer for finer scales ( in m and the conditioning loss Lcond ( Lzone). The results (Table indicate consistent performance drop when RTG is removed, underscoring critical role in aligning the generated trajectories with the desired return across temporal scales.
Inference Speed. As summarized in Table 8, MAGE achieves favorable balance between perfor- mance and efficiency. It runs approximately 50x faster than HD and 80x faster than DD. While slightly slower than some other Transformer-basse of 27 ms per step. This rate is well within the 20 Hz requirement for real-time robotic control (Reed al., 2022), demonstrating its practical applicability.
methods, MAGE maintains low inference time
RELATED WORK
5.1 GENERATION-BASED OFFLINE RL
Offline RL (Kostrikov et al., 2022; Kumar al., 2020; Fujimoto et al., 2018: Fujimoto & Gu, 2021; Fujimoto et al., 2019; Kumar et al., 2019) aims to learn policies from static datasets. A prominent branch of work generation- 2024; Zhang et al., 2023), which leverage generative models like Transformers (Vaswani al., 2017), flows (Kingma & Dhariwal, 2018), and diffusion models (Ho et al., 2020) to model the data distribution. Among these, diffusion-based approaches have been widely adopted.
methods (Chen al., 2021; Lee al., 2023; Ye & Gombolay,
Despite their powerful modeling capacity, diffusion-barts RL methods face notable challenges. Approaches like Diffusion--Ql (Wang et al., 2023) learn policy with Q regularization, while Diffuser (Janner et al., 2022), Decision Diffuser (Ajay et al., 2023), and RGG (Lee et al., 2023) generate trajectories for planning. However, they are plagued by local generation bias (Lu et al.,

---

Published as a conference paper at ICLR 2026
2025), which compromise global coherence, especially in long-horizon sparse-reward tasks.
Their iterative denoising also results in slow inference.
5.2 HIERARCHICAL RL
Recent hierarchical methods in offline RL are often inspired by human decision-making processes. Hierarchical offline RL (Ajay et al., 2021; Rao al., 2022) decomposes long-horizon tasks into man- ageable subproblems. which can be broadly categorized subgoal- based or skill-based (Hutsebaut- Buysse et al., 2022). Subgoal-base bagsellbased methods identify intermediate targets (Pateria et al., 2020), while skill-based approaches learn reusable low-level behaviors (Villecroze al., 2022). Although MAGE can be viewed as subgoal-based, it differs fundamentally by learning a single unified policy across all latent temporal hierarchies, rather than separate policies for each level.
Existing two-level hierarchical models include HDT (Correia & Alexandre, 2023) and ADT (Ma et al., 2024), which an autoregressive framework: a high-level policy generates subgoals or prompts, and low-level policy produces actions conditioned on them. Similarly, HDMI (Li et al., 2023) and HD (Chen et al., 2024) employ a diffusion- baasdd two-stage process, first generating subgoals under reward guidance and then producing Sobgoal-conditione trajectories. While effective, these methods often fail to capture the full spectrum of multi-scale temporal dependencies long-horizon tasks.
CARP (Gong et al., 2025) is multi-level method that generates action sequences based on current state. Due their high-frequency and non-smooth characteristics, action sequences (like joint torques) are considerably more difficult to predict (Ajay et al., 2023; Tedrake, 2009). Moreover, without explicit return conditioning, the approach cannot guarantee high returns. contrast, MAGE utilizes return-conditionas multi-scale auto-regressive process over states and RTG, ensuring high-pun gesh-performace outcomes. Please refer to Appendix B.7 for more in-depth discussion of MAGE and others.
DISCUSSION
MAGE provides coarse-to-fine framework for long-horizon trajectory modeling, achieving strong performance across diverse benchmarks. However, its hierarchical design involves natural trade-offs, committing to global plans at coarse scales can limit the flexibility of fine-scale refinements. In environments with extremely sparse rewards and long horizons (i.e., OGBench (Park et al., 2025)). We compared several strong algorithms on these difficult tasks and found that MAGE performs competitively (Table 24), although handling such extreme scenarios remains open challenge. Moreover, managing distribution shifts in quitt--diisribution scenarios remains critical challenge, and further research is needed to better handle such situations.
The multi-scale mechanism is extensible to multi-agent reinforcement learning research (Qin et al., 2024; Shen et al., 2022). The MAGE framework provides flexible, effective way for capturing intricate multi-agent coordination patterns (Qin et al., 2025).
7 CONCLUSION
We propose MAGE, multi-scale autoregressive generation method for offline reinforcement learning. consists of multi-scale condition-guie -uiid autoencoder and a multi-scale transformer. The transformer generates trajectories in multi-time-scale approach conditioning state. Extensive experiments on five offline RL benchmarks against fifteen approaches validate the effectiveness of MAGE. The results demonstrate that MAGE successfully integrates multi-scale trajectory modeling with conditional guidance, enabling the generation of coherent and controllable trajectories, and could effectively handle tasks with long horizons and sparse rewards.
return-to-goal and current
ACKNOWLEDGMENTS
This work was partially supported by the Fundamental Research Funds for the Central Universities (No. 20720230033), by Xiaomi Young Talents Program. We would like to thank the anonymous reviewers for their valuable suggestions.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Pseudocode and framework diagrams of our proposed method are provided in Appendix A, allowing readers to understand the algorithmic structure and workflow. All datasets used in experiments are publicly available from the D4RL (Fu et al., 2020) benchmark. Detailed hyperparameter settings for training and evaluation can be found in Appendix B. We have included the source code of MAGE in the supplementary.
ETHICS STATEMENT
This work focuses on developing and evaluating reinforcement learning methods in simulated environments. Our study does not involve human subjects, personally identifiable information, or sensitive data. The datasets and benchmarks used are publicly available and widely adopted in the reinforcement learning community. We believe that our research does not raise ethical concerns related to privacy, fairness, or potential misuse.
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: offline primitive discovery for accelerating offline reinforcement learning. In ICLR, 2021.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In ICLR, 2023.
Per-Arne Andersen, Morten Goodwin, and Ole-Christoffer Granmo. Deep rts: game environment for deep reinforcement learning real-time strategy games. CIG, 2018.
Michael Bain and Claude Sammut. framework for behavioural cloning. In Machine intelligence 15, 1995.
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In ICLR, 2024.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021.
André Correia and Luis A. Alexandre. Hierarchical decision transformer. In IROS, 2023.
Ziwei Deng, Mian Deng, Chenjing Liang, Zeming Gao, Chennan Ma, Chenxing Lin, Haipeng Zhang, Songzhu Mei, Cheng Wang, and Siqi Shen. Planu: Large language model decision making through planning under uncertainty. In NeurlPS, 2025.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-bias based reinforcement learning via w-weigtted variational policy optimization. In NeurlPS, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. In NewrIPS, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In JCML, 2018
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
Zhefei Gong, Pengxiang Ding, Shangke Lyu, Siteng Huang, Mingyang Sun, Wei Zhao, Zhaoxin Fan, and Donglin Wang. Carp: Visuomotor policy learning via coarse-to-fine autoregressive prediction. In ICCV, 2025.
11

---

Published as a conference paper at ICLR 2026
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving Long-horizon tasks via imitation and reinforcement learning. In CoRL, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML. PMLR, 2019.
Matthias Hutsebaut- Buysse, Kevin Mets, and Steven Latré. Hierarchical reinforcement learningg survey and open research challenges. Mach. Learn. Knowl. Extr. 4(1): 221, 2022.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In NeurlPS, 2021.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In ICML, 2022.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurlPS, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In ICLR, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learming via bootstrapping error reduction. In NeurlPS, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurIPS, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-arr. Springer, 2012.
Jung Min Lee, Shengpu Tang. Michael Sjoding, and Jenna Wiens. Optimizing loop diuretic treatment for mortality reduction in patients with acute dyspnea using a practical offline reinforcement learning pipeline for health care: Retrospective single-center simulation study. JMIR Med Inform, 13:e69145 145, Oct 2025. ISSN 2991-9694. doi: 10.2196/69145 URL https:/mmedinfommjmmn foriiiiioooommi.oogggg/ jmir.org/ 2025/1/c69145.
Kyowoon Lee, Scongun Kim, and Jaesik Choi. Refining diffusion planner for reliable behavior synthesis by automatic detection of infeasible plans. In NeurlPS, 2023.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement Icarning: Tutorial, review, and perspectives open problems. In arXiv, 2020.
Chao Li, Ziwei Deng, Chenxing Lin, Wenqi Chen, Yongquan Fu, Weiquan Liu, Chenglu Wen, Cheng Wang, and Siqi Shen. Dof: A diffusion factorization framework for offline multi-agent reinforcement learning. In ICLR, 2025.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. ICML. 2023.
Rui Lu, Runzhe Wang, Kaifeng Lyu, Xitai Jiang, Gao Huang, and Mengdi Wang. Towards under- standing text hallucination of diffusion models via local generation bias. In ICLR, 2025.
Yi Ma, Jianye Hao, Hebin Liang, and Chenjun Xiao. Rethinking decision transformer via hierarchical reinforcement learning. In ICML, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In ICIR, 2025.
12

---

Published as a conference paper at ICLR 2026
Shubham Pateria, Budhitama Subagdja, and Ah-Hwee Tan. Hierarchical reinforcement learning with integrated discovery of salient subgoals. In AAMAS, 2020.
Haoyuan Qin, Chennan Ma, Deng Deng, Zhengzhu Liu, Songzhu Mei, Xinwang Liu, Cheng Wang, and Siqi Shen. The dormant neuron phenomenon in multi-agent reinforcement learning value factorization. In NeurlPS, 2024.
Haoyuan Qin, Zhengzhu Liu, Chenxing Lin, Chennan Ma, Songzhu Mci, Siqi Shen, and Cheng Wang. Gradps: Resolving futile neurons in parameter sharing network for multi-agent reinforcement learning. In ICML, 2025.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems, 2018.
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, and Raia Hadsell. Learning transferable motor skills with hierarchical latent mixture policies. In ICLR, 2022.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. In TMLR, 2022.
Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. Resq: A residual function- approach for multi-agent reinforcement learning value factorization. In NeurIPS, 2022.
Siqi Shen, Chennan Ma, Chao Li, Weiquan Liu, Yongquan Fu, Songzhu Mei, Xinwang Liu, and Cheng Wang. Riskq: Risk-sensitive multi-agent reinforcement learning value factorization. In NeurlPS, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Richard Sutton and Andrew Barto. Reinforcement learning an introduction. MIT Press, 2018.
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi Doshi-Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. In NeurlPS, 2022.
Russ Tedrake. Underactuated roboties: Learning, planning, and control for efficient and agile machines course notes for mit 6.832. In Working draft edition, 2009.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurlPS, 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In IROS, 2012.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, pp. 5998-6008, 2017.
Adam Villaflor, Zhe Huang, Swapnil Pande, John M Dolan, and Jeff Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. In ICML, 2022.
Valentin Villecroze, Harry J. Braviner, Panteha Naderian, Chris J. Maddison, and Gabriel Loaiza- Ganem. Bayesian nonparametrics for offline skill discovery. In ICML, 2022.
Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusson policies as an expressive policy class for offline reinforcement learning. In ICLR, 2023.
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg. and Evangelos A Theodorou. Aggres- sive driving with model predictive path integral control. In ICRA, 2016.
13

---

Published as a conference paper at ICLR 2026
Yigin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang. Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offiie multi-agent reinforcement learning. In NeurIPS, 2021.
Sean Ye and Matthew Gombolay. Efficient trajectory forecasting and generation with conditional flow matching. In IROS, 2024.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBÓ: conservative offline model-based policy optimization. In NewrIPS, 2021.
Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefenstette, Yuandong Tian, et al. Efficient planning in compact latent action space. In ICLR, 2023.
14

---

Published as a conference paper at ICLR 2026
Appendix
A ALGORITHM
For completeness, we provide detailed algorithmic descriptions and the overall framework in this section. Together, these pseudocode listings provide a transparent view of both the quantization and latent prediction stages, complementing the high-level descriptions in the main text. They serve as a step-by-step reference for reproducing our method and clarifying the implementation details that underpin the proposed framework.
In addition, the framework illustration (Figure 2) offers an intuitive overview of how these components interact, highlighting the multi-scale representation and condition-guides generation process that form the core of MAGE.
A.1 BACKGROUND
A.1.1 VECTOR QUANTIZED VARIATIONAL. AUTOENCODER
The Vector Quantized Variational Autoencoder (VQ-VAE) (Van Den Oord et al., 2017) encodes an input into discrete tokens. It extends the standard Variational Autoencoder (VAE) (Kingma & Welling, 2014) by introducing discrete latent variables through vector quantization. It consists of an encoder Es that maps an input E X to a continuous latent 2e (x(x) E RD, learnable codebook with K embedding vectors {ex 1K k=1 where eg E DR and decoder Da that maps discrete codes by to a reconstruction f = Dz(zz) Each encoder output ze is replaced by its nearest neighbor in the codebook to obtain the discrete token Zq-
Zq Quantize(z)] = Ck. k arg
mig || - cyll2-
(A.1)
This enables the model to learn discrete representations suitable for autoregressive modeling.
The VQ-VAE is trained by jointly optimizing reconstruction quality and aligning encoder outputs with their assigned codebook vectors. The overall loss is
= || 212 + [|sg[=e] ex112 112 fl| sgle*]ll.
(A.2)
where sg[+] denotes the hhop-gradient operator and / controls the encoder's commitment to codebook entries. Gradients are copied from 2g to ze during backpropagation, enabling end-to-end learning despite the discrete bottleneck.
A.2 MAGE FRAMEWORK
The MAclcMult--scale Autoregressive GEneration) framework is designed to model trajectories in hierarchical and multi-scale manner, capturing both global structures and local dynamics.
As shown in Figure 2(a), trajectories are first quantized across multiple scales, where higher-level latents encode coarse, long-horizon patterns, while lower-level latents capture fine-grat graigeraddd short- horizon variations. This hierarchical representation enables the model to preserve coherent structures across temporal scales while effectively propagating high-level information to guide the generation of long-horizon trajectories.
Figure 2(b) illustrates the conditional trajectory generation process MAGE. At each scale, MAGE autoregressively predicts latent variables conditioned on past latents and return-to-go These pre- dictions are then refined and fine-tuned to better align the generated trajectories with the provided conditions. The refined latents are finally decoded into states, producing coherent trajectories across scales.
A.3 MULTI-SCALE QUANTIZATION
The first part presents the multi-scale quantization procedure of our VQ-VAE framework.
This pseudocode illustrates how the encoder maps a trajectory into hierarchical continuous embed- dings, how these embeddings quantized across scales using codebook lookups, and how the
15

---

Published as a conference paper at ICLR 2026
discrete hierarchy is aggregated into the final latent representation f for trajectory reconstruction. The pseudocode emphasizes the residual connections across scales, ensuring that finer levels progressively refine the coarser representations while mitigating information loss.
Algorithm 3 Multi-scale Encoding
Require: raw Trajectory T {(80,
(ST, RT)}; Number of scales K, scales
4 1 codebook C
f = {T, Ro), M = D. = []:
2: for k 1, .... K do
3:
mi = @(Scale ftctiiddoo[[____ (): M queue_pushe
4:
ma);
5:
Zk = Lookup(C, mR);
6:
ZA Scale_up up( Ex, lk):
7:
f f= f 2k
8: end for
Ensure: multi-scale token maps M;
Algorithm 4 Multi-scale Decoding
Require: multi-scale token maps M; Number of scales K, scales 4 :eurrent state sp; codebook
C
1: for k = 1, ...... K do
2:
soreuc_pop(Ale Zạ = Lookup(C, m,);
my
3:
4: 24 Scale_upe zk, k):
5: end for
6: Z ;
7: f D(Z, Ro);
Ensure: reconstructed trajectory
A.4 AUTOREGRESSIVE GENERATION PROCESS
The second part of the appendix provides the pscudocode for the Transformer- baasdd latent prediction process. In this procedure, the Transformer autoregressively predicts codebook indices conditioned on the initial state so and target return Ro, thereby capturing temporal dependencies in the discrete latent space.
The pseudocode explicitly demonstrates how predicted indices are embedded into tokens, how the model outputs categorical distributions over codebook entries, and how the training is performed with cross-entropy loss against the ground-truth indices.
It further outlines the inference phase, where the most probable indices are selected, the corresponding codebook vectors are retrieved, and the resulting multi-scale latent representation decoded to reconstruct the trajectory.
A.4.1 GUIDING MODELS WITH CONDITIONAL. CONSTRAINTS
While the cross-entropy loss enforces consistency between predicted and ground-truth -trth latent variables. it does not guarantee that the generated trajectory strictly matches the prescribed initial state and target RTG. As result, the generated rollouts may deviate from the desired conditional targets. Moreover, since the latent variables are discrete and quantized representations, information loss inevitable; even perfectly predicted latents can still lead to biased reconstructions. To address these limitations, introduce condition-guide refinement mechanism that enables end- to- optimization, ensuring that generated trajectories adhere both to the autoregressive latent dynamics and to the input conditions.
Concretely, we decode the autoregressively predicted multi-scale latents using the decoder D, with its parameters frozen preserve the trajectory prior it has leamed. However, fixed decoder cannot
16

---

Published as a conference paper at ICLR 2026
Algorithm 5 Training Multi-scale Transformer with Cross-Entropy and Conditional Fine-Tuning
Require: training trajectories {-): VQ- encoder for ground-trute truth tokens; codebook C; number of scales K; scales (4x )==)== hyperparameters Acond
1: initialize Transformer parameters and adapter parameters
2: for each mini-hatch of trajectories do
3: LCB 0
4: for each trajectory т in the mini-batch do
Kids obtain ground-truth multi-scale tokens maps = ( using the VQ-VAE
5:
encoder
6:
Mann < 0 for k = 1, for i = 1-...,,." do
7:
K do
8:
9:
prepare input tokens with 8p, Ro, Moott
10:
obtain logits 4,, f Transformerg (tokens)
11:
compute predicted categorical distribution my softmax
accumulate cross-entropy loss LCE += log rix [m]].]]
/1 Straight -thraugh through estimatoo
13:
Uk. *, arg max(vf.f)
soft +- HuumbelSoftmax((I.... Ty)
14:
15:
hard construct one-hot yk, with at index m,t soft
16:
apply STE: Vk,,i + skir stopgrad (v(.ft + yk, compute soft codcbook vector 2kea + V end for
18:
19:
scale up zı: < Scale_up(zx,44 (x)
append soft token Msot < queue push(Mattt ykk end for
21:
22:
7 < (21.... ,z)
23:
decode with dilatee--uggment decoder & t- D' Ro)
24:
compute condition loss Lcond [1(Ro, su) fu
25:
end for
26: total loss LCE Arond
27:
update 0 and á by descending gradient of C
28:
end for
Ensure: trained Transformer parameters 0 and adapter parameters 6
dynamically adapt to specific conditional inputs, limiting its ability in conditional generation. Inspired by parameter-efficient fine-tuning, we insert lightweight adapter (Houlsby al., 2019) modules between decoder layers. These adapters specialize in modulating internal representations according the conditional signals, thereby enhancing the sensitivity of the decoder to sy and the target RTG.
To enable gradient propagation through the inherently non con-differentiable codebook lookup, we adopt the Gumbel-Softmal relaxation. Instead of sampling hard indices directly, we draw differentiable approximations from the categorical distribution. In the forward pass, the straight- through estimator discretizes these samples via a argma for codebook indexing, while in the backward pass, gradients computed with respect to the continuous relaxation. This mechanism preserves the discrete struc- ture required for decoding while ensuring differentiability. thereby allowing end-to-end optimization under conditional constraints.
Under this end-to-end differentiable framework, the latents Z is decoded through the adapter- augmented decoder to produce the final trajectory f. To enforce strict conditional alignment, introduce a condition loss defined as the mean squared error between the decoded initial state-return pair and the target condition.
Loond = 1D' (so, Ro)l
(A.3)
Here, denotes the decoder equipped with adapters, and Z represents the latent representation retrieved from the codebook based the indices predicted by the model. This loss not only adapts
17

---

Published as a conference paper at ICLR 2026
the decoder to the current condition but also guides the latent prediction process, encouraging the model to compose optimal discrete tokens from the fixed codebook so that the decoded trajectory precisely satisfies the specified initial conditions.
B EXPERIMENT DETAILS
B. EXPERIMENTAL SETUP
We describe the baseline algorithms used in our experiments in detail, organized into three categories based on their approach.
von-generation-base Methods. This category includes approaches that do not rely on explicit generative modeling. BC learns policies by directly imitating expert demonstrations through supervised learning. while CQL regularizes Q-value estimation to prevent overestimation and im- prove stability in offline settings. IQL further decouples Q-function learning from policy updates, achieving better robustness by avoiding direct policy constraints. In contrast, MPPI (Williams al., 2016) is a model-based control method which uses learned known dynamics to sam- ple candidate trajectories and optimizes them with a cost function, representing a trajectory optimization approach rather than a pure policy learning method.
Generation-based methods. These methods reframe offline RL as conditional generative mod- eling task, learning to generate trajectories actions conditioned on states and task signals (e.g., desired returns or Q-values) instead of directly optimizing a policy, DT and TT employ Trans- former architectures model long- horizon dependencies. DT conditions desired returns and generates actions autoregressively, while TT focuses on trajectory- prediction by learning a sequence model over state-action pairs. Diffuser and Decision Diffuser adopt diffusion models to synthesize entire trajectories under task constraints; the latter incorporates classifier-fre guidance balance multiple conditional signals. Diffusion- QL differs by integrating Q-learning signals directly into the diffuusin process, generating high-value actions rather than full trajectories, thereby bridging generative modeling with value-based RL.
Hierarchical generation methods. Long-horizon tasks particularly challenging for flat generative models, motivating hierarchical designs. ADT introduces a hierarchical reinforcement learning framework, where a high-level policy generates prompts that guide a low-level Decision Transformer to produce actions, thus enhancing the ability to stitch suboptimal trajectories into coherent solutions. HDMI leverages graph- based planning to extract subgoals and incorporates them into diffusion-based trajectory generation. HD improves diffusion planning efficiency by using jumpy hicrarchical planning to expand the temporal horizon effectively. CARP uses a coarse-to-fine generation strategy for imitation learning, first producing coarse action chunks and then refining them into precise actions.
The work used for comparison is listed as shown in table 9,
B.1.1 COMPUTING RESOURCES
The experimental work was carried out on a performerfe computing cluster that includes several NVIDIA GeForce RTX 4090 GPUs to supply the required computing power. The cluster is also
https///ithub.com/B//////
lcarming
http:://giihbb.coo//////deioooon hanildddccssioonttaanssmmerr
hhtp://gghhbbbcmmmmmmmngggg///uu oanp:///ssso/tuuuuuuumeeeeesiooo Decision- Transformer
transformer
"butss//gtthu..ccm//ekwwwonngg
10
http::/ggihnn..cccchhhhhhhcccchp Hiaieeeeccerrrcal--Piiiei withte--ttttiiiiii Diffusion
18

---

Published as a conference paper at ICLR 2026
Table 9: Baseline algorithms
No.
Algorithm
Brief Description
Trains model by directly learning from examples provided by an expert, enabling the model mimic the expert's s behavior
BC (Bain & Sammut, 1995)
Updates Q-values conservatively to improve stability and sample efficiency
2
CQL. (Kumar et al., 2020)
Decouples policy updates from Q-value estimation improve the stubility and performance offline reinforcement learning
3
IQL. (Kostrikov et al., 2022)
Uses a Transformer architecture to model sequences of states, actions, and rewards, cnabling it to make decisions based on the entire history interactions and desired outcomes
4
DT (Chen et al., 2021)
Addresses the challenge of high-dimensional control by planning over temporally abstract latent actions, drastically roducing decision latency while improving performance
5
TAP* (Zhang et al., 2023)
Jointly optimizes high-level prompt and low-level action policies improve Decision Transformer's ability to stitch trajectories
6
ADT (Ma al., 2024)
Predicts future states and actions hy modeling sequences of past trajectories
7
TT (Janner al., 2021)
Uses a probabilistic approach to optimize control inputs by sampling future trajectories sclecting the hest one based a cost function
8
MPPI (Williams al., 2016)
9
Diffuser? (Janner et al., 2022)
Leverages diffusion models generate high-reward trajectories conditioned past experiences and guided by rewards
Improves diffusion--aase baasid planners by training a ggap predictor" that guides trajectory generation away from unreliable plans
10
RGG (Lee et al., 2023)
Incorporates classifier-free guidance dynamically fuse multiple
II
Decision Diffuser (Ajay et al., 2023)
conditions, enabling flexible and diverse policy generation
Combines behavior cloning with Q-learnin guidance during training to generate high-valuc actions through an itcrative denoising process
12
Diffusion " (Wang et al., 2023)
Extracts subgoals by using a graph-based planning method that constructs weighted graph from the dataset and finds optimal subgoal sequences through shortest path search
13
HDMI (Li et al., 2023)
14
HD (Chen et al., 2024)
Improves diffusion planning efficiency and gencralization in long-horizon tasks by using jumpy planning strategy
CARP12 (Gong et al., 2025)
Introduces coarse-to-fine generation method for the refinement of action chunks
equipped with Intel(R) Xeon(R) Gold 6348 CPUs, each operating frequency 2.60GHz. In order to verify the reliability of our findings, we executed our algorithm on five separate occasions for each experimental configuration, utilizing distinct random seeds each time.
B.1.2 ENVIRONMENT AND HYPERPARAMETERR
Our method is implemented based on the source code of DT (Chen et al., 2021), TAP (Zhang et al., 2023) and VAR (Tian et al., 2024).
The training hyperparameters for the Adroit environment in trajectory generation are shown in Table 10. The training hyperparameters for the Kitchen environment trajectory generation are shown in Table 11. The training hyperparameters for the Antmaze environment in trajectory generation are shown Table 12. The training hyperparameters for the Maze2D and Multi2D environment in trajectory generation are shown in Table 13.
The learning rate controls the step size during optimization, ensuring stable updates to model parameters. The horizon specifies the number of future steps considered in trajectory prediction, which is crucial in tasks requiring long-term reasoning. Dropout used to prevent overfitting and improve generalization, while the discount factor balances immediate and long-term rewards. Higher
19

---

Published as a conference paper at ICLR 2026
values encourage the agent to prioritize delayed outcomes, which are common in sparse reward scenarios. The codebook size defines the capacity of discrete latent representations for trajectory modeling, and the number of Transformer blocks, attention heads, and embedding dimensions together determine the expressive power of the sequence model. Batch size influences both training stability and computational efficiency, while the Adam optimizer is employed to adaptively adjust learning rates during training. All experiments use datasets provided by the D4RL benchmark suite (Fu al., 2020), which standardizes offline reinforcement learning evaluation and ensures comparability across methods.
Adroit The Adroit benchmark focuses on dexterous robotic hand manipulation and is widely regarded as one of the most challenging domains in offline reinforcement learning due to its high- dimensional continuous action space, contact-rich dynamics, and extremely sparse rewards. It contains four primary tasks that evaluate different aspects of fine-grained control. In the pen task, the robot hand must rotate and manipulate pen to match desired orientation, requiring precise finger coordination and continuous adjustment of forces. The door task involves opening a door by turning the handle and pulling it open, which demands both grasping and forceful motion under complex dynamics. The hammer task requires the agent to pick up hammer and strike a nail into a board, posing difficulties due to unstable contact interactions and the need for accurate motion sequencing. Finally, the relocate task challenges the agent to grasp a ball and place it at designated target location, which requires the integration of grasping, lifting, and accurate placement Together, these tasks test whether an algorithm can generate coherent long-horizon action sequences that achieve realistic manipulation skills.
dimensinnl space.
Each task is available with three types of datasets that reflect different data collection strategies. The expert dataset collected from demonstrations generated by near-optimal policy and represents high-quality trajectories that closely follow the desired behavior. The human dataset is collected from human teleoperation, resulting in diverse but suboptimal demonstrations that include natural variations and mistakes. The cloned dataset is generated by behavior cloning policy trained on human demonstrations, which often produces noisy and inconsistent trajectories due to compounding errors. These datasets pose varying levels of difficulty: while the expert data is relatively easier to learn from, the human and cloned datasets are much more challenging, as they require the algorithm to handle noisy, imperfect trajectories and extract meaningful learning signals from suboptimal behaviors, For the Adroit tasks, we adopt the hyperparameter configurations summarized in Table 10.
Table 10: Hyperparameter Settings for Adroit environment
Value
learning rate
2e-4
horizon
24
dropout rate
0.1
discount
0.99
codebook size
512
transformer blocks attention head embed dim batch size optimizer
8
4
512
512
Adam optimizer
Franka Kitchen The Franka Kitchen environment is multi-task, high-dimensional manipulation benchmark designed to evaluate planning and generalization in realistic, non-navigation settings. It involves controlling 9-DoF Franka robot to interact with several common household items, including a microwave, a kettle, an overhead light, cabinets, and an oven. Each task requires the agent to manipulate these objects to reach specific goal configuration, often involving multiple
20

---

Published as a conference paper at ICLR 2026
sub-goals executed in a partieular sequence. For example, goal state may require opening the microwave and a sliding cabinet door, placing the kettle on the top burner, and turning on the overhead light. The main challenge of this environment lies in its long- monizon,, sequential, and combinatorial nature. Agents must plan over multiple sub-goals while respecting their dependencies, and trajectories collected in this domain often contain complex, non-trivial paths through the state space. Success therefore depends on effective generalization to unseen states, rather than simply reproducing training trajectories.
D4RL provides three types of datasets to study this environment. The complete dataset consists of trajectories in which the robot performs all tasks order, offering demonstrations that are relatively easy for imitation learning methods. The partial dataset contains undirected subtasks, but subset of trajectories can still solve the tasks, allowing agents to succeed by selectively choosing relevant segments. Finally, the mixed dataset contains only undirected subtasks with no complete solutions, requiring agents to stitch together relevant sub-trajectories and generalize the most in order to achieve successful task execution. These datasets collectively benchmark an algorithm's 's ability to handle multi-task manipulation, sequential planning, and generalization in a realistic robotic environment.
Table 11: Hyperparameter Settings for Franka Kitchen environment
parer-p
Value
learning rate
2e-4
horizon
24
dropout rate
0.1
discount
0.99
codebook size
2048
transformer blocks attention head embed dim batch size optimizer
8
4
512
512
Adam optimizer
Antmaze The AntMaze benchmark is one of the most challenging tasks in offline reinforcement learning, designed to evaluate long-horizon planning and effective credit assignment under sparse reward settings. In this environment, quadrupedal ant robot must navigate through maze-like structures to reach a distant goal location. The task is difficult due to the combination of a high- dimensional continuous action space, complex dynamics, and extremely sparse feedback, where the agent is rewarded only after successfully reaching the goal. Mazes of different sizes (U-shaped, medium, and large) also vary in complexity, with larger mazes requiring more complex planning and longer-tene vision. Therefore, AntMaze is a standard platform for testing algorithms' ability to perform global reasoning and generate coherent sequences of actions with long horizons. For this environment, we adopt the hyperparameter settings summarized in Table 12.
Maze2D & Multi2D The Maze2D benchmark provides a series of 2D navigation tasks where a point-mass agent must traverse complex maze layouts to reach a goal position. Although the dynamics are simple, the challenge lies in long-term planning under sparse rewards, as the agent only receives a positive signal when the goal is reached. Maze2D consists of several difficulty levels, such as U-maze, medium, and large mazes, with increasing structural complexity that requires the agent to discover feasible paths across longer horizons. In this setting, the start position of the agent is fixed while the goal position randomized across episodes, which prevents overfitting to a single target location. The Multi2D variant further increases the difficulty by randomizing both the start and goal positions in each episode, forcing the agent to generalize over a much broader distribution of navigation tasks. Together, Maze2D and Multi2D serve canonical benchmarks for evaluating the ability of algorithms to plan effectively under sparse feedback and diverse conditions. For these environments, adopt the hyperparameter settings reported in Table 13.
21

---

Published as a conference paper at ICLR 2026
Table 12: Hyperparameter Settings for Antmaze environment
Hyper-paramesn
Value
learning rate
2c-4
horizon
dropout rate
0.1
discount
0.998
codebook size
1024
transformer blocks attention head
embed dim
512
batch size
512
optimizer
Adam optimizer
Table 13: Hyperparameter Settings for Maze2D and Multi2D environment
parper-arrmmter
Value
learning rate
2e-4
horizon
dropout rate
0.1
discount
0.99
codebook size
256
transformer blocks attention head embed dim batch size optimizer
256
512
Adam optimizer
Gym locomotion control The Gym locomotion control benchmark comprises a set of continuous control tasks including HalfCheetah, Walker2d, and Ant, where agents with articulated bodies must learn to move forward efficiently. These tasks are built simplified physics engines that capture essential dynamics of legged locomotion. each timestep, the agent observes vector of physical variables such as joint angles, joint velocities, and body orientation, and outputs continuous torque commands to control its joints. The reward functions are typically dense, combining terms for forward velocity, stability, and control cost, which guide the agent toward producing smooth and sustainable gaits. Each environment poses distinct locomotion challenges: Ha fCheet ah focuses generating rapid forward motion in planar setting, Malker2d requires maintaining upright balance while walking bipedally, and Ant: involves coordinating multiple legs to achieve stable quadrupedal movement. Collectively, these benchmarks evaluate algorithm's ability to learn coordinated control policies under continuous dynamics and varying morphological structures. For these environments, we adopt the hyperparameter settings reported Table 14.
B.2 ADDITIONAL RESULTS FOR COMPARISON STUDY
In this section, provide additional experimental results that were not included in the main text due to space limitations. These results cover a broader set of baseline methods, including flat reinforcement learning algorithms well as several hierarchical reinforcement learning approaches.
22

---

Published as a conference paper at ICLR 2026
Table 14: Hyperparameter Settings for Gym locomotion control tasks
Value
learning rate
2c-4
horizon
24
dropout rate
0.1
discount
0.99
codebook size
512
transformer blocks attention head embed dim
512
batch size optimizer
512
Adam optimizer
Table 15: Additional baseline comparisons Adroit scenarios. Results are averaged over random training seeds, with each seed tested 20 times. Bold numbers indicate the best performance.
Scenario
BC
CQL
TT
TAP
MAGE
Expert
94.6*3.2
-1.4+2.3
101.8+13.8
127.4a7.7
147.8+4,9
Pen
Human
71.0s6.2
13.7
2.0=3.4
76.5=8.5
137.1
Cloned
51.9=15.1
1.0#6.6
38.8+13.3
57.4+8.7
108.4+17.6
Expert
105.1+2.4
-0.3+0.0
101.624.8
104.8=0.8
106.820.1
Door
Human
2.6+5.7
5.5+1.3
0.1=0.0
8.8=1.1
16.5#0.9
Cluned
-0.1+0.0
-0,3+0.0
0.0=0.0
11,7=1.5
20,5+2.5
Expert
126.7+3.8
0,2+0.0
1.1
127.6
131.7=0.2
Hammer
Human
1.2+2.7
0.1+0.1
1.4+0.1
1,420.1
10.4+1.2
Cloned
0.6+0.1
0.3+0.0
0.420.0
1.240.1
13.2+4.7
Expert
107.7+5.8
-0.300.0
8.5+3.1
105.8
109.621.6
Recolate
Human
0.0+0.0
0.0=0.0
0.1
0,2+0.1
0.3+0.1
Cloned
00.20.2
-0.3+000
-0.2+0.0
-0.2+0.0
0.0+0.0
Average(w/o expert)
15.9
2.5
5.3
19.6
38.3
Average(all settings)
46.8
1.5
21.3
51.9
66.9
The purpose of this comparison is to offer a more comprehensive view of the performance landscape, complementing the results reported in the main paper.
Adroit We additionally evaluate several representative baselines on the Adroit benchmarks. BC learns policy by supervised imitation of expert demonstrations. CQL regularizes Q-laarnnn avoid overestimation of out-of-distributtin actions. TT models trajectories as autoregressive sequences with transformer. TAP (Zhang et al., 2023) leverages a discrete latent action space learned with VQ-VAE to enable efficient planning in continuous control tasks. As shown in Table 15, all these baselines are clearly outperformed by our method across different scenarios.
Franka Kitchen We additionally evaluate several representative baselines on the Franka Kitchen benchmarks. BC learns a policy by supervised imitation of expert demonstrations, CQL regularizes Q-learning to avoid overestimation of denoising diffusion processes to generate state and action sequences. As shown in Table 16, all these baselines are clearly outperformed by our method across different scenarios.
actions. Diffuser models trajectories as
Maze2D & Multi2D We further include comparisons on the Maze2D and Multi2D benchmarks with several additional baselines. MPPI (Williams et al., 2016) is sampling-basee model predictive
23

---

Published as a conference paper at ICLR 2026
Table 16: Additional baseline comparisons for the Franka Kitchen Scenarios. Results averaged random training seeds, with each seed lested 20 times. Bold numbers show the best performance.
Scenario
BC
CQL
Diffuser
MAGE
Partial
41.3+3.7
$1,3+7.7
52.5+2.5
91.3dt3.2
Kitchen
Mixed
48.940.7
51,347.7
55.7
86.343.3
Average
45.1
51.3
54.1
88.8
Table Additional baseline comparisons for the Maze2D and Multi2D scenarios, Results are averaged over random training seeds, with each seed tested 100 times. Bold numbers indicate the best performance.
Scenario
MPPI
Diffuser
RGG
MAGE
U-maze
33.2
113.9=3.1
108.8#1.4
145.4+3.2
Maze2D
Medium
10.2
121.5+2.7
131.840.5
155.0+3.3
Large
5.1
123.026.4
135.4#1.7
159,4+2,9
Single-task Average
16.2
119.5
125.3
153.3
U-maze
41.2
128.9 1.8
128.3+0.8
150.4=1.8
Multi2D
Medium
15.4
127
127.243.4
130.0#0.9
147.7#3.1
Large
8.0
132.
148.3 1.4
166.8#3.6
Multi-task Average
21.5
129.4
135.5
155.0
Table 18: Baseline comparisons across different Gym locomotion control tasks. Bold numbers indicate the best performance.
Task
Dataset
IQL
DT
CARP
TAP
Diffuser
HD
MAGE
Medium Medium--Expert
55.2
91.6
86.7
86.8
95.0..02
57.112.7
91.810.8
88.910.3
92.510.3
95.2.10.2
HallCheetah
Medium
38.2114
45.010.1
42.810.3
46.7100.
43.9.1..
Md-Replay
36.6
45.5
44.2
36.6
41.9-2.5
34.6+0.66
40.8+:0.6
37,7+0.5
38, ++0.7
46.0--0.2
Medium-Fxpert
101.9-63
102.242..8
107.4+0.99
107.1+0.1
110.3-0.1
Walker2d
Medium
75.3
83.0
78.3
74.0
79.0-2.8
60.71118
64.922.1
79.6:10.6
84.0100.6
83.520.5
Md-Replay
32.3
77.2
42.71:111
66.813311
70.61::.6
87.8=2.0
Medium Expert
114.2
115.8
125.6
122.3
116. 1-
107.6:111.8
128.81:2.4 2.4
101 8:1 17.0
109.2::11.7
135.1 1.9
Ant
Medium
92.1
83.1
10.2
79.3
Md-Roplay
89.2
93.9
88.8
88.7
77.0=6.8
80.2+1.2
96.71:1.4
88.12662
83.2+:1.
99.3=20.9
Average
77.3
89.8
control method. Diffuser leverages diffusion probabilistic models generate trajectories for planning. RGG (Lee et al., 2023) enhances diffusion planners by introducing the recovery gap metric to detect and mitigate infeasible plans, improving both reliability and interpretability. As shown in Table 17, our method consistently achieves the best performance across both single-task and multi-tase balti-task settings, outperforming all baselines.
B.3 RESULTS FOR GYM LOCOMOTION CONTROL TASKS
We evaluate our method on standard Gym locomotion control environments, including HalfCheetah, Walker2d, and Ant. These environments involve controlling simulated agents with continuous action spaces to achieve stable and efficient locomotion. While our approach focuses on long-horizon and sparse-reward settings, it also demonstrates competitive performance on short-horizon and dense- reward tasks compared to other baselines. This indicates that method is capable of generating coherent trajectories, maintaining fine- iie-igrined control, and effectively aligning actions with target returns across wide range of locomotion scenarios.
24

---

Published as a conference paper at ICLR 2026
(a) MAGE w/o foodd
(b) MAGE
Figure 4: Effect of Conditional Constraints on MAGE: without Loved, the agent can still locate coins and reach the goal, but the agent generates the wrong trajectory that crosses the wall.
Table 19: Comparison between explicit and latent inverse dynamics models.
Scenario
Explicit
Latent (Ours)
Pen-Expert
136.546.7
147. 447.844.9
Door-Cloned
0.420.3
20.542.5
Table 20: Effect codebook size on performance.
Scenario
128
256
512
1024
Pen-Expert
112.9=19.0
146.5+3.1
147.8+4.9
145.3+1.9
Door-Cloned
14.4+2.9
19.743.4
20.5+2.5
11.342.1
B.4 RESULTS FOR THE MAZE GAME IN FIGURE
To investigate the model's ability to capture multi-scale temporal dynamics, we designed a simple Maze game. In the game, the agent only succeed if it starts from the initial position, collects the silver coin and the gold coin in sequence, and finally reaches the goal. In the dataset, the start and goal positions randomized, while the silver and gold coin locations remain fixed but not explicitly revealed by the environment. Therefore, the agent must rely on its understanding of long-horizon spatial information identify the coin positions and navigate toward them. We study the performance Decision Transformer, Decision Diffuser, Hierarchical Diffuser, and our method in this setting, with the results shown in Figure of the main text. The results show that MAGE can finish such that long-horizon task while others cannot.
In Figure 4, analyze the impact of conditional constraints Lcond MAGE. Even after removing the conditional constraint Leand. MAGE still shows an understanding of the Long-horizon structure, locating the silver and gold coins and reaching the goal. However, the resulting trajectories often become distorted, sometimes walking through walls. This demonstrates that our multi-scale trajectory modeling approach can effectively capture the long-horizon temporal dynamic, while the conditional constraints Lcond ensure fine-gan grain control and guide the model to generate more condition-compliant trajectories.
B.5 ABLATION STUDY RESULTS
Explicit or Latent Inverse Dynamics Model. As illustrated in Table 19, we perform an ablation study on where to incorporate the inverse dynamics model. Explicit denotes decoding the trajectory first and then applying the inverse dynamics model to recover actions, and Latent directly decodes actions from the latent trajectory. Our approach consistently outperforms the explicit inverse model, indicating that modeling inverse dynamics the latent space can more effectively capture trajectory relationships and yielding more accurate action predictions for improved planning performance.
25

---

Published as a conference paper at ICLR 2026
Table 21: Effect of Transformer depth performance.
Scenario
2
4
8
Pen-Expert
109.
124.1=17.1
147.8+4.9
145.6#3.4
Door-Cloned
12.4+2.2
14.8+2.4
20.5+2.5
18.4+2.5
Table 22: Effect of the adapter module for applying Leond-
Scenario
MAGE (direct decoder)
MAGE
Pen-Expert Door-Cloned
132.418.3
147.814.99
12.0+11.7
20.5
Evaluating Codebook size and Transformer layers. In this experiment, examine two hyperpa- rameters our method, the size of the codebook and the number of Transformer layers. As shown in Table 20, increasing the codebook size improves performance by enriching the discrete representation, but when the codebook becomes too large the performance declines due to overfitting and excessive partitioning of the state space. Table 21 illustrates that adding more Transformer layers initially enhances performance. However, once the network reaches sufficient depth (2 8), the performance declines, indicating that further scaling provides only limited benefit.
Role of the Adapter for Conditional Guidance. To evaluate whether Lcond can be applied without the adapter, we introduce a variant named MAGE (direct decoder) in Table 22, where the conditional loss is added directly to the main decoder. This variant results in noticeable performance drop, indicating that the decoder's reconstruction behavior is adversely affected. The conditional objective interferes with the learned decoding distribution, leading to degraded trajectory consistency. In contrast, the adapter cleanly separates Loond from the decoder, ensuring stable optimization and stronger overall performance.
Effect of RTG-Based Reweighting, Inspired by wrrattagoweighhee formulations such as QVPO (Ding al., 2024), we further examine whether explicit reweighting ternative to RTG conditioning. In our variant, each trajectory RTG is first normalized to the interval [0, to obtain weight W;, and the reconstruction loss becomes i; instead of being averaged uniformly. We compare three settings: MAGE (w/o condition), which removes RTG conditioning entirely; MAGE (reweight), which applies the normalized RTG weights during training; and the full MAGE model. This scheme increases the influence of high-retum trajectories and suppresses low- return ones. shown in Table 23, reweighting provides clear improvement over the no-condition baseline, indicating that it can partially leverage RTG information. However, it still falls significantly short of the full MAGE model, suggesting that direct RTG conditioning remains more effective and reliable mechanism for guiding the generative process toward high-return behaviors.
al-
Evaluating performance on OGBench tasks. We further evaluate MAGE on the extremely long- horizon and rpware maze environments in OGBench (Park et al., 2025). These tasks present significant challenges due to their extended trajectories and limited feedback. As shown in Table 24, MAGE performs competitively across all scenarios and achieves the best results in three cases. While these findings demonstrate the potential of our coarse-to-fie generative framework in highly demanding settings, fully addressing such environments still requires additional exploration.
Evaluating the default configuration. We additionally conduct experiments using a default MAGE configuration (K 8, codebook size 512, transformer blocks 8). As shown in Table 25, MAGE (fixed config) denotes this default setup, while MAGE refers to our final tuned model. The results show that even with the default configuration, MAGE achieves strong performance across all tasks and consistently outperforms ADT and HD.
B.6 ANALYZING THE MULTI-SCALE DESIGN AND COMPONENTS OF MAGE
Our work is a multi-scale generation method for offline RL. It consists of 4 parts: a multi-scale autoencoder, multi-scale autoregressive transformer, inverse dynamics model with multi-scale input, and a condition- condition-guided decoder. The multi-scale autoencoder leverages multi-scale temporal information for encoding and decoding, whereas the multi-scale transformer leverages multi-scale
26

---

Published as a conference paper at ICLR 2026
Table 23: Effect of RTG-based reweighting compared with RTG conditioning.
Scenario
MAGE (w/o condition)
MAGE (reweight)
MAGE
Pen-Expert
92. 1 13.2
131.7 1 10.4
147.8 = 4.9
Door-Cloned
0.0 + 0.0
6.1 + 1.8
20.5 + 2.5
Table 24: Performance OGBench long- aon-horizoo maze tasks.
Scenario
Diffuser
ADT
HIQL
MAGE
pointmaze-giante -iviiaat-aa---i
0
19 19
46
52 5
antmaze-giant-ine giatt-naviggat--o
2#1
26
65
+
uatmuzz--deeprr--naiiitee--
8 =
23 +
42
49 - 5
numaanoidmazz-giin--naavva
0
2+1
12
Table 25: Performance with the default MAGE configuration.
Scenarlo
ADT
HD
MAGE (fixed config)
MAGE
Maze2d- Medium
109 4+6.2
135.6+3.0
146.9+4.2
155.0+3.3
Multi44 Medium
108.544 5+6.2
140.2+1.6
142.1+1.
147.743.1
Antmaze- Maetee-Medii--lle
82.0=17
42,0=1.9
90.0=3.0
92.0=2.7
83.4+19
88.7+8.1
94.8=2.2
98.2=1.3
Kitchen- Mixed
69.2+3.3
71,7=2.5
80.05..8
86.3+3.3
Kitchen Partial
64.245.1
73.3&1.4
86.344.1
91.3+3.2
Table 26: Effect of temporal scales the condition condition-guided decoder module.
Scenarlo
K=8 Lenns
K=1 Loon!
K=1
K=2
K=4
K=8
Pen-Expert
116.3 1.2
136.6 7.4
119.5 11.5
123.5
5.2
131.2 7.7
147.8 4.9
Door-Cloned
7.6
16.6 2.1
3.4
6.2 1.8
6.0
10.7
10.7
20.5
Hammer-Expert
117.4.6..6
128.3 + 0.3
113.2 6.2
116.5
121.7
127.9
1.1.7
Relocate- Expert
104.2 1: 0.4
108.9 +: 1.7
100.1 = 29
101.3 3.8
102.5 :: 1.8
105.9 1.4
109.6 + 1.6
temporal information for generation. The inverse dynamics model / makes use of latent multi-scale trajectory information to determine action. Moreover, the condition anddition-guieed guided decoder refines the finest scale information, which implicitly optimizes the multi-scale information too.
We conduct an experiment to understand the impact of multi-scale with different scales K and without the condition-guiee guided decoder (Multi-scale w/o Leone). The results are depicted the following Table 26. In this table, the Multi-scale K=8 w/o Leond column corresponds removing the condition condition-guided decoder module from MAGE. It still models the multi-scale information through a multi-scale autoencoder, a multi-scale transformer, and the inverse dynamics model with multi-scale input. Multi-scale K=1 w/o Leond is similar to Multi-scale K=8 w/o Leond with K equal to 1. K = 1 and K = 8 represent the case where is configured to or for MAGE, respectively. We have the following findings.
As we can observe from the table that using the multi-scale information can indeed performs better than its single-scale counterpart. For example, Multi-scale K=8 w/o Trcand performs better than Multi-scale K=1 w/o Trondd and K = 8 performs better than K
Using the cundition-guided decoder module Loond improve the performance of MAGE, but its contribution is not as high as increasing the scale. For example, on the Door-Cloned environment, Multi-scale K=8 w/o Leond is 13.2 higher than Multi-scale K=1 w/o Leond, while K 1 (with Leond) is only 1.8 higher than Multi-scan scale K=1 w/o Leond-
We show that through setting the scale and removing Leond, MAGE performs similarly to DT. For the Door-Cloned, the Hammer- Hammer-Expert, Expert, and the Relocate-Expor environment, MAGE without multi-scale and Ieond performs even slightly weaker than Decision Transformer (DT).
27

---

Published as a conference paper at ICLR 2026
B.7 DIFFERENCE AMONG MAGE AND OTHER HIERARCHICAL METHODS
MAGE distinguishes itself from other hierarchical methods through its multi-level structure and trajectory modeling. While ADT, HDMI, and HD adopt two-level hierarchy, MAGE employs a multi-level hierarchy that captures global route structures at a coarse level while refining local movements at finer levels. This design supports more coherent and consistent trajectory generation over long horizons.
The methods differ in their generation models and conditioning mechanisms. MAGE uses Transformer-basee generator to produce return- state pairs (R,S), and its condition includes the current state, target return (RTG), and outputs from higher levels. ADT and CARP generate full trajectories or action sequences, while HDMI and HD rely on diffusion models and mainly condition on state or subgoals. this table, G, R, S, and A represent subgoal, return-to-go, state, and action, respectively Table 27 summarizes these structural and conditioning differences among the methods.
Table 27: Comparison of different methods.
ADT
HDMI
CARP
MAGE
Level/scales
Two-level
Two-level
Two-level
Multi-scale
Multi-scale
Number of Policies
2
2
Generation Model
Transformer
Diffusion
Diffusion
Transformer
Transformer
Generated Data at the first level
G
G
latent of A
latent of (S,R)
Generated Data at the last level
S,A
S,A
S,A
latent of A
latent of (S,R)
Condition at the first level
$
S,R
S,R
S
S.R
Condition at the last level
S,G
S,G
S,G
S. latents of A
(S,R). latents of (S,R)
Return Alignment
Yes
Yes
Yes
No
Yes
C STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
Large language models were used solely as general-purpose tools to assist with language refinement, including improving grammar, style, and clarity of exposition. They were not involved in generating research ideas, designing methods, conducting experiments, or analyzing results. All scientific insights and contributions presented in this paper are entirely the work of the authors.
28


Reviews:
[{"review_id": "0mIMeCy9t0", "reviewer": "Reviewer_8qyC", "summary": "The paper tackles challenges in offline reinforcement learning, particularly for long-horizon tasks with sparse rewards, where existing generative models struggle due to inadequate modeling of multi-scale temporal dependencies. It introduces MAGE, a method that uses a multi-scale autoencoder to encode trajectories into hierarchical token maps at varying temporal resolutions and a multi-scale transformer to autoregressively generate these maps from coarse to fine scales, conditioned on initial state and return-to-go, with a condition-guided decoder for precise trajectory control. Key contributions include integrating multi-scale trajectory modeling with conditional guidance to produce coherent trajectories. Main results from experiments on five benchmarks demonstrate superior performance against fifteen baselines, especially in sparse-reward settings, with fast inference.", "questions": "- Could you please precisely summarize the novel algorithmic contributions of MAGE that are distinct from the VAR architecture?\n- The paper states that the action is determined by a latent inverse dynamics model $a = I(Z)$. The latent representation $Z$ is defined as the set of multi-scale latents $Z = (z_1, \\dots, z_K)$. How are these separate latent vectors aggregated (e.g., concatenation, summation, pooling) to form the single input $Z$ for the model $I$? This seems to be a critical implementation detail that is not specified.\n- Have you performed experiments comparing MAGE (with its latent inverse model) to a simpler variant where the main decoder $\\mathcal{D}$ is also trained to reconstruct actions, e.g., by modeling (R, S, A) triplets as in the ablated scheme from Table 6?\n- What is the specific advantage of using a separate adapter module for this task? Why not apply the $\\mathcal{L}_{cond}$ loss directly to the main decoder $\\mathcal{D}$ and update its parameters? Does the adapter-based approach offer better stability, prevent catastrophic forgetting of the trajectory prior, or lead to better performance?\n- Have the authors experimented with varying the number of scales K, and if so, how does performance change?\n- Is there a plan to release the code or datasets upon acceptance?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The multi-scale autoencoder captures hierarchical temporal dependencies by encoding trajectories into token maps from coarse to fine resolutions, enabling better handling of long-term structures, which enhances novelty by extending autoregressive models like VAR to temporal domains in RL.\n- Figure 1 effectively contrasts MAGE's hierarchical generation with Decision Transformer's sequential and Decision Diffuser's all-at-once approaches, immediately conveying the core conceptual contribution.\n- Temporal scale analysis demonstrates performance improvement with increasing K up to 8 scales, confirming the benefit of multi-scale modeling.", "weaknesses": "- The paper credits VAR for visual autoregressive modeling and states in the appendix that MAGE is \"implemented based on the source code of... VAR\". The core idea of MAGE (multi-scale, coarse-to-fine autoregressive generation) appears to be a direct application of the VAR architecture to RL trajectories. The paper should more clearly delineate its own novel contributions from the base architecture it adapts.\n- The final MAGE system is highly complex, involving a multi-scale VQ-VAE, a multi-scale autoregressive transformer, a separate latent inverse dynamics model, and an \"augmented decoder D' with a parameter-efficient refinement module\"for the $\\mathcal{L}_{cond}$ loss.\n- The necessity of the latent inverse dynamics model $I(Z)$ is not fully justified. The autoencoder's decoder $\\mathcal{D}$ already reconstructs the state trajectory $\\hat{\\tau}$. Why not decode actions as well, or infer them from consecutive reconstructed states? The ablation in Table 19 only compares a latent vs. explicit inverse model, but not the necessity of an inverse model at all.\n- The introduction of a separate adapter module just to implement the $\\mathcal{L}_{cond}$ loss adds complexity. It is not clear why this loss could not be applied directly to the main decoder $D$, or why this adapter-based approach is superior.\n- No formal analysis or theoretical bounds explaining why the specific hierarchical factorization improves long-horizon credit assignment or reduces error propagation compared to flat models.\n- The optimal number of scales K appears task-dependent but lacks principled guidance—no connection to trajectory length, reward sparsity, or environment dynamics is established.\n- The condition loss L_cond (Equation 7) only constrains the initial timestep, but provides no guarantee that subsequent states respect environment dynamics or maintain trajectory coherence beyond t=0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VUJIQayOgN", "reviewer": "Reviewer_6ooy", "summary": "The paper introduces MAGE, which (i) learns multi-scale discrete trajectory tokens via a condition-guided autoencoder, and (ii) performs coarse-to-fine autoregressive (AR) generation across scales with a Transformer. A lightweight adapter and an initial-condition consistency loss enforce alignment with the input condition (initial state and RTG). Discrete tokens are trained with cross-entropy and Gumbel-Softmax/STE. Experiments on long-horizon suites (Adroit, Franka Kitchen, AntMaze, Maze2D/Multi2D) show strong gains, with ablations on the number of scales, sequence schemes, and conditioning terms.", "questions": "1. Related to weakness 3, is it possible to use value/success reweighting?\n2. Related to weakness 4, in sparse-reward settings, if the target or waypoint distribution shifts, and especially when new destinations are not well represented in the codebook, I wonder if the performance is likely to degrade.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a coherent multi-scale discrete trajectory representation coupled with cross-scale AR generation (global dependencies at coarse scales; local refinements at fine scales).\n2. This paper proposes a condition-guided decoder with explicit initial-condition loss to reduce quantization/AR mismatch.\n3. This work presents strong empirical results on long-horizon, sparse-reward tasks, plus ablations on hierarchy depth and conditioning.", "weaknesses": "1. No RTG-value generalization study. The method conditions on RTG but does not report performance vs. different target RTGs (e.g., low/medium/high percentiles or out-of-distribution RTGs).\n2. The same framework requires different codebook sizes, hierarchy depth K, and network depth across suites, indicating high sensitivity to task distribution and substantial tuning burden.\n3. Using inverse dynamics cloning plus conditional reconstruction encourages alignment with the behaviour distribution, which can yield trajectories that “look aligned” but do not guarantee high-return action. Is it possible to use value/success reweighting?\n4. The model conditions only on (s0, R) and does not explicitly encode goals or intermediate waypoints. In sparse-reward settings, if the target or waypoint distribution shifts, and especially when new destinations are not well represented in the codebook, I wonder if the performance is likely to degrade. Moreover, the coarse-to-fine AR procedure locks in global structure at coarse scales, while fine scales can only make local corrections, making early mistakes difficult to recover over long horizons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "By2bzZb840", "reviewer": "Reviewer_gDvQ", "summary": "The paper proposes MAGE, a novel approach for offline RL by introducing a multi-scale autoencoder and a transformer to generate trajectories from coarse to fine scale. It achieves strong empirical results on several offline RL benchmarks, which require planning on long horizons with sparse rewards.", "questions": "- It would be nice to place the overview figure in the appendix to the main text. It is hard to figure the main concept without that figure.\n\n- It seems that with a very small number of K (e.g.,1,2), it achieves better performance compared to the baselines. Could the authors explain why such a phenomenon is possible?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Hierarchical generation with autoregressive models is a relevant topic in offline RL.\n\n- Strong empirical results in several offline RL benchmarks.\n\n- Idea is clear and also it exhibits short inference time, which is a main bottleneck of diffusion-based method for offline RL.", "weaknesses": "- While authors list up hyperparameters for each task in the appendix, several crucial hyperparameters are missing: RTG conditioning value, $\\lambda_{\\text{cond}}$, and number of temporal scales (K). \n\n- It would be better to evaluate the method with much larger mazes, such as pointmaze-giant and antmaze-giant suggested in OGBench, to clearly verify the effectiveness of the proposed method in long-horizon settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZBxY1rf2aO", "reviewer": "Reviewer_da3r", "summary": "This paper presents MAGE, a novel generation-based offline reinforcement learning framework designed to tackle long-horizon tasks with sparse rewards. Unlike prior hierarchical generation methods that fail to capture the intrinsic multi-scale temporal structure of trajectories, MAGE introduces a condition-guided multi-scale autoencoder for learning hierarchical trajectory representations and a multi-scale transformer that autoregressively generates trajectories from coarse to fine temporal scales. This approach effectively models temporal dependencies across resolutions while enabling precise short-term control via a condition-guided decoder. Extensive experiments on five offline RL benchmarks against fifteen baselines demonstrate that MAGE produces coherent and controllable trajectories, achieving state-of-the-art performance in challenging long-horizon, sparse-reward settings.", "questions": "- Could the authors provide additional results on more challenging tasks from OGBench to further demonstrate the effectiveness of the proposed method?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed idea is novel, adapting the concept of multi-scale generation to the offline RL setting, which was originally explored in some computer vision literature (e.g., [1]).  \n- The empirical evaluation is comprehensive, covering a diverse set of baselines (including model-free, autoregressive, and generative approaches) and a wide variety of offline RL benchmarks such as maze navigation, MuJoCo locomotion, Adroit manipulation, and kitchen environments.  \n- The paper is clearly written and easy to follow, supported by intuitive illustrations (Figure 1), a well-designed toy experiment that highlights the advantages of the proposed method (Figure 2), and a clear architectural overview (Figure 3).\n\n[1] Tian, K., Jiang, Y., Yuan, Z., Peng, B., & Wang, L. (2024). Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37, 84839-84865.", "weaknesses": "- Since the authors claim that the proposed method performs well on long-horizon tasks, it would be interesting to evaluate it on more challenging maze tasks in OGBench [1], such as *antmaze-giant-navigate*, *antmaze-teleport-navigate*, and *humanoidmaze-giant-navigate*.  \n- There are a few typos in the paper — for example, “Casual” should be corrected to “Causal” in Figure 3(b).\n\n[1] Park, S., Frans, K., Eysenbach, B., & Levine, S. (2024). Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)