Scalable Offline Model-Based RL with Action Chunks
Authors: Kwanyoung Park
Seohong Park
Youngwoon Lee
Sergey Levine
Venue: ICLR 2026 Poster (2026)
Keywords: Offline RL
World Models
Model-based RL
Action chunking
Long-horizon tasks
Primary area: reinforcement learning
TL;DR: Action chunking enables long-horizon rollouts, scaling offline model-based RL to complex long-horizon tasks.

Abstract: In this paper, we study whether model-based reinforcement learning (RL), in particular model-based value expansion,
can provide a scalable recipe for tackling complex, long-horizon tasks in offline RL. 
Model-based value expansion fits an on-policy value function using length-$n$ imaginary rollouts generated by the current policy and a learned dynamics model.
While larger $n$ reduces bias in value bootstrapping, it amplifies accumulated model errors over long horizons, degrading future predictions.
We address this trade-off with 
an *action-chunk* model that predicts a future state from a sequence of actions (an "action chunk")
instead of a single action, which reduces compounding errors.
In addition, instead of directly training a policy to maximize rewards,
we employ rejection sampling from an expressive behavioral action-chunk policy,
which prevents model exploitation from out-of-distribution actions.
We call this recipe **Model-Based RL with Action Chunks (MAC)**.
Through experiments on highly challenging tasks with large-scale datasets of up to $100$M transitions,
we show that MAC achieves the best performance among offline model-based RL algorithms,
especially on challenging long-horizon tasks.

Published as a conference paper at ICLR 2026
SCALABLE OFFLINE MODEL-BASED RL WITH ACTION CHUNKS
Kwanyoung Park' Seohong Park'
Youngwoon Lee²
Sergey Levine!
UC Berkeley' Yonsei University?
https://kwwanyougggae www.gh.i.t..h....e.....o
ABSTRACT
In this paper, we study whether model-based reinforcement learning (RL), in par- ticular model-based value expansion, can provide a scalable recipe for tackling complex, long-horizon tasks in offline RL. Model-based value expansion fits an on-policy value function using length-n imaginary rollouts generated by the cur- rent policy and learned dynamics model. While larger reduces bias in value bootstrapping. it amplifies accumulated model errors over long horizons, degrad- ing future predictions. We address this trade-off with an action-chunk model that predicts a future state from a sequence of actions (an "action chunk") instead of a single action, which reduces compounding errors. In addition, instead of directly training a policy to maximize rewards, we employ rejection sampling from an ex- pressive behavioral action-chunk policy, which prevents model exploitation from -itor--distritttion actions. We call this recipe Model-Based RL with Action Chunks (MAC). Through experiments on highly challenging tasks with large- scale datasets of up to 100M transitions, we show that MAC achieves the best per- formance among offline model-based RL algorithms, especially on challenging long-horize horizon tasks.
One-Step Model
Actien-Chunl Xodel
Gauesian Policy Learning
Flow Rejection Sampling
Two main components MAC. (Left) Action-chunk models predict future state given sequence actions (an "action chunk"), reducing compounding errors and enabling long-horizon model rollouts. (Right) Rejection sampling from an expressive (flow) behavioral action-chunk policy enables modeling multi-modal action distributions, while preventing model exploitation from out-of-distribution actions.
Figure 1:
INTRODUCTION
Offline reinforcement learning (RL) holds the promise of training effective decision-making agents from data, leveraging large-scale datasets. While offline RL has achieved successes in diverse do- mains (Kumar al., 2023; Springenberg et al., 2024), its ability to handle complex, long-horizon tasks remains an open question. Prior work has shown that standard, model-free offline RL often struggles to scale to such tasks (Park et al., 2025b), hypothesizing that the cause lies in the patholo- gies of off- off-policy, temporal difference (TD) value learning.
In this work, we investigate whether an alternative paradigm, namely model-based RL, and in par- ticular model-based value expansion (Feinberg et al., 2018), provides a more effective recipe for long-horizon offline RL. this recipe, we first train a dynamics model, and fit an on-policy value function by rolling out the current policy within the learned model, which is then used to update the policy. Since on- on-pol value learning has demonstrated promising scalability to long-horizon tasks (Berner et al., 2019; Guo al., 2025), in contrast to the relatively limited evidence for off-

---

Published as a conference paper at ICLR 2026
policy TD learning (Park et al., 2025b), hypothesize that the combination of on-policy value learning and dynamics modeling may also exhibit strong horizon scalability.
However, there is a tricky trade-off in this recipe. In model-based value expansion, we typically train a value function by rolling out the policy for n steps within the model and regressing toward the Here is the dilemma. On the one hand, we want to use a large n in this value update, as this reduces the bias in the bootstrapped target value, y ("V(s)+n)) This is particularly important given that bias accumulation one of the major factors that hinder the scaling of offline RL (Park et al., 2025b). On the other hand, we want to keep n small enough, as errors in the dynamics model accumulate through autoregressive queries over the horizon. Is there a solution to this trade-off that enables long-horizon model rollouts while preventing error accumulation?
target: (($) < Ei=
Our main hypothesis in this work is that action models and policies, combined with recent innovations expressive generative models, can provide natural solution the above dilemma, enabling scaling of offline model-based RI. to hong-horizon tasks. Namely, instead of training a single-step model p(5++1 St, at), we train a multi-step model p(*++n 8t,, 0c:t+n-1) that takes an action-chunk au.++-- as input and predicts a future state that is n-step ahead. This substantially reduces the number of recursive model calls and mitigates compounding errors (Figure I, left),
enabling long-horizon imaginary rollouts over 100 environment steps.
To use an action-chunk model the model-based actor-critic framework, we need an action-chunk policy. However, directly training a reward- maximizing action-chunk policy is challenging in of- fline RL, due to the potentially multi-modal, high-dimensional action-chunk distributions in the dataset (Li et al., 2025). Hence, employ rejection sampling based on samples from an expressive behavioral action-chunk policy trained with flow matching (Lipman et al., 2024). By simply defin- ing the policy as the behavioral action-chunk sample that maximizes the value function, we can not only capture complex action distributions from the dataset (Figure 1, right), but also effectively pre- vent model exploitation (Kidambi et al., 2020).
We call this recipe Model-Based RL with Action Chunks (MAC), which constitutes the main con- tribution of this work. Experimentally, we show that MAC vastly improves the horizon scalability of offline model-based RL. In particular, we demonstrate that our scalable model-based RL recipe can consume 100M-scale data to achieve state-of-the-art performance on highly complex, long-horizon robotic manipulation tasks from OGBench (Park et al., 2025a), often outperforming previous model- free and model-based approaches.
RELATED WORK
Offline model-free RL. Offline RL aims to learn a return-maximizing policy from previously col- lected dataset, without interaction with the environment (Lange et al., 2012; Levine et al., 2020), As in online RL, offline RL methods can be categorized into model-free and model-based ones. Offline model-free RL methods train a policy without learning a dynamics model. Prior works have pro- posed number of model-free approaches based on diverse techniques, such as conservatism (Ku- mar et al., 2020), behavioral regularization (Wu et al., 2019; Peng et al., 2019; Nair et al., 2020; Fu- jimoto & Gu, 2021; Tarasov al., 2023; Park et al., 2025c), uncertainty estimation (An al., 2021; Nikulin et al., 2023), in-sample maximization (Kostrikov et al., 2022; Xu et al., 2023; Garg et al., 2023), rejection sampling (Chen et al., 2023; Manssn-Estrrchh et al., 2023), and more (Brandfon- brener et al., 2021; Sikchi et al., 2024).
Offline model-based RL. In this work, we focus on offline model-based RL, a paradigm that first trains a dynamics or trajectory model, and then trains a policy based on rollouts generated from the learned model. line of work trains generative models (e.g., Transformers (Vaswani et al., 2017) and diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020)) to model the entire trajectory distribution of the dataset, and typically use conditioning and guidance to compute actions (Chen et al., 2021; Janner al., 202 2022; Lee et al., 2022; Ajay et al., 2023; Jiang et al., 2023; Li et al., 2023; Chen et al., 2024; Ding et al., 2024; Jackson et al., 2024; Cheng et al., 2025). Another line of work trains a (typically single-step) dynamics model, and trains policy based on rollouts autore- gressively sampled from the learned model. These approaches employ the learned dynamics model for (1) "Dyna" -style data augmentation (Sutton, 1991; Janner et al., 2019; Yu et al., 2020; Kidambi
2

---

Published as a conference paper at ICLR 2026
et al., 2020; Yu et al., 2021; Rigter et al., 2022; Sun et al., 2023; Sims et al., 2024; Lu et al., 2023a), (2) planning (Testud et al., 1978; Argenson & Dulac-Amold, 2021; Chitnis et al., 2024; Zhou et al,, 2025), and (3) value estimation (Feinberg et al., 2018; Jeong et al., 2023; Park & Lee, 2025; Hafner et al., 2025), with diverse techniques to prevent model exploitation and distributional shift, such as ensemble-based uncertainty estimation. Our method is based on model-based value expansion and falls in the third category. However, unlike most of the previous works in this category, we employ an action-chunk model instead of single-step dynamics model to reduce effective horizons and thus error accumulation.
Horizon reduction and model-based RL. The curse of horizon is fundamental challenge in rein- forcement learning (Liu et al., 2018; Park et al., 2025b). In the context of model-free RL, previous studies have proposed diverse techniques to reduce effective horizon lengths, such as n-step returns to reduce the number of Bellman updates (Sutton & Barto, 2005), and hierarchical policies to reduce the length of the effective policy horizon (Nachum et al., 2018; Park et al., 2023). Long horizons are central challenge in model-based RL too, since model rollouts suffer from compounding errors as the horizon grows. Prior works in model-based RL address this challenge with trajectory model- ing (Janner et al., 2021b; 2022), hierarchical planning (Li et al., 2023; Chen et al., 2024), skill-based action abstraction (Shi et al.. 2022), and action-chunk multi-step dynamics modeling (Asadi et al., 2019; Lambert al., 2021; Zhao et al., 2024; Zhou al., 2025). Our work is closest prior works that use action-chunk dynamics models. However, these works either use the action-chunk model only for planning without having the full actor-critic loop (Asadi et al., 2019; Lambert et al., 2021; Zhou et al., 2025), or model the entire state-action chunks (Zhao et al., 2024). Unlike these prior works, we perform on-policy value learning with an action-chunk model and policy, while not in- volving additional planning or full trajectory generation.
3 PRELIMINARIES
Problem setting. We consider a Markov decision process (MDP) defined as M (S, A,, p),) where S is the state space, A Rd is the action space, r(s, a) function, pus) E ASS) is the initial state distribution, and p(s s,a) : S x A + ASS) is the transition dynamics kernel. A(X) denotes the set of probability distributions on space X, and we denote placeholder variables in gray. For a policy T(a s) S + ( we define (*) wpp (1===))) E > 7'r(st, az)] and (s,ssa) a) = Ex (r (r130=
S x A + R is the reward
Lt= (F((ss,a))) (50, ao, ro, .1,...) denotes a trajectory, and p" denotes the trajectory distribution induced by 14, p. and т. The goal of offline RL is to find a policy v that maximizes Ee V (so) from an offline dataset D (7) consisting of previously collected trajectories, with no environment interactions.
where y (0, 1) denotes the discount factor,
Flow matching. Flow matching (Lipman et al., 2023; Albergo & Vanden- Vanden-Eijnden, 2023; Liu et al., 2023) is a technique in generative modeling to train a velocity field whose flow generates target distribution of interest. As with diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020), flow models iteratively transform noise distribution to the target distribution, and have been shown to be highly expressive and scalable (Esser et al., 2024; Lipman et al., 2024).
Formally, assume that we are given a target distribution p(z) E A(R*). For time-dependent
velocity field v(u,z) : 10, 1] x R -> R (we use u to denote times in flow matching to avoid notational conflicts with environment steps in MDPs), define its flow, th(,, x) [0, 1] xR* R$,
as the unique solution to the following ordinary differential equation (ODE) (Lee, 2012):
d du
(w, x)) v(u, x))
Flow matching aims to find velocity field whose flow transforms a noise distribution (e.g., k- dimensional standard Gaussian, (N La)) at u = 0 to the target distribution at u = 1.
Prior work (Lipman et al., 2023; Albergo & Vannden-Eijdden, 2023; Liu al., 2023) has shown that we can train such a velocity field by minimizing the following loss:
E
(0,, 2))p(2),
[lle(u, x.)) - (21 - =x0)
(0, x. x.=(1- tuar
3

---

Published as a conference paper at ICLR 2026
We refer to the tutorial by Lipman et al. (2024) for detailed explanations and proofs. After training the velocity field, we can obtain samples from the target distribution by numerically following the velocity field to solve the ODE in practice (e.g., with the Euler method).
4 OFFLINE MODEL-BASED RL WITH ACTION CHUNKS
Motivation. Our high-level goal is to scale up offline model-based RL to complex, long-horizon decision-making problems. Among model-based RL frameworks, we specifically focus model- based value expansion (Feinberg et al., 2018), which combines dynamics modeling and on-policy value learning. This is because each of these components, namely generative modeling and on- policy RL, has individually been shown to scale to long- horizon tasks (Berner et al., 2019; Harvey al., 2022; Guo et al., 2025).
In model- based value expansion, we first train dynamics model, and train an on-policy value function with the following update:
V(S)) + Σ (r((ste;((+.))) (((+n))
where (st št, åe, sc+1..... Stnn) is length-n imaginary rollout sampled from the model using the current policy, and V is a target value function. The policy is then updated to maximize the learned value function, and we repeat this procedure.
The problem is: how long should model rollouts be? Unfortunately, we have two seemingly contra- dictory desiderata.
On the one hand, we want model rollouts to be long enough. If n is too small, we end up with large number of biased value updates with short-horizon bootstrapping Equation (3). This causes the biases to accumulate over the horizon, which is known to be one of the main obstacles hindering value-based RL from scaling to long-horizon tasks (Park et al., 2025b). Hence, we want to keep n large enough.
On the other hand, we want model rollouts to be short enough. If we use a standard policy x(a 8), we need to autoregressively call learned dynamics model n times generate length-n model rollout (Št, å, $1+1..... $t+n). This makes errors in the dynamics model accumulate within the trajectory chunk, which would degrade performance. Hence, want to keep n small enough.
there a way to naturally resolve this dilemma?
4.1 THE IDEA
Our main idea in this work is that a combination of an action-chunk policy and an action-chunk model can provide clean solution to the above dilemma, enabling scaling complex, long-horizon tasks. Specifically, we train an oction-chunk model p(5t4n St, 4a.t-n1) S x Ar + ASS) and an action-chunk policy π T(der+n-1 st) S + (AA"), where ag; denotes the action chunk (a,, 0.+1...... a,). Since each individual call of the model generates n actions at once, we can reduce the number of recursive model calls by a factor of n. This way, we can mitigate both bias accumulation in value learning and error accumulation in model rollouts.
However, several challenges remain in implementing this idea in practice. First, Gaussian policies, used in many previous works in offline model-based RI (Yu et al., 2020; Sun et al., 2023; Lu et al., 2023b; Chitnis al., 2024; Park Lee, 2025), are generally not expressive enough to model complex, multi-modal action-chunk distributions (Figure 1). Second, penalizing out-of-distribution actions based on uncertainty in the dynamics model, as typically done by prior work offline model-based RL (Yu al., 2020; Kidambi al., 2020; Sun al., 2023), can be challenging due to the potentially high complexity of the action-chunker dynamics distribution.
To handle these challenges, chunk policy. Specifically, we use flow matching (Lipman et al., 2024) to train a behavioral cloning (BC) action-chunk policy, and define policy the arg max action chunk (among N chunks sam-
employ rejection sampling from expressive behavioral action-
4

---

Published as a conference paper at ICLR 2026
Algorithm Offline Bosel- RI. with Action Chunks (MAC)
Input: Dataset D, rollout length H, action chunking size rejection sampling size M
Training loop
while not converged do
Sample action-chunked batch from the datasct (a: = Octin-1: nn=1 y'rot)
Sample batch {(t,, (
ae, rt, St-n)) D
P Train BC policy using dataset transitions
Update flow BC policy Ty with flow-matching loss (Equation (7))
Update one-step BC policy Tu with distillation loss (Equation (8))
Train dynamics and reward model using dataset transitions
Update dynamics model pe to minimize B[ilps (8t, ax) 8t+ n [ (Equation (5))
Update reward model To minimize [s ax) -rellg (Equation (6))
D. Generate model rollouts (&+ = 8t) for l 0.1,.... H do 0rtkn <- POLICY(&L+AR))
#t+(k+1)m= Pr+kn ~ Po(: 8etkn, dc+kn)s r ( Betkn, kz+kn)
Update value using model rollouts
Update value Vi with n//-step targets from the rollout (Equation (10))
Learn critic for faster rejection sampling
Update critic Qp with the learned value function Vp (Equation (11))
// Extract action flow BC policy me with rejection sampling function POLICY(S)
(0, I) ( Th (s)
return argmax ..) a00) (p(s, (1)
pled from the BC policy) that maximizes the learned value function:
x (Bt)
arg max
(ss (1t+n-1))
fa I o (auton- (sc)
where n7 (ous4n-1
8t) A(A") denotes action-chunk flow BC policy, Q(st, o
S x _ R denotes an action-chunk value function, and d denotes equality in distribution.
Compared to Gaussian policies, the flow- behavior policy better models multi-modal action distributions, allowing to sample action chunks that stay in-distribution, which obviates the need for an additional uncertainty penalization mechanism. Moreover, rejection sampling is generally more robust to hyperparameters (Zhou et al., 2025; Park et al., 2025b), making method simpler and easier to tune than other alternatives, which may require tuning an uncertainty penalization coefficient for each task.
4.2 PRACTICAL ALGORITHM
Based on the idea discussed the previous section, we now describe the full details our method for scalable offline model-based RL, which we call Model-Based RL with Action Chunks (MAC). MAC consists of the following components: an action-chun chunk dynamics model pv. an action-chuni reward model Tsh. flow action-chunk policy na and value functions Vp and Q.. For notational simplicity, we override the symbols th, and &p to denote all model-, and value-related parameters, respectively. Moreover, denote discounted rewards for steps Σn11 7'repi-
A to be the action chunk aua+n and r. be the sum of
Action-chunk dynamics and reward models. For dynamics modeling, we minimize the following losses train deterministic action-chunk dynamies model po 0z) S x An S and an
5

---

Published as a conference paper at ICLR 2026
action-chunk reward model ro( :8 A + R:
IdN E (s (
p ($t, ai) sttn
(5)
LTT 16 (s, (14)))
[|r ar) rel12
(6)
where trajectory chunks are uniformly sampled from the offline dataset. The dynamics function pa modeled by a deterministic multi-layer perceptron (MLP). While we found this to be sufficient in our benchmark environments, we note that it is possible to replace the MLP with an expressive flow model (as in our policy) in stochastic or partially observable environments.
Flow action-chunk policies. For the actor, we employ rejection sampling using a behavioral flow action-chunk policy, as described in Section 4.1. To train flow BC policy. we train a state- dependent velocity field vo IR x S x A" + A", with the flow-matching loss (Equation (2)):
L Row (0) = EE a0, (e, (1), [Ilu(u, St. ag) - (a, - 2)
w>-Unif([0.1]), a. dI=(1-w)a+ur
We define To (s 2) A" as the destination of the induced flow at 1 when starting with (st, 2) at u = and following the velocity field Ug. Then, by sampling multiple noises & ~ NO, Ina) and computing (8t. z), we can obtain behavioral action-chunk samples, which are then used for rejec- tion sampling (Equation (4)) along with a learned value function (described in the "Value learning" section below).
One issue with this rejection sampling framework is speed. To compute a single action chunk using Equation (4), we need N queries of the velocity field up, where N is the number of samples and F is the number of flow steps in the Euler method For example, with N = 8 and F = 10, we need to query the velocity field 80 times to sample single action chunk. This is particularly prohibitive in bosel- RL, as we need to sample multiple imaginary rollouts during training in batches, unlike methods that employ rejection sampling only at test time (Hansen-Estruch al., 2023; Park et al., 2025b; Zhou et al., 2025).
To address this issue, we train an additional one-step" flow policy that directly predicts the output of the ODE flow policy. Specifically, we train a one-step MLP action-chunk policy nu (st,2) =) x A + A" parameterized by w, with the following flow distillation loss (Park et al., 2025c):
L disttill (w) E,,,D, so,, [ 2) [To [x)xx11]]x113 11]]
where |xx denotes the "stop gradient" operation.
Unlike the ODE policy, The only requires a single network call to produce an action chunk, reducing the number of queries from NF to N for rejection sampling in Equation (4). This substantially reduces both the training and inference cost of MAC.
Value learning. In MAC, value functions are trained from on-policy model rollouts (i.e., imaginary trajectories). To train value functions, we first generate M imaginary (action-chunk) trajectories of length H.
(9) where à denotes the action chunk âz:t+ generated from the rejection sampling policy, and f. denotes the discounted sum of rewards EnD "rr++ predicted from the reward model r-p-- a.). Here, initial states s are uniformly sampled from the dataset, and subsequent actions, rewards, and next states are synthesized by respectively, hence the hat notation.
Diug = ((5 d
staeliigg policy, reward model, and dynamics model,
After collecting Dimg, we update the value function V₂(*t): S R with the following loss:
L (4) = E [ (Vo(
#-1
27
Σ
(i-)))
(M (MM-R)aV (2(
(10)
WWe note that wall-clock training time heavily depends on F' than N, since rejection sampling can be parallelized, while flow sampling is not.
We emphasize that "one-step" is different from "environment steps" in RL. Although the phrasing can be ambiguous, "one-step" (or "single-step") is the standard term for single-step distillation procedures (e.g., as used in Park et al. (2025c); Frans et al. (2025)). Because this terminology is already conventional, we would like retain "one-step" for consistency with prior works.

---

Published as a conference paper at ICLR 2026
where o denotes exponentially averaged target parameters (Mnih al., 2013), and the expectations are over (s; = St, ag f....., H1) uniformly sampled from Dimg and uniformly sampled from HO 1}.
Finally, we train the action-chunk Q function with the following loss:
A R for the rejection sampling
L (g) ES,DD - (St4nn)))
(1I)
We do not reuse Dimg after performing one gradient update of value functions; i.e., we generate new model rollouts every epoch. We provide pseudocode for MAC in Algorithm 1.
Notes on hyperparameters. While MAC has several learnable components, MAC is comparatively easier to tune the hyperparameters than prior methods in our experiments. In particular, we use the same horizon hyperparameters of (n, H) (10, 10) for all tasks considered in this work. We also use the same number (N 32) of samples for flow rejection sampling during evaluation across all tasks. That is, we can use the hyperparameters across all tasks, while prior model-based baselines (e.g., MOBILE, MOPO) requires task-speciti rollout horizons and uncertainty penalties to remain stable. See Appendix A for the full details.
5 EXPERIMENTS
Now, we empirically evaluate the performance of MAC through a series of experiments. Our main rescarch question is how well MAC scales to long- Long-horizon tasks compared to previous offline model- based RL approaches, which we answer in Section 5.1. Then, we compare MAC with previous methods standard offline RL benchmark tasks assess its effectiveness as a general offline RL algorithm (Section 5.2). Finally, we provide several analyses and ablation studies to understand the importance of each component of MAC (Section 5.3). In experiments, (unless otherwise mentioned) and report standard deviations in tables and 95% confidence intervals plots. In tables, we highlight numbers that are above or equal 95% of the best performance.
use four random seeds
5.1 EXPERIMENTS ON LARGE- LARGE-SCALE,
TASKS
We first study the horizon scalability of MAC by evaluating it on large-scale, long-horizon bench- mark tasks.
Tasks and datasets. To assess the scalability limits of each algorithm, employ three highly challenging. long-horizon simulated robotic tasks used in the work by Park et al. (2025b) modified from OGBench (Park et al., 2025a): humanoidmazer wwwraniidmzzeegiane -giant, cube-octuple, and puzzle-4xt5 These tasks are not just long-horizon but also goal-conditioned (Le., the agent must reach any goal states given test time), requiring complex, multi-task reasoning over a long episode. They present a variety of control challenges from high-dimensional humanoid navigation to complex ob- ject manipulation and combinatorial puzzle solving. The hardest task in each environment requires 700-3000 environment steps and 8-20 different atomic motions to complete. In addition to these long-horizon tasks, we also evaluate methods on thorter-horizon variants in each category (i.e., humano iirtanoddmazeiieee handle different horizon lengths.
cube-double, and puzzle-3x3) to examine each method's ability to
For datasets, we mainly employ the large-scale datasets are collected in a task-agnostic manner (e.g., trajectories consisting of random atomic motions), meaning that the agent must understand the dynamics and stitch different parts of trajectories to achieve
datasets provided by Park et al. (2025b). These
tasks.
Methods. We mainly compare MAC against six previous model-based RL methods across diverse categories, including flat and hicrarchical, and actor-critic and planning approaches.
Among standard model-based RL approaches, we consider MOPO, MOBILE, LEQ, and F-MPC. MOPO (Yu et al., 2020) and MOBILE (Sun et al., 2023) are Dyna-style methods (i.e. ones that gen- erate imaginary rollouts, augment the dataset, and off off-policy RL) based on different uncertainty penalization techniques. LEQ (Park & Lee, 2025) is a model-based actor-critic method based on conservative return estimation. F-MPC is flow-based variant of D-MPC (Zhou et al., 2025). which
7

---

Published as a conference paper at ICLR 2026
Table 1: Results on large-scale, long-horizon tasks. MAC achieves the best performance among model- based RL algorithms.
Modol-Free
Beq. Nodeling
Model-Basee Baseed
Ervironnen:
OCICL
n-SAC+BC
SEARBA
Diffuser
HD-DA
NOFO
MOBILE
LSQ
FXFC
NAC
humanol draze-medi unw.nnugatee mnv.gaterrracllec erappro
55
86
0.
0.
27
23
18
36 =2
drwze-giant aataaiigattaalrrreeeeee
0=0
cube-double-playy waallll--p----lllrrraaa cracleerc0----0000
74
22
1:43
24:
25
cube-oc sabe--ctappe-pe wecttt-lpply---ee ora.e-ppa-rrrccoogepeee
0-o
0.
19
0se
0.4
0-
0-
0.
0.
30 44
puzzle 3ax3 -aappppay--ae olay-orac
90
11
100
1.
1
19
15.
11
12
100 Lo
puzzle-4e Sasilll-----ll---- praylooaal
20 19
91
0se
0ss
0.e
0.e
1.9
0.m
99
trains an action-chunk dynamics model as in our method, but performs planning (based on behav- ioral Monte Carlo value function) instead of training an on-policy value function with actor-critic.
Among sequence modeling approaches, we consider Diffuser and HD-DA. Diffuser (Janner al., 2022) models trajectories with diffusion (Ho et al., 2020) for planning, and HD-DA (Chen et al., 2024) extends Diffuser using hierarchical models and high-level planning to handle long horizons.
For reference, we additionally consider three performant model-free RL algorithms as well: IQL, n- SAC+BC, and SHARSA. IQL (Kostrikov al., 2022) a standard model-free offiine RL algorithm based on in-sample value learning. n-SAC+BC (Park et al., 2025b) is a behavior-regularized offline RL method that employs n-step returns handle long horizons. SHARSA (Park et al., 2025b) is a sfate-of-thh-art offline RL algorithm designed for mong-oorizon tasks that employs hicrarchical policies and flow rejection sampling.
5.1.1 RESULTS
We present the main comparison results on six tasks in Table 1. The results suggest that MAC achieves the best performance across all settings among model-based RL algorithms. In particular, none of the previous model-based RL approaches achieves non-trivial performance on three long- horizon tasks. This is likely because they either use single-step models, which suffer from error accumulation (see Figure 2), or are based on planning, which is insufficient to perform full-fledged long-horizon dynamic programming. Moreover, even compared state-of -ftie-of----arrt model-free RL approaches (e.g.. SHARSA), MAC achieves the best performance on four out of six tasks, especially on long-horizon manipulation tasks (cube-octuple and puzzle-4x5)
Negative results. Despite its strength on manipulation tasks, MAC, as well as all other model- based RL approaches, struggles on long-horizon robotic locomotion tasks (e.g., numanoidmant dmaze-giant)) This is widely known phenomenon; prior works (Chitnis et al., 2024; Park & Lee, 2025) have also found that model-based RL particularly struggles in similar robotic maze navigation environments (e.g., antmaze- in D4RL (Fu et al., 2020)). We believe this is mainly due to the difficulties in modeling contact-rich dynamics in locomotion domains, where dynamics tend to be highly erratic due discontinuities, resulting in severe model error accumulation. While MAC's action-chunk dynamics model does mitigate this issue to some extent, leading to the best performance among model-based RL approaches (Table 1), it is not sufficient to fully close the gap between model-free and model-based approaches on these locomotion tasks. We believe this issue may be addressed by more expressive generative models latent dynamics models, which we leave for future work.
5.2 EXPERIMENTS ON STANDARD BENCHMARKS
Next, we evaluate MAC on standard, reward-based benchmark tasks to assess its ability to serve as a general offline RL algorithm under limited data.
Tasks and datasets. We employ 25 single-task manipulation tasks from five environments OG- Bench (Park et al., 2025a): cube-(single, double), scene, and puzzle-(3x3, 4x4). Unlike in Section 5. these tasks are reward-based (i.e., not goal-conditioned), where the agent gets a re- ward according the progress of the task. We use the 1M-sized play datasets given by the bench- mark. We report the average success rate across tasks for each environment.
Methods. For model-based approaches, we consider the four standard model- RL algorithms used in Section 5. Additionally, we consider four standard, performant model-free RL algorithms used in the work by Park al. (2025c): IQL. (Kostrikov al., 2022), ReBRAC (Tarasov et al.,
8

---

Published as a conference paper at ICLR 2026
Table Results on standard reward-based benchmark tasks. MAC achieves the best performance among both model-base based and model-free RL algorithms.
Kodel-Free
Hodel-Basee
Bavironnen:
IQL
ReZRAC
IDQL
FQL
NDFC
NOBILE
LEQ
FEPC
MAC
cuberaing wnl.lnglllllee plmplayy-vO (5 tasks)
83
91
06
98
12.s4
81
0:50
9.
99
cube-doubl 1appayy (5 taaks)
acene-play-v0 (5 taaks)
28
6**
0:00
puzzle-Bxz 3x3-play-vV (5 tasks)
9
21 23
10:4
30 km
20
12
10
1.4
20 10
puzzle-txc tat---ayy--eeeee (5 taske)
7.14
14.1
29 13
17
0.1
0.1
0.10
0.16
70
2023), IDQL HHansen-Estruce et al., 2023), and FQL (Park et al., 2025c). Among them, FQL is ofte-of-thhear model-free offline RL method on these tasks.
5.2. RESULTS
Table summarizes the comparison results on 25 standard benchmark tasks. The results show that MAC achieves the best average performance on four out of five environments. Notably, MAC achieves substantially better performance than all other methods especially on (relatively) long- horizon environments, such as cube-double, scene, and puzzle-4x4. MAC also outperforms state-of-the-ar model-free RL algorithms, showing the promise of offline model-based RI. in ma- nipulation domains.
5.3 Q&
In this section, we discuss and analyze the components of MAC through the following Q&As.
Q:
Do action chunks actually mitigate error accumulation?
A: Our main motivation for using action chunking is to re- duce error accumulation in autoregressive trajectory genera- tion. However, one might question whether it is actually the case, given that increasing the action chunk length also in- creases the difficulty of learning the model. To examine this, we analyze how the chunk length affects model errors. Specif- ically, we train dynamics models with action chunk lengths of (1,5, 5, 10, 25} and measure their mean squared prediction errors along length-100 dataset trajectory in puzzle-4xt Figure 2 presents the result, suggesting that longer action chunks indeed substantially mitigate error accumu- lation. Notably, the errors from standard one-step model diverge over time, substantiating the ne- cessity of multi-step prediction for long-horizon tasks.
Model Brrar
Chunk
10
10
100
Rollout Length
Figure 2: Action chunking reduces model errors.
Q: How does the action chunk length affect performance?
A: To answer this question, we train MAC with four action chunk lengths (f1,5, 10, 10, 25)) on one short-horizon and one long-horizon task (cube-double and cube-octuple, respec- tively) used in Section 5.1. As shown in Fig- action chunking with an appropriate chunk size can substantially improve performance on both tasks. Notably, while cube-double can still be partially solved without action chunking, cube-octuple cannot solved at without it. This demonstrates that action chunking is crucial especially on long-horizon tasks. However, Figure 3 also shows that too long action chunks can de- grade performance, mainly due to the difficulty open-loop multi-step future prediction.
cube-double
cube-octuple
Succeas
20
60
0
10
6
10
Action Chunk Length
Action Chunk Lengeh
Figure 3: Action chunk length performance.
Q: How important is flow rejection sampling?
A: Another key feature MAC is its use of flow rejection sampling. To understand the importance of this component, we conduct an ablation study of MAC by using (1) Gaussian ("Gau") action- chunk policy instead of flow policy, and (2) gradient-based based policy extraction (one-step distil-
Table 3: Ablation study of MAC.
Task
MAC (Gas)
MAC (PQL)
XAC
cube-eingle-play--dd lon--dooble-play-d
2
77 sm
D
2
50 212
seeye-play-v0
0
40 .m
100
1uzz1e-3x3-play-10
0
0gg
puzzle 4xd 1-d-----ppaaa v0
0
25
85 114

---

Published as a conference paper at ICLR 2026
lation from FQL (Park al., 2025c)) instead of rejection sampling. We present the ablation results on the default tasks for five reward- reward-based envi- ronments used in Table 3. The results indicate that the use of expressive flow matching is crucial for MAC, and that rejection sampling generally yields better performance on most tasks.
CLOSING REMARKS
In this work, we introduced MAC model-based actor-critic algorithm that combines an action- chunk policy and an action- chunk model. MAC enables generating imaginary autoregressive roll- outs up to 100 steps, achieving the best performance among model-based RI. approaches on chal- lenging, long-horizon tasks.
We now revisit the initial promise of this paper. In Section 1, we motivated offline model-based RL as a promising alternative to offline model-free RL terms horizon scalability. Our answer is variety of long-horizon manipulation tasks, we show that MAC
(at least partially) affirmative: does outperform state-of-the-an model-free RL algorithms (Table 1). However, as discussed in Sec- tion 5.1, even the best model-based RL algorithm (MAC) underperforms tion tasks (e.g., humanoi dmaze), suggesting room for improvement in sequential dynamics model- ing. Moreover, value learning can become challenging when chunk sizes are very large (Figure 3), and rejection sampling may limit performance on low-quality datasets (e.g., random datasets) as the behavioral policy cannot provide useful guidance for policy extraction. We believe that incorporat- ing more advanced modeling techniques and policy extraction techniques could address these limi- tations, which we leave for future work.
contact-rich locomo-
REPRODUCIBILITY STATEMENT
For the reproducibility of work, provide the code of MAC in https://github. com/ kranyoungpark//MG We fully describe the experimental details and hyperparameters to repro- duce the results for our method and baselines in Appendix A.
ACKNOWLEDGEMENT
Kwanyoung Park and Seohong Park are partly supported by the Korea Foundation for Advanced Studies (KFAS). This research used the Savio computational cluster resource provided by the Berke- ley Research Computing program UC Berkeley. This research was partly supported by ONR 100014-22-1-2773 and the National Research Foundation of Korea (NRF) grants funded by the Ko- rean Government (MSIT) (RS-2024-0033363)) RS-2025-25396144 RS-2025-25448259..
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making making?? International Conference on Learning Representations (ICLR), 2023.
Michael S Albergo and Eric Vanden-Fijnden. Building normalizing flows with stochastic inter- polants. In International Conference on Learning Representations (ICLR), 2023.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline re- inforcement learning with diversified q-ensemble. In Neural Information Processing Systems (NeurIPS), 2021.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. In International Con- ference Learning Representations (ICLR), 2021.
Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michel L Littman. Combating the compounding-eroo problem with a multi-step model. arXiv preprint 19X..v/905./3330 2019.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/ 160.006450, 2016.
10

---

Published as a conference paper at ICLR 2026
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep inforcement learning. ArXiv, abs/1912.06680, 2019.
David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. In Neural Information Processing Systems (NeurIPS), 2021.
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In International Conference on Learning Representations (ICLR), 2024.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fin fideliiilty generative behavior modeling. In International Conference Learning Repre- sentations (ICLR), 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems (NeurIPS), 2021.
Jie Cheng, Ruixi Qiao, Gang Xiong, Qinghai Miao, Yingwei Ma, Binhua Li, Yongbin Li, and Yisheng Lv. Scaling offline model-based rl via jointly-optimized world-action model pretraining. In International Conference on Learning Representations (ICLR), 2025.
Rohan Chitnis, Yingchen Xu, Bobak Hashemi, Lucas Lehnert, Urun Dogan, Zheqing Zhu, and Olivier Delalleau. Iql-td-mpe: Implicit q-learning for hierarchical model predictive control. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model. ArXiv, abs/2402.03570, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value expansion for efficient model-free reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In International Conference on Learning Representations (ICLR), 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning, ArXiv, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. In Neural Information Processing Systems (NeurIPS). 2021.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Pciyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Danijar Hafner, Jurgis Pasukonis. Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647-653, 2025.
Philippe HaansnnEEsrrut Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-leanning as an actor-critic method with diffusion policies. ArXiv, abs/2304.115777 2023.
11

---

Published as a conference paper at ICLR 2026
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Neural information processing systems (NeurlPS), 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). ArXiv, abs/ abs/1606 606.08845. 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems (NeurIPS), 2020.
Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, and Jakob Foerster. Policy-guided diffusion. In Reinforcement Learning Conference (RLC), 2024.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Neural Information Processing Systems (NeurIPS), 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Neural Information Processing Systems (NeurIPS), 2021a.
Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence mod- eling problem. In Neural Information Processing Systems (NeurIPS), 2021b. 20211..
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learming (ICML), 2022.
Jihwan Jeong, Xiaoyu Wang, Michael Gimelfarb, Hyunwoo Kim, Scott Sanner, et al. Conservative bayesian model-based value expansion for offine policy optimization. In International Confer- ence on Learning Representations (ICLR), 2023.
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktaschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in a compact latent action space. In International Conference on Learning Representations (ICLR), 2023.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel Model- based offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Iyya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. In International Conference on Learning Representations (ICLR), 2023.
Nathan Lambert, Albert Wilcox, Howard Zhang, Kristofer SJ Pister, and Roberto Calandra. Learn- ing accurate long-term dynamics for model-based reinforcement learning. In 2021 60th IEEE Conference on decision and control (CDC), pp. 2880-2887. IEEE, 2021.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning: State-of-the-ee the-rarr, pp. 45-73. Springer, 2012.
John M Lee. Introduction to Smooth Manifolds. Springer, 2012.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang. L. Y. Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian S. Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. In Neural Information Processing Systems (NeurIPS), 2022.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. ArXiv, abs/2005.01643, 2020.
12

---

Published as a conference paper at ICLR 2026
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint arXiv:2507.0799 00099, 2025.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference Machine Learning (ICML), 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations (ICLR), 2023.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q- Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. ArXiv, abs/2412.06264, 2024.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off- oof-policy estimation. Advances neural information processing systems, 31, 2018.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023.
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. In Neural Information Processing Systems (NeurIPS), 2023a.
Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Oshorne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions Machine Learning Research (TMLR), 2023b.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv, abs/2006.000 0935,. 2020.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning (ICML), 2023.
Kwanyoung Park and Youngwoon Lee. Model-based offline reinforcement learning with lower expectile g-learning. International Conference on Learning Representations (ICLR), 2025.
Scohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Higl: Offline goal- conditioned rl with latent states as actions. In Neural Information Processing Systems (NeurIPS), 2023.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. value learning really the main bottleneck in offline ri? In Neural Information Processing Systems (NeurIPS), 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned In International Conference on Learning Representations (ICLR), 2025a.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv, abs/2506.04168, 2025b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML), 2025c.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantago- weighted regression: Simple and scalable oof-policy reinforcement learning. ArXiv, abs/1910.00177, 2019.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Neural Information Processing Systems (NeurIPS), 2022.
13

---

Published as a conference paper at ICLR 2026
Lu Shi, Joseph J. Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. In Conference on Robot Learning (CoRL), 2022.
Harshit S. Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. International Conference on Learning Rep- resentations (ICLR), 2024.
Anya Sims, Cong Lu, Jakob N Foerster, and Yee W Teh. The edge-of-reach problem in of- fline model-based reinforcement learning. Advances Neural Information Processing Systems (NeurIPS), 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing (ICMI.), 2015.
Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Manfred Otto Heess, and Martin A. Riedmiller. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning (ICML), 2024.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. International Conference on Machine Learning. pp. 13177-33194, PMLR, 2023.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 2(4):160-163, 1991.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transac- fions on Neural Networks, 16:285-2888 286, 2005.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
J Testud, J Richalet, A Rault, and J Papon. Model predictive heuristic control: Applications to industial processes. Automatica, 14(5):413-422) 1978.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Neural Information Processing Systems (NeurIPS), 2017.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Neural Information Processing Systems (NeurIPS), 2017.
Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, ArXiv, abs/1911. 1916,11361,,1,111111111111111111 2019.
Haoran Xu, Li Jiang, Jianxiong Li. Zhuoran Yang, Zhaoran Wang, Victor Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems (NeurIPS), 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-basee policy optimization. In Neural Information Processing Systems (NeurIPS), 2021.
Hanye Zhao, Xiaoshen Han, Zhengbang Zhu, Minghuan Liu, Yong Yu, and Weinan Zhang. Long-horizon rollout via dynamics diffusion for offline reinforcement learning. arXiv preprint arXiv:2405. 19/89, 2024.
14

---

Published as a conference paper at ICLR 2026
Guangyao Zhou, Sivaramakrishnan Swaminathan, Rajkumar Vasudeva Raju, J Swaroop Guntupalli, Wolfgang Lehrach, Joseph Ortiz, Antoine Dedieu, Miguel -reallGr and Kevin Patrick Murphy. Diffusion model predictive control. Transactions Machine Learning Research (TMLR), 2025.
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
We implement MAC on top of the codebase Park et al. (2025b). Each experiment takes approxi- mately 2 days for large-scale benchmarks, and around 3 hours for single-task benchmarks on a sin- gle A5000 GPU. Please refer to Appendix A.2 for detailed time measures.
A.1 IMPLEMENTATION DETAILS
Network architectures. We follow the setup of the work by Park et al. (2025c;b), using 4-layer MLPs with layer normalization (Ba et al., 2016) for all neural networks (the policy, critic, dynamics model, and reward model). For large-scale benchmarks, we parameterize the reward model and the terminal model using single success prediction network (st, a.), where termination is calculated as of s, a,) > 0.5) and reward calculated as (f a,) > 0.5) - 1. For reward-baar based tasks, reward model rip (st, az) without termination.
Accelerating rejection sampling during training. To improve training time, we use different numbers of samples for rejection sampling during training and evaluation (which call Nurain and Noest Specifically, we use Nurain = during training (except in puzzle-4x5, where a larger Ntrain = 32 was necessary due the BC policy branching over 20 possible actions) and Nrest = 32 at test time.
Implementation details for the compared methods. We implement MOPO, MOBILE, and LEQ in our codebase. For MOPO, epistemic uncertainty is estimated as the maximum standard deviation across ensemble members (Yu et al., 2020). For L.E, we omit dataset expansion, which we found to have negligible impact in our benchmarks. We use 5 dynamics model ensembles for all methods and disable early stopping and validation filtering when training the model, as we found they are un- reliable large-scale datasets (training and validation metrics are nearly identical in these settings).
For D-MPC (Zhou et al., 2025), we implement the flow variant of D-MPC (F-MPC) our code- base. Specifically, we train a flow BC policy x<4eatn-1 8)) and flow dynamics model Pv (o(Stt1:tn St, 41:++n-1) instead of using diffusion models. For reward-based benchmarks, we calculate the return-to-go G; Et- Fx without discounts, in the original paper. For goal- conditioned (large-scale) benchmarks, we similarly define the return-to-go without discounts for goal-conditiones tasks as G, 1(g E {st,... ST) Unlike the original architecture, do not use history conditioning and transformers (as all tasks are Markovian) and use the same MLP archi- tecture other methods for fair comparison.
For sequence modeling approaches (Diffuser and HD-DA), we follow the official implementation for D4RL's maze2d environment (Fu et al., 2020), and adjust the maximum length of the trajectory generation and the number diffusion steps (of the high-level policy for HD-DA) to be the maxi- g.. H 4000 for humanoi dmaze-giant) We re-plan the trajec-
mum length of the environment tory every 100 steps, as we found that this is necessary to achieve a non-zero performance on long- horizon tasks, unlike in the maze2d benchmark.
For other model-free methods, we use the implementations by Park et al. (2025b) and Park et al. (2025a). We also take the results from these papers for the corresponding methods.
Implementation details for ablation experiments. For the ablation study on the action-chunk length, we fix the horizon length to 10 and only change the action- length E [1,5, 10, 25). For MAC (Gau) of the ablation study on flow rejection sampling, we parameterize the action-chunk policy with a = (anh( where T4 N (8)( (x))).
Figure 4: OGBench tasks.
16

---

Published as a conference paper at ICLR 2026
A.2 TRAINING TIME
We report the average training time and inference time for single-task and multi-task experiments in A5000 for MAC and prior MBRL methods in the table below. MAC trains in around 3 hours for single task and 55 hours for multi-task experiments, which is 1.2 2.2 times longer than other methods. Inference speed MAC similar 1.5 times longer than other methods. All models use identical architecture sizes across methods.
Table 4: Training time of MAC and prior MBRI. methods.
Training time (hours)
MOPO
MOBILE
FMPC
LEQ
(H=5)
MAC
Single-task
1.4
2.6
1.7
1.6
3.1
Multi-task
25.1
36.7
28.4
25.2
55.5
Table 5: Inference time of MAC and prior MBRL methods.
Inference time (ms)
MOPO
MOBILE
FMPC
LEQ
MAC
Single-task
1.8
2.3
1.6
2.5
Multi-task
5.2
7.3
5.2
7.2
A.3 HYPERPARAMETERS
Shared hy hyperparameters. Here, we report shared hyperparameters for MAC, MOPO, MOBILE, and all model-free baselines. The hyperparameters for goal-conditioned tasks are presented in Ta- ble 6, and those for reward-based tasks are in Table 7. We note that these hyperparameter con- figurations mostly follow those of SHARSA (Park et al., 2025b) for multi-task experiments, and FQL (Park et al., 2025c) for single-task experiments.
Table 6: Shared hyperparameters for large-scale benchmark tasks.
Hyperparameters
Value
Gradient steps
Learning rate
3 x 10
Optimizer
Adam (Kingma & Ba, 2015)
Batch size
1024
MI. size
[1021, 1024, 1024, 1024]
Actor (peur Prrom Preej prand) ratio
(0, 1, 0, (cube)
(0, 0.5.5, 0.5) (puzzle)
(0,0, 1, 0) (humanoi dmaze)
Value (pear Pecom- Peraj Prand) ratio
D
(0.2, 0, 0.5,
Table 7: Shared hyperparameters reward-based benchmark tasks.
Hyperparameters
Value
Gradient steps
1M
Learning rate
X 10-4
Optimizer
Adam (Kingma Ba, 2015)
Batch size
256
MT.P size
512,,512,52,,13
MAC hyperparameters. We report the hyperparameters for method in Table 8. Note that MAC uses the same (n, H, Nurain Nisst) = (10, 10, 8, 32) across all tasks, except for puzzle-4x65 where using Nerin = 32 during training important the BC policy has 20 possible branches.
17

---

Published as a conference paper at ICLR 2026
Accordingly, only in muzzle-4xte we decrease the hidden dimensionality of the networks to 256 to compensate for the increased training time from Nirain 32.
Table 8: Hyperparameters MAC.
Hyperparameters
Value
Learning rate
3 10
Optimizer
Adam
Nonlincarity
GELU (Hendrycks Gimpel, 2016)
Layer normalization
Truc
Target network update rate
0.005
Discount factor 7
0.999
Flow steps
10
Nirain
8 (default), 32 (puzzle-4x5)
Noest
32
Rollout length H
10
Action-chumi size
10
Hyperparameters for baselines. We report the optimal hyperparameters of all baselines for goal- conditioned experiments in Table 9 and reward-baas -base experiments Table 11.
For MOPO and MOBILE, we perform hyperparameter sweep over rollout lengths H E {1, 10) and penalty coefficients 3 {0.1, 0.5, 1.0, 2.0, 3.0, 5.0), where denotes the model rollout horizon and [3 is the penalization coefficient for model uncertainty or Bellman inconsistency. respectively. We note that reducing the MBPO loop's model batch ratio f from 0.95 for training on long-horizon tasks, as also noted by Park & Lee (2025).
(0.5, 0.25] is crucial
For LEQ, we search over rollout lengths E {1, 5, 10) and expectiles r E (0.1, 0.5), where the expectile controls the degree of conservatism for critic and policy learning.
For model-free methods large-scale benchmarks, we follow the list hyperparameters to search over in the work by Park et al. (2025b). For SHARSA, we searched over E {25,50). For n- SAC+BC, we search over n E 10, 25, 50 and regularization coefficients a E f0.01, 0.03, 0.1, 0.3). For GCIQL, we follow (Park et al., 2024) and extract policies with DDPG+BC, searching over a E (0.003, 0.01, 0.03, 0.1, 0.3, 3.0).
We denote "N/A" in the tables if method achieves performance across all hyperparameters tested in our sweep. If not specified, all other hyperparameters follow the defaults provided in the original papers.
Table 9: Hyperparameters for baselines for large-scale benchmark tasks.
Enviromment
MOPO (H, 88
MOBILE (H,
LEQ (H,
cube-double-play-vv0
(10, 1.0,0.25) N/A
(5.0.5.0.5)
N/A
cube-octuule-pllay-v0
N/A
N/A
(1, (1,0.5, 0.5, 0.5) N/A
(1,10,,0 0.5)
N/A
N/A
N/A
puzzle-3x3-ppay--dd
(5.5.0,0.0 0.5) N/A
(10. 3.0, 0.25)
(1,0.1)
puzale-4x6-play-ro
N/A
(1,0.1)
Table 10: Hyperparameters for baselines for reward-based benchmark tasks.
Environment
MOPD f)
MOBILE (H, 8.
LEQ (H,
cube-single-play-ve
(10, 2,0, 0.25)
(10, 5.0, 0.25)
N/A
cube-dubble-play-vv
(10, 1.0, 0.25)
N/A
N/A
acene-play-vo
(10, 2.0. 0.25)
N/A
N/A
puzzle-3x3-play-v0
N/A
N/A
N/A
puzzle-4x4-play-00
N/A
N/A
18

---

Published as a conference paper at ICLR 2026
Hyperparameters for ablation studies. We report the optimal hyperparameters for the ablated variants of MAC: one that replaces the flow policy with a Gaussian policy ("Gau"), and another that replaces rejection sampling with FQL's one-step distillation ("FQL"). For the Gaussian policy variant, we reuse the same hyperparameters as our main method. For the FQL. variant, we search over the behavior cloning coefficients (0.1, 0.3, 1.0,
Table 11: Hyperparameters ablation experiments.
Environment
MAC (FQL) (a))
cube-single-plaayvvo
1.0
cube-double-play-
0.3
scene-play-v0
1.0
puz2le-3x3-play-v0
1.0
puzzle-4x4-play-vo
1.0
19

---

Published as a conference paper at ICLR 2026
B COMPLETE NUMERICAL RESULTS
For completeness, we provide the full per-task results for large-scale, long-horizon environments and reward-based environments in Table 12 and Table 13 (corresponding Table and Table 2). The results are averaged over seeds and we report the standard deviations for each tasks. We highlight the numbers that are above or equal to 95% of the best performance.
Table 12: Complete results large-scale experiments.
Kodel- Pree
Seq. Kodeling
Model-3ane
c4
task2
95 A4
100 aa
100 -0
0.50
BE +2:
75 4:3
0.
22 =11
87
task3
0.1o
90
100
0
0.is
0.
0.10
18
oraclerep-v0
task!
0.
82
0.1>
0.00
5
0.
taski
0.
58 44
43 -11
0.
0
0-4a
0
0-1
task2
10
07 19
0
0
0
bunanoi dnaze- naviggat- aneg-vD
tack3
5
0.
task5
10
overall
82
:6
0.
20
0.2x
0.
20
0.00
0.2
taski
100
67
100
6.
62 a30
50 asa
0
73
100 29
task2
100
13
100
0.2
100
cube-double waabbblepplyy-ocleerrpppp
task5
38
20 mm
100.0 20
0:1
1:
asa
s0
100
overall
74
32
95
1
2
15
0
100
task!
0.n
0.
0:s
83
ciberocttglleeplly-rrcccc wrapp--
tankd
0.
0.e
0.2
0.m
0ax
0.ax
0.
0.0
5:s
task5
0g
0an
0gg
gn
0a
0.gh
0un
3
overall
0.
19
0.
Das
0.s
0.so
0
30
taaki
100
100
gs
23
100
0:n
0.
0.a
0
100
task4
s0
92
100
0.g
0
2
100 =3
task5
93
96
100
0.
Das
0.aa
0
7
100
overall
100-0
0:m
0:
0.
0:s
0mn
100
puzzle-tzb Az5-pa Sasee-tt---lly---- play-oraclerep--00
task3
0:
0:s
0.
0
0
0.e
100 =3
taski
0.
8
0
G
0.4a
0
0
100
task6
20

---

Published as a conference paper at ICLR 2026
Table 13: Complete results for reward-based experiments.
Pavironnant:
Taak
IQL.
RaBRAC
TDQr.
FQT.
MOPO
YOBTLE
1.F0]
FMFC
MAC
tank1
88
89
95
12:13
85
0.0
10
100
tasle2
taak?
91 2a
93
98 21
98
15
83 gm
0
10
98 ct
92.
13
taak5
78
87 219
0
3
97
everall
83 19
91
95
96
12 :4
81
0.
95
99
task!
taek2
7.33
16 2:0
0:5
50 =t
tack]
0.
4.
17
22
2 13
0a
56 s
28
task5
4
1:11
19
2:13
O
0an
everall
12 137
15 n.07
29
1:
0.
3
53 :4
taski
37
taek?
12 a3
am
33 2:4
2:23
3
100
sona-ppaay-singllttattv-g
task3
32 an
65 21
94
98
0.s
0
0
Dan
96 96
tasks
s0
0
2.a
93
overall
28
41 axr
46
56
8
4
97
tasic2
1:49
16
Ousa
task3
10
0.s
0
Dann
tasks
6:43
16 =a
Olsa
ma
2
overall
21 334
10 am
30
20
12
10
20
task?
7
12:14
14 26
16
0.2
=
0
0
33
tack3
34
0
100
D
task5
4-1
7.2
24 am
T
0.s
0
0
Dan
72
overall
7.
14
29
0.s
0.
0.
76
21

---

Published as a conference paper at ICLR 2026
C D4RL RESULTS
We report the scores for the D4RL (Fu et al., 2020) environments, which has been used as a standard dataset for offline RL evaluation. Same OGBench experiments, we report the normalized score across seeds and report the standard deviation for each tasks. The results for prior works are reported following their respective papers. MOPO is improved version of MOPO, introduced in Sun et al. (2023). We highlight the numbers that are above or equal to 95% of the best performance.
For sequence modeling methods, we consider TT (Janner al., 2021a), which predicts offline tra- jectory with Transformer model (Vaswani et al., 2017) and find the best trajectory by conditioning with target return-to-go, and TAP (Jiang et al., 2023), which improves TT by quantizing the action space with VQ-VAE (van den Oord et al., 2017).
Table 14: DARL MuJoCo Gym results.
Model-free free
Seq modeling
Model-baaed
Dataset
CQL
ReBRAC
MOPO
MOBILE
LEQ
hopper-r
32
28 =
hopper-m
66
103
hopper-mr
86
98
94
87
99
104
103
96
hopper-me
96
107
91
110
105
81
111
109
110 #1
walker2d-r
18
5
5
7
17
21
5
walker2d-n
78
81
87
82 1.
salker2d-nr
77
73
92
86 +
walker2d-ne
108
halfcheetah-r
30
11
38
32
30
12
halfcheetah-n
46
66
47
46
45
73
74
71
47 +1
halfcheetah-me
45
51
44
44
40
72
66
65
38 1:
halfcheetah-me
101
86
95
90
105
102
68 1.
Total
740
852
717
747
844
960
923
771
22

---

Published as a conference paper at ICLR 2026
D TRAINING CURVES
We provide the training curve of MAC for large-scale, long-horizon environments and reward-based environment in Figure 5 and Figure (corresponding Table and Table 2). We plot the mean and the standard deviation (across 4 seeds) by covering [mean- - std, mean + std] area with a lighter color.
cube-double-pl.
cube-octuple-pplay
hunano1 dmaze--gi gtaaaz---gaatttoaiggate
100
100
100
80
80
80-
60
60-
60
Soccess
40
40-
40
20
20
20
0
0.0
0.5
1.5
0.5
1.0
20
0.0
15
Trashing Stops (x101)
Training stopo (810f)
Training asoga (XLIP)
hunanoi dar.d
puzzle-3x3-playy
puzzle-4x6-pp.ay
Suncees (3)
100
100
20
20-
0
0.0
1.5 Trosning stope (X11P)
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Training stopo (N10f)
Trasning 0oope (XLIP)
Figure 5: Training curve of MAC in large-scale, long-horizon environments. We report the success rate for
15 evaluation episodes across seeds (total 60 episodes). Shaded region represents the [mean std, mean + std].
play-
play
play (tazbs)
play
(-09)
cobe- zingle play of (exect)
cribe <sing play-10 10 (taee))
c1hbe singln play-1o 1o (taets)
cotee single play 30 (corb))
othe stngle play vn (tarks)
110-
100.1
1184
2314
101
10-
30-
0.0
25
14
3.0
05
0.1
655
101
0.
10
0.]
3.
Trassing stege (x3P)
Crassing stogs (412)
Trassnn stops (x17))
Trassnng stapa (40r)
Trassng scape
prazzle-2aa-ppa po (saak))
pezzle 3x0 p (caak2)
puzcle Sx8-peer-000 (caak))
puzzle Sebeleerbr-l (eaalt)
pozzle-asS-plee (sua15)
20
20
20
Pizzzle 4454- play-r0 (taak1)
Fuzzle 4od play (tark))
puzrle 4o0 piay (cank))
purzle Got si (tuzlc))
pazzie daz playprp (tane))
110
100
05
05
01
Trarinn 10
(110)
Theminng strpe (vo))
KORDA play (tadee)
ZOSDA play 10 (taee))
ncace play 70 (task))
Ec20e play-10 (tatbo)
sc00s play (tanld)
12-
12-
40
-31
41
20
20-
31-
21
01
Figure 6: Training curve of MAC in reward-based environments. We report the success rate for 15 evalu- ation episodes across 4 seeds (total 60 episodes). Shaded region represents the [mean- std, mean + std].
23

---

Published as a conference paper at ICLR 2026
randoz-Y2
hopper mediun-v2
hepper-meding reppp---odu--rrplayy-
hopper aedi expert-v2
Scrnsteoed poore
100+
100
50
3-36
0-6
03
Trassiog steps (x(0))
Training stopa (xIF))
Training staps (x11f)
Trainior steps (XIPP)
wwT-ee2dd-rrdde 2d
val 2deedd-eededdee re- rew.rrelllayee lay-v2
vnl valker2ddmeed wws- 1m-expert--22
Domattan accre
100
100-
100
50
50
53-
50-
0.0
0.5
1.0
01.0
0.5
1.0
0.0
0.5
10
0.3
0.5
2.0
(XII))
(XI2f)
halfchee rcheetab-randooee
halfchheatal--nstur--22
haIfcheetta-ssedi---ppppt
Tratstin etage (x110)
Tratstin staps (x(if)
Tratning staps (x1(f)
Trateing etape (x19P)
Figure 7: Training curve of MAC in D4RL environments. We report the success rate for 15 evaluation episodes across seeds (total episodes). Shaded region represents the [mean- std, mean + std].
24

---

Published as a conference paper at ICLR 2026
E MORE ABLATION EXPERIMENTS
Q: Is distillation with To necessary?
A: To understand the importance of this com- ponent, we conduct an ablation study of MAC removing the distillation. Specifically, we di- rectly train the one-step flow model Tw with flow matching BC loss, instead of distilling the multi- step flow model sults on the default tasks for five reward-based en-
Table 15: Ablation of using no.
Task
MAC (w/o m²)
MAC
cubereing inbeeringge-pplymee
100
uube-dauhla-play--
scene scene-play--e
5:s
100
0.
We present the ablation re-
puzzle 4x4 pvay
15:24
134
vironments used in Table 15, The results indicate that the use of Ta is crucial for MAC in OGBench tasks where behavioral policies are highly multi-modal.
Q: How does model error correlate with the performance?
A: Figure 8 shows the policy performance and rollout error with respect to the rollout length for various action chunk sizes. For chunk sizes of 1, 5, and 10, we observe consistent trend that at a given rollout length, smaller chunks produce larger model-predictee respondingly lower policy performance (clearly shown in rollout length of 50 and 100). How- ever, excessively large chunk sizes (25) breaks this trend, where it achieves lower rollout error, but does not yield better performance. It is because while larger chunks helps reducing the compound- ing model eπror, they also make both policy learning and Q-function estimation harder since the ac- tion space grows exponentially. While flow rejection sampling mitigates this problem by limiting the action sequence to in n--distriuttio actions, extremely large chunk sizes exacerbate this problem, hurting performance despite improved model accuracy.
1.0
10-
PoT2 Parformatce
Bollont Lengez
Rollout
errors and cor-
Figure 8:
Model error correlate with the perfor-
mance, unless action chunk size is too large.
25


Reviews:
[{"review_id": "1Msqa4P8O1", "reviewer": "Reviewer_9Rxo", "summary": "The paper introduces Model-Based RL with Action Chunks (MAC), an offline RL algorithm designed to address two critical challenges: compounding errors in model-based rollouts and action selection that leads to out-of-distribution (OOD) model exploitation.", "questions": "[Q1] Following Weakness 1: How does the simpler reward- or goal-conditioned BC policy trained on the same flow-based architecture compare to the proposed rejection sampling plus value function approach? This comparison will help reveal the true advantage of value-based return maximization over the BC baseline policy.\n\n[Q2] Given that performance peaks at an intermediate chunk length ($n=10$) and degrades with larger sizes ($n=25$), could the method be improved on large $n$ by using a mixture of chunk sizes, similar to the idea presented in [(1)](https://arxiv.org/pdf/2412.11253)? This might allow the agent to balance the low compounding error of large chunks with the more effective sampling of smaller chunks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] MAC's action-chunk model, combined with in-distribution rejection sampling, offers a promising solution to the problems of compounding model error (a key issue in MBRL) and OOD model exploitation (a key issue in offline RL). It has achieved excellent empirical performance in long-horizon and large-scale benchmark tests.", "weaknesses": "[W1] The main problem lies in the scalability of sampling with increasing action chunk length $n$. As $n$ increases, the dimensionality of the action chunk space also increases, making it exponentially more difficult to find high-value, in-distribution action sequences by rejection sampling. This sampling challenge may weaken the \"return maximization\" aspect of RL and may cause the agent's final behavior to be closer to behavior cloning (BC) rather than RL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZuQEcZEvnc", "reviewer": "Reviewer_Qjkz", "summary": "This paper introduces MAC, an offline MBRL algorithm designed particularly for long-horizon tasks. They use action-chunking to reduce the number of recursive calls to the dynamics model to hopefully reduce compounding error. They also use flow-based rejection sampling to aim to prevent model exploitation.", "questions": "**Q1.** What do the authors mean by “Thanks to the expressivity of the flow BC policy, we query the model only with in-distribution\naction-chunk samples”? Relatedly, isn't it easier to be OOD since \"action\" space is exponentially larger in the number of action steps included as input to the model?\n\n**Q2.** The baselines are a mixture of results from the prior papers and from the author's reimplementations. Are the authors confident they understood implementation details that might have unfairly weakened results from prior paper, and also that they correctly implemented and fairly tuned their reimplemented baselines?\n\n**Q3.**  Why was flow matching chosen over diffusion? Did the authors try/consider diffusion as an alternative?\n\n**Q4.** The method involves training 6 models at the same time (?). What was this like in terms of convergence and stability? It would be interesting to include training curves.\n\n**Q5.** Related to ^, often you would train the dynamics model p_\\phi, r_\\phi to convergence first, and then train the policy. Why train them side by side?\n\n**Q6.** Is \\pi_\\theta really needed if you then replace it with a less expressive model. For completeness, the authors should include and ablation bypassing \\pi_\\theta to show whether it is important to their method.\n\n**Q7.** (Comment) “one-step” is confusing since \"steps\" are also used to refer to RL environment steps. Is there another term you can use?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The issue they address is relevant, the results seem strong, and the paper is reasonably clear in explaining a complicated method.", "weaknesses": "**W1.** There are so many moving parts, including training 6 models. This is expensive, and probably also more difficult to implement and debug compared to many other methods.\n\n**W2.** The main benchmark in offline RL is probably D4RL. The authors have not provided results on this.\n\n**W3.** As far as I understand the proposed method is roughly N times more expensive than prior algorithms(?) Furthermore, N is high (~32), and they potentially need larger models since action-chunk inputs to the model are n times larger. The authors should include a comparison and discussion of the runtime to help justify this significant extra cost.\n\n**W4.** Performance appears highly sensitive to the action-chunk length hyperparameter. This could make it difficult to tune, especially offline.\n\nMore weaknesses: See Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z49WvU9ZMx", "reviewer": "Reviewer_ZGYu", "summary": "Model-based offline reinforcement learning (RL) approaches often face a key dilemma: model rollouts must be sufficiently long for the value function to capture accurate long-horizon information, yet they suffer from error accumulation as the rollout horizon increases. To address this, the paper employs action chunks, a common technique in robotics that predicts a sequence of actions rather than a single step. This reduces the number of function calls needed to generate future trajectories, enabling longer horizons while simultaneously mitigating error accumulation. Once the value functions are obtained, the policy is implicitly extracted via rejection sampling from an expressive behavior policy modeled by a distilled one-step flow matching policy.  Experiments on OGBench and D4RL demonstrate the superior performance of the proposed method, MAC (Model-based Offline RL with Action Chunks), in robotic manipulation tasks. Nonetheless, the method falls short in locomotion tasks that require precise control, where approaches based on differentiable reward maximization or policy gradients may be more effective. Moreover, all experiments are conducted with low-dimensional state inputs, without evaluation on image-based tasks.", "questions": "1. I understand that rejection sampling from a behavior policy implicitly imposes a behavior constraint. However, I still wonder whether additional penalties are needed to mitigate value overestimation. Can simply applying best-of-N sampling alone account for the scale of performance improvements reported?\n\nAt this point, I have no further questions. Please see my comments under Weaknesses for more details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Scale up RL methods to tackle long-horizon problems is very important.\n2. The idea of adopting action chunks to address the dilemma between long horizons and small functional calls is quite straightforward.\n3. The proposed method (MAC) is quite simple to be implemented. So, this paper offers a good starting point for future research on this direction.\n4. The empirical performance on robotics manipulation tasks in OGBench are also strong, offering potential solutions for robotics.", "weaknesses": "I have a few concerns regarding the novelty and the complexity of the proposed method:\n\n1. `Novelty of Action Chunks` The idea of adopting action chunks to extend RL horizons is not particularly novel. In fact, a recent work [1] has already explored a model-free version of this idea. Therefore, the novelty contribution of this paper is somewhat limited. However, since action chunking appears to be a principled and simple approach to addressing the dilemma highlighted by the authors, I would not consider the lack of novelty a major concern.\n\n2. `Method Complexity` While the high-level idea of MAC is straightforward and the implementation relatively simple and robust to hyperparameter variations, the method introduces multiple components, including a flow policy, a distilled flow policy, a reward function, a state value function, a state-action value function, and a dynamics model. This design may help address the horizon-scaling issue, but it also adds significant complexity, potentially restricting MAC to smaller model sizes. In the long run, a unified model or framework that consolidates these components would be preferable. Nonetheless, given that the primary contribution of MAC lies in leveraging action chunks to scale up model-based RL horizons, I would also not consider this limitation a major concern.\n\n[1] Reinforcement Learning with Action Chunking, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uOYYi6PlUq", "reviewer": "Reviewer_P1Zz", "summary": "The paper introduces Model-Based Reinforcement Learning with Action Chunks (MAC), a scalable offline model-based RL framework designed to handle long-horizon tasks. The key idea is to mitigate the trade-off between long-horizon bias reduction and model error accumulation in model-based value expansion by using action-chunk models—multi-step dynamics that predict future states from sequences of actions rather than single actions. Complementarily, an action-chunk policy is trained via flow matching and rejection sampling to model complex, high-dimensional action distributions while avoiding out-of-distribution exploitation. Experiments across large-scale datasets (100M transitions) and standard offline RL benchmarks show that MAC achieves state-of-the-art results among model-based methods and competitive performance compared to model-free baselines, particularly in complex, long-horizon manipulation tasks (Tables 1–2; Sec. 5.1–5.2).", "questions": "- Could the authors include a performance comparison with ADMPO? Since both methods apply multi-step dynamics models to offline RL, such a comparison would better highlight whether MAC offers advantages in terms of methodological design.\n- Could the authors provide experimental results on D4RL? This is essential for evaluating the performance of an offline RL algorithm.\n- The experiments in the paper show that the multi-step dynamics model effectively reduces rollout errors, and in particular, the larger the chunk size, the more significant the reduction—this aligns well with intuition. However, it remains unclear whether MAC’s performance gain truly comes from the reduction in rollout error, or merely from the flow action-chunk policy. Although the paper provides results for different chunk sizes, this does not fully address the issue. Could the authors provide additional experiments comparing MAC’s performance across different chunk sizes and rollout lengths, explicitly reporting the rollout error and policy performance for each configuration? This would help verify whether smaller rollout errors indeed correlate with better policy performance.\n- From the experiments in the paper, when the action-chunk length is set to 25 the rollout error is still effectively reduced, but the algorithm’s performance degrades. Could the authors provide a deeper explanation for this?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper focuses on model-based value expansion in the offline setting, aiming to balance value bias reduction and model error reduction. This is a highly valuable and meaningful problem, as it contributes to improving the performance of offline reinforcement learning on long-horizon tasks.\n- The method proposed in the paper mainly adopts a multi-step dynamics model (action-chunk model) to reduce rollout errors and uses rejection sampling to model complex, high-dimensional action distributions. Such an approach is intuitively effective, and the experiments show that MAC achieves outstanding performance on long-horizon offline RL tasks.", "weaknesses": "- The multi-step dynamics model (action-chunk model), which takes s_t and a multi-step action sequence (a_t, a_{t+1}, …, a_{t+k-1}) as input to predict s_{t+k}, has already been proposed in previous works [1][2][3]. The multi-step dynamics model used in MAC does not differ from these earlier approaches. Similarly, the flow policy [4], model-based value expansion [5] are also not introduced for the first time in this paper. Therefore, I believe the contribution of this work is relatively limited.\n- The application of multi-step dynamics models to the offline RL domain has been explored in ADMPO [6]. Similar to MAC, ADMPO investigates how multi-step dynamics models can reduce rollout errors in the offline setting. The key difference is that ADMPO adopts a more traditional policy learning approach, representing another way of leveraging multi-step dynamics models in offline settings. I believe ADMPO is highly related to MAC, as ADMPO has already demonstrated that multi-step dynamics models can significantly improve policy optimization performance in offline RL. Therefore, MAC should be compared against ADMPO.\n- Although this paper targets long-horizon offline tasks, it is inappropriate for an offline RL algorithm to lack experiments on the standard D4RL benchmark.\n\n\n[1] Kavosh Asadi, Evan Cater, Dipendra Misra, and Michael L. Littman. Towards a simple approach to\nmulti-step model-based reinforcement learning. CoRR, abs/1811.00128, 2018.  \n[2] Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michael L. Littman. Combating the\ncompounding-error problem with a multi-step model. CoRR, abs/1905.13320, 2019.  \n[3] Tong Che, Yuchen Lu, George Tucker, Surya Bhupatiraju, Shane Gu, Sergey Levine, and Yoshua\nBengio. Combining model-based and model-free RL via multi-step control variates. https://\nopenreview.net, 2018. URL https://openreview.net/forum?id=HkPCrEZ0Z.  \n[4] Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on\nMachine Learning (ICML), 2025c.  \n[5] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E.\nGonzalez, and Sergey Levine, ‘Model-based value estimation for ef-\nficient model-free reinforcement learning’, CoRR, abs/1803.00101,\n(2018).  \n[6] Haoxin Lin, Yu-Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing Jia, Junyin Ye, Jiaji\nZhang, and Yang Yu. Any-step dynamics model improves future predictions for online and\noffline reinforcement learning. In ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)