Decoupled Q-Chunking
Authors: Qiyang Li
Seohong Park
Sergey Levine
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
action chunking
offline RL
Primary area: reinforcement learning

Abstract: Bootstrapping bias problem is a long-standing challenge in temporal-difference (TD) methods in off-policy reinforcement learning (RL). Multi-step return backups can alleviate this issue but require delicate importance sampling to correct their off-policy bias. Recent work has proposed to use chunked critics, which estimate the value of short action sequences ("chunks") rather than individual actions, enabling unbiased multi-step backup. However, extracting policies from chunked critics is challenging: policies must output the entire action chunk open-loop, which can be sub-optimal in environments that require policy reactivity and also challenging to model especially when the chunk length grows. Our key insight is to decouple the chunk length of the critic from that of the policy, allowing the policy to operate over shorter action chunks. We propose a novel algorithm that achieves this by optimizing the policy against a distilled critic for partial action chunks, constructed by optimistically backing up from the original chunked critic to approximate the maximum value achievable when a partial action chunk is extended to a complete one. This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning policies over long action chunks. We evaluate our method on challenging, long-horizon offline goal-conditioned benchmarks and show that it reliably outperforms prior methods.

Published as a conference paper at ICLR 2026
DECOUPLED Q-CHUNKING
Qiyang Li UC Berkeley gcli@berkeley.c edu
Seohong Park UC Berkeley
Sergey Levine UC Berkeley svlevine@berkeley. edu
seohong0berkeley.com
ABSTRACT
Temporal-differee difference (TD) methods learn state and action values efficiently by bootstrapping from their own future value predictions, but such self- self-bootstrapping mechanism is prone to bootstrupping bias, where the errors in the value targets accumulate across steps and result in biased value estimates. Recent work has proposed to use chunked critics, which estimate the value short action sequences ("chunks") rather than individual actions, speeding up value backup. However, extracting policies from chunked critics is challenging: policies must output the entire action chunk open-loop, which be sub-optimal for environments that require policy reactivity and also challenging to model especially when the chunk length grows. Our key insight that of the policy, allowing the policy to operate over shorter action chunks. We propose a novel algorithm that achieves this by optimizing the policy against a distilled critic for partial action chunks, constructed by optimistically backing up from the original chunked critic to approximate the maximum value achievable when partial action chunk is extended to a complete one. This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning action chunking policies for long action chunks. We evaluate our method on challenging, long-hone horizon offline goal- conditioned tasks and show that it reliably outperforms prior methods.
decouple the chunk length of the critic from
Code: github. com/ Col aaniiiangi//dee
large critic chunk
Actions
Aggregated Score
V(O)
100
82
baclup
fall chunk
80
Q
Decoupled
distill
States Time
60
Chunking
44
Q(O,
25
20
Actions
partial chunk
small policy chunk
DQC
QC
NS
SHARSA HIQL
Figure 1: Decoupled Q-chunking (DQC). Left: The key idea of our method is 'decouple' the action chunk size of the critic Q from that of the policy æ. A large critic chunk size allows for efficient value learning while small policy chunk size makes policy learning more tractable and allows for better policy reactivity. Right: Our method outperforms all baselines on six hardest environments on OGBench, an offline goal-conditioned RL benchmark with challenging long-horizon tasks.
INTRODUCTION
Temporal-diffee differfffeeene (TD) methods are powerful reinforcement learning (RL) techniques that can directly learn from off- off-colicy prior data without requiring explicit dynamics model, making them well-suited for offline RL (Levine et al., 2020) and sample-efficient online RL (Chen et al., 2021). Despite their successes, key challenge remains: bootstrapping bias (Jaakkola et al., 1993; Sutton

---

Published as a conference paper at ICLR 2026
et al., 1998; De Asis et al., 2018; Park et al., 2025b). This bias stems from the core design of TD updates, where the value at the current state is learned by regressing towards the learner's 's own predictions at the next time step. As a result, any prediction error is compounded backward across steps, making learning particularly challenging in long-horizon, sparse-reward tasks.
Multi-step return backups (Sutton et al., 1998) can alleviate bootstrapping bias by shifting the regression target further into the future and effectively reducing the time horizon. However, naively applying them introduces additional biases because computing the target involves summing rewards along off-policy trajectories that may deviate from the actions that the agent would take. Although importance sampling can in principle correct such off off-policy biases by reweighting the off-policy trajectories (Munos et al., 2016), it often suffers from high variance and thus requires truncation and other heuristics for numerical stability, making it difficult to tune in practice. Recent works (Seo & Abbeel, 2025; Li et al., 2025a; Tian et al., 2025; Li et al., 2025b) leverage chunked value functions, which estimate the value of short action sequences ("chunks"" rather than a single action. This formulation allows n-step return backup without the pessimistic bias (under some condition we formalize in Section 4). However, theoretical guarantees of action chunking Q-learning, especially on arbitrary off- off-policy data, are still an open problem existing analysis (e.g., in Li et al. (2025b)) only considers the case where the data is collected by an action chunking policy. Moreover, on the empirical side, directly optimizing policy over full action chunks is difficult, particularly as the chunk size grows, and it is still unclear how to best extract a policy from a chunked critic.
In this work, lay the theoretical foundation of action chunking Q-learning where we identify the key open-loop consistency condition (Definition 2) under which Q-learning with action chunking critic is guaranteed to produce near-optimal action chunking policy. On top of it, we characterize the condition when closed-loop execution (i.e., only executing the first action in the predicted action chunk) of such action chunking policy is expected be even close to the optimal closed-loop policy. Motivated by our analysis, we develop a simple practical algorithm that builds on top of the idea of closed-loop execution of action chunking policies to address the action chunking policy learning challenge. The key insight is that we can avoid training the policy to predict the full action chunks and instead to only predict shorter, partial action chunks against the chunked critic. To achieve this, we use 'distilled' chunked critic with chunk size that matches the policy: it optimistically regresses to the original chunked critic to approximate the maximum value that the partial action chunk can achieve after being extended into full action chunk. Conceptually, while the action optimization is still done for the longer, complete action chunks, the policy network is only trained to output the partial action chunk of an optimized complete action chunk. This way, the policy only needs to predict a much shorter action chunk (e.g., in the extreme case, only one action), which often admits a much simpler distribution, while enjoying the value learning benefits from the use of chunked critics.
Our main contributions are two-fold. On the theoretical side, we provide the first formal analysis of Q-learning with action chunking, focusing on characterizing the value learning bias of the bellman backup of action chunking critic. Specifically, we introduce the open-loop consistency condition under which we exactly characterize the worst-case value estimation bias (Theorems 1 and 4) and sub- optimality gap (Theorems 2 and 5) at the fixed point of the bellman optimality equations. Moreover, we characterize (i) the conditions under which action chunking critic backup is preferable over ni-tep return backup with a single-step critic (Proposition 3), and (ii) the conditions under which closed- loop execution of the action chunking policy further mitigates the open-loop bias (Theorem 3). On the empirical side, we propose a new technique, Decoupled Q-chunking (DQC), that addresses the policy learning challenge in action chunking Q-learning by decoupling the policy chunk size from the critic chunk size. DQC trains a policy to only predict a partial action chunk, significantly reducing the policy learning challenge, while retaining the value learning benefits of the chunked critic. We instantiate this technique as a practical offline RL algorithm that outperforms the previous state-of- the-art art method on the hardest set of environments in OGBench (Park et al., 2025a), a challenging, long-horizon goal-conditioned RL benchmark.
RELATED WORK
Offline and offline-to-onlile reinforcement learning methods assume access to an offline dataset to
learn policy without interactions with the environment (offline) (Kumar et al., 2020; Kostrikov et al., 2022; Tarasov et al., 2024) or with as little online interaction with the environment as possible (offline- to-online) (Lee et al., 2022; Ball et al., 2023; Nakamoto et al., 2024). Q-leaming TD-based RL.
2

---

Published as a conference paper at ICLR 2026
algorithms have been popular choice for these problem settings as they naturally handle off-policy data without the need for on-policy rollouts, and also exhibit great online sample-efficiency (Chen et al., 2021; D'Oro et al., 2023). A large body of literature in these two problem settings has been focusing on tackling the distribution shift challenge by appropriately constraining the policies with respect the prior offline data, and most of them use the standard 1-step TD backup for Q-learning, which has been known to suffer from the bootstrapping bias problem in the RL literature (Jaakkola et al., 1993; Sutton et al., 1998). To tackle this, recent work (Jeong et al., 2023; Park & Lee, 2025; Park et al., 2025b; Li et al., 2025b) has shown that multi-step return backups are effective for improving offline/offline-toee tonolinn Q-learning agents. These methods either use a standard single-step critic network (Park et al., 2025b) that suffers from the off-policy bias, or use 'chunked,' multi-step critic network (Li et al., 2025b) that does not have such bias but poses huge policy learning challenge when the chunk size is too large. Our method brings the best of both worlds-it uses critic chunking to avoid the off-policy bias while simultancously avoiding the policy learning challenge by extracting simpler policy that extracts a shorter action chunk from the full-chunk critic.
Multi-step return backups are computed with multi-step off-policy rewards that can lead to system- atic value underestimation (Sutton et al., 1998; Peng & Williams, 1994; Konidaris et al., 2011; Thomas et al., 2015), and there has been a rich literature (Precup et al., 2000; Munos et al., 2016; Rowland at. 2020) dedicated to fix these biases via importance sampling (Kloek & Van Dijk, 1978) with trun- cation (Ionides, 2008). These approaches often require a careful balance between bias and variance that can be tricky to tune. More recently, Sco & Abbeel (2025); Li et al. (2025a); Tian et al. (2025); Li et al. (2025b) group temporally extended sequences of actions as chunks and directly estimate the value of an action chunk rather than a single action. Such formulation allows the value backup to op- erate directly in the chunk space, which allows multi-step return backup without the systematic biases from the sub-optimal off- offlcolicy data. Despite their empirical success, we still lack a good theoretical understanding of the convergence of TDDlearning with 'chunked' critics, as well as when it should be preferred over the standard re-step returns. Our work lays out the theoretical foundation for Q-learning with critic chunking, and identifies an important yet subtle, often overlooked bias in the chunked TD- backup. We quantify such bias and provide the condition under which TD backup using critic chunk- ing is guaranteed perform better than the standard n-step return backup with single-step critic.
See additional discussions of related work on hierarchical reinforcement learning and theoretical analysis for action chunking and confounding variables in Section E.
PRELIMINARIES
Reinforcement learning can be formalized as a Markov decision process, M A.T,..),, where S is the state space, A is the action space, T : S x A + As is the transition kernel that defines the next state distribution conditioned on the current state and the current action (e.g., s' ~ T(- a)), r S A + [0, 1] is the reward function, P As is the initial state distribution, and 7 E [0, 1) is the discount factor. We also assume we have access to a prior offline dataset D {(s). ab. ro. st at ri
(1)) where the goal is to learn policy, π S - AA that maximizes its return, TTTT) = (1~T((1)).))))) aa))))))11(())))0--- (EPo 1'r(St, a.)]. We call a policy that attains the maximum return as optimal policy, n².
Temporal difference learning. Modern value- based reinforcement learning methods often learn a critic network, Q S x A- - IR parameterized by é approximate the expected return starting from state s and action a, and Φ is often trained using the temporal-difference (TD) loss:
((Φ)= = E. [(oo(s r(s, a) - 7 max @a())
where Ō is often set to be an exponential moving average of Ø.
Implicit value backup. Instead of using max, Q(s', a's as the TD target, we can use an implicit maximization loss function fimp to learn Ve(s) to approximate (Kostrikov et al., 2022):
L(E) = Egaa-D [frimp (Q(s, a) - VV(ss))
Two popular choices of fimp are (1) expectile: fespectite (c) |K Lecole and (2) quantile: fquaatile (c) = |K Iecullc|, for any real value ε E [0.5, 1). At the optimum of L(E), Vs(() approximates the x-expectile//uanniiee of the distribution of the TD target for Q(s, a), induced by the
3

---

Published as a conference paper at ICLR 2026
data distribution D. With such technique, longer need explicitly find the action that maximizes Q(s, a) and can use (s(s) as the backup target:
L ( s' [(Q((s,)) a) r(s,a) - >V{(8)))
Multi-step return backup. TD learning can sometimes struggle because regressing the value network towards its potentially inaccurate value estimates amplifies the value estimation errors further. To tackle this challenge, we can instead sample a trajectory segment, (8t, dt, 81+1, 0ttn-1, 8t4n),
construct an n-step return backup target from states h steps ahead:
(n)(d) = Eecaeer" 1e
[Qo(st, ar) - Res4n 7" (setnn ((s))))((+n))))))))))))))))))))) a(tn))
where a'nn arg maxx (s ac+n), Rectin == 2t "r(e dr). The n-step return backup reduces the effective horizon by a factor of n, alleviating the bootstrapping bias problem.
Action chunking critic. Alternatively, one may learn an action chunking critic to estimate the value of short sequence of actions (an action chunk), Ocrth == (at, 01+1,"' .0++--1) instead: Qs:;: (ut+k) (Seo & Abbeel, 2025; Li et al., 2025a; Tian et al., 2025; Li al., 2025b). The TD backup loss for such critic is naturally multi-step:
[o)(@) = -Ss (Qo(sr, amtth) Reath - - ((
(5)
where again a hh arg max Q(stth, a(4h4421)-
4 WHEN AND HOW SHOULD WE USE ACTION CHUNKING FOR Q-LEARNING?
In this section, we build a theoretical foundation for Q-learning with action chunking critics. We start by providing a formal definition of our key open-loop consistent condition (Definition 2), quantifying the value estimation bias incurred from backing up on non-action chunking data (Theorem and the optimality of action chunking policy (Theorem 2) using this condition. Finally, we characterize the key optimality variability conditions under which the closed-loop execution of learned action chunking policy is close to the optimal closed-loop policy (Theorem 3).
4.1 ASSUMPTIONS AND NOTATIONS
To build the foundation of our analysis, we start by describing the trajectory data distribution that we use for Q-learning and the trajectory distribution induced by an action chunking policy. In particular, we assume that the trajectory data distribution obeys the transition dynamics T:
Assumption (Data Obeys the Transition Dynamics) E AT is a trajectory distribution generated by rolling out behavior policy from distribution of μ.. The behavior policy can be non-Markovian (i.e., Ta Ta((++ 81at+++1 (te+4)). Each subsequent state is generated according the dynamics of the MDP M: 8148+1 T(: 8e+ky a++*), vk E 1, h-- 1).
The resulting trajectory is {st, St41, St+h az, a141, (asth] E T = Sh x AA.
Next, we formally define the open-loop trajectory distribution that we would obtain if take the same actions in the data and roll them out open-loop for h steps in the MDP.
Definition (Open-loop Trajectory) From any data distribution D. we use T S + A to denote an action chunking policy that admits the same marginal distribution as D:
(a ss)) := Pp st).
(6)
Rolling out this action chunking policy by carrying out actions chunks induces a trajectory distribution P3 E Agh+s Ar that is generally different from Pp:
A
Pe ($t41:24h+1 C st) = T m
8t
T T(8
S14k, at+k).
(7)
k=
4

---

Published as a conference paper at ICLR 2026
Finally, we introduce set of notations and conventions that we use in our theoretical analysis. We use Oct++ to denote an action chunk of length h: (aq. ac+1, , (not including as+h). We use the subscript Ha for all action chunking policies value functions, ['] to denote the nominal (i.e., estimated) value (in contrast to the actual without the ')),, and [-] to denote something that is learned from the data (usually defaults to D). For example, V++ : - [o, 1/(1 - -))] is the nominal value (i.e., expected discounted return) of an action chunking policy The learned from D, whereas Va is the actual value of the same action chunking policy (where the value is obtained by rolling the policy out in the MDP with open-loop action chunks). As we will elucidate in the next section, the nominal value and the actual value of policy are usually different, and hence making this differentiation critical in our analysis. We also use [-]* to denote the optimal policy or value function under the constraint of the policy class (e.g.. Tae for the optimal action chunking policy and x for the optimal closed-loop 1-step policy). Finally, we use H 1/(1 - 7), = 1/(1 7²) to denote the effective horizon for 1-step TD backup and h-step TD backup respectively.
4.2 WEAK AND STRONG OPEN-LOOP CONSISTENCY (OLC)
From the definition above, we have demonstrated that replaying the actions from the trajectory data distribution PD in an open-loop manner may result different trajectory distribution, PB- This discrepancy between Pe and Po has not been carefully analyzed by prior work but can play huge role in the optimal policy that action chunking Q-learning converges to. characterize this discrepancy, we use notion of consistency defined below.
Definition 2 (Open-Loop Consistency) D is weakly En op-en-ooo consistent if for every St E with (D))s:) > (i.e., St E supp(Po(s:))).
Dry (P a+t st) PD(s4N, aew st)) S Eh, wh (1,2, ... h 1}.
(8)
Dry (Pi st) || PD(sLth s:)) S Eh
(9)
D is strongly on-opel-o consistent if additionally for every ait+h E supp Coop((Poooccaa st)).
Dry(T(setH) 8t, (eathr) ) || (s St, (cs++)) S ch 4h E 21,2.... ,h,,
(10)
where we use T(s+4h 8t, (ra+w)) ) denote the distribution of the future state st after carrying out the action sequence OCe+ in the environment open-loop from the current state 8t-
Intuitively, D is -x-open-loop consistent if, when executing the same sequence of actions from it open-loop from 8t, the resulting marginal distribution of the state-action h steps into the future (i.e., $141) deviates from the corresponding distribution in the dataset by at most Ej, in total variation distance. The strong version (Equation (10)) requires the total variation distance bound to hold for every action sequence in the support, whereas the weak version (Equations (8) and (9)) only requires the bound to hold in expectation. See Section G.2 for examples of weakly open-loop consistent data.
4.3
VALUE LEARNING BIAS OF ACTION CHUNKING Q-LEARNING
Next, show that the weak open-loop consistency of D alone is sufficient to show that behavior value iteration of an action chunking critic results nominal value function (i.e., Vac) with bounded bias from the true value (i.e., Vac) of the behavior cloning action chunking policy Nac:
Theorem 1 (AC Value Bias) Let Vac S + [0, 1/(1 7)] be a solution of
Vac(st) =
Reuth + + Vac Vac(s++))
(11)
with Reath = 14t ott - (sse, ar) and Vac is the true value of Nac : st + Polacah If D is weakly En oppen-oo consistent, then for all s, supp(P:(s.)).
s1)-
7Ek (se) V.c( (8t) S (1->) (1 =n))))) S HH.
(12)
Furthermore, we show that this bound is tight (Theorem 4 in Section in Section H.2 and Section H.3. A direct consequence of these results is that the true value of the optimal action chunking policy close to that of the optimal closed-loop policy:
F). The proofs can be found
5

---

Published as a conference paper at ICLR 2026
Corollary (Optimality Gap for AC Policy) Let D be the data collected by any optimal policy x². If D' is weakly En- =--oee--o consistent, then for all st E supp (5t)),
(8t) - (st) I (*+) Vac( S (1 - 7) - (1 - =,)r") S HH,
TEa
(13)
where V* is the value of the optimal policy π², Vac the true value of the optimal action chunking policy, and Vue is the true value the action chunking policy from cloning the data D*:
म ac (att+h 8x) 8t + PD- 8t))
(14)
We show that this bound is also tight (Corollary 2 Section F). The proofs can be found in Section H.4 and Section H.5. Next, we analyze the performance of the action chunking policy obtained by Q- learning as a solution of the bellman optimality equation under supp(D):
Qte (St, (e.+hh) =
|Reehh + 7 Q (Scthi The Te (bc+)))
(15)
where the is defined as THe + arg Eouppsttnenappppp l. tte (st, (1:4+)).
With only the weak open-loop consistency condition, the worst-case performance of the action chunking policy may be arbitrarily low (see formal statement in Proposition 2 with proof available in Section H.6). Intuitively, the chunked critic ((5t, dz:t+h) has no way of differentiating a low- probability, 'lucky" success from a closed-loop. high-probability success. This can cause the learned policy The to erroneously prefer very low-value action chunks even when the optimal action chunks are available in the data distribution. Fortunately, the strong open-loop consistency (Equation (10)) is sufficient for optimality of action chunking Q-learning as quantified by the following bound:
Theorem 2 (AC Q-Learning under Strong OLC) I and D' are strongly ex-open-loop con- sistent and supp(Pp(s;, (1:++h)) 2 supp(Pp= (St, at:t+h)), then for all st E supp(Po (st)),
E
2
(*+(s) - V nc
(s)) - 7 7 1 1- (1 2
+ - (1 - =x)r I HH,
(16)
where V* is the value of a closed-loop optimal policy and Vt is the true value of n
Theorem 2 (proof in Section H.7) shows that as long both D and D* satisfy the strongly open- loop consistency condition and DD contains the behavior in D*, Q-Jearning with action chunking is guaranteed to converge with Theorem in Section (with proof in Section H.8). Up to now, none of the bounds that we have shown far depend on the sub-optimality of the data. Indeed, can make the data arbitrarily sub-optimal while the action chunking policy learning is still guaranteed to be near optimal wheras naïve n-step return may suffer (see Section G.1).
near-optimal action chunking policy. Also, we show this bound is light
4.4 CLOSED-LOOP EXECUTION OF ACTION CHUNKING POLICY
If we reuse the same strongly Ey En-open-looy consistency assumption, can guarantee that closed- loop execution of the action chunking policy is also near-optimal (proof available in Section H.9):
Proposition 1 (Optimality of Closed-loop Execution of Action Chunking Policy) Let V be the value of the one-step policy, π as result of the closed-loop execution of the action chunking policy tे. learned from D. That is, for each E supp(
(sx) = where Ttrth =T(((s))
(17)
If D and D are both strongly Ea on-open-ooop consistent and Supp(Po(ssu (e)+))) supp Pr (st, (r then for all 8t E supp( (Po: (st)),
(*(x)) - (*) <
ENT
1 - -, + 1
2
1
S 33 (18)
2
wwhh
However, are paying up horizon factor H in sub-optimality gap. Can we do better?

---

Published as a conference paper at ICLR 2026
In practical applications, the data distributions that dealing with often have structure. For example, it is common to have a dataset consisting of multiple sources where each data source is collected by either human expert (e.g., after a robot arm picks up a cube, will always move up rather than dropping right away). We formalize this kind of structures notion of optimality variability follows:
scripted policy that exhibits somewhat predictable behavior
Definition 3 (Optimality Variability) 'D exhibits On-bounded variability in optimality condi- tioned an event X if
[Reath + V (upp(Po(IX)))
min [Reath + +7" s Un. (19)
max supp(Po(-IX))
It turns out that if (1) the data distribution is a mixture of a bunch of data sources where the optimality variability conditioned on the current actions is bounded within each data source, and additionally (2) the optimality variability conditioned on the current action chunks is bounded globally across mixture, we can form a much stronger bound on the optimality of n
Theorem 3 (Closed-loop AC Policy under Bounded OV) Let D' be the data distribution collected by an optimal policy. Assume D can be decomposed into a mixture of data distributions {D", D1, D2 Das} } such that each data distribution component satisfies Assumption and for some 0/ 0% 2 0, they satisfy the following two conditions:
1. Locally bounded optimality variability condition:
every D; (including D*) exhibits 0 bounded variability in optimality conditioned 8t, ag for all (st,a)) E supp Pb. (st, az)). and
2. Globally bounded optimality variability condition: D as a whole exhibits uh -variability in optimality conditioned on St, ata+hh for all (st, ame+h) supp(Pp(ss) ac.++)))
Then for all st E supp(Pp-(se)),
0/
+ 06 + 7" min min(0k, 00)
(*(s,) V - 1-7 (1 7) - 7h) S 0 H + 20 HH.
(20)
We also show that Theorem is tight (see Theorem 6 in Section with proof in Section H.)3).
Overall, compared to executing the action chunking policy in open-loop chunks, closed-loop execution attains a similar bound under the strongly En op-oo-oooo consistent assumption, and excels under the bounded optimality variability assumptions. Conceptually, closed-loop execution of the learned action chunking policy decouples the open-loop execution horizon (policy chunk length) from the value-learning horizon (critic chunk length). Such decoupling inherits the strength of action chunking TD and I-step TD: (1) the value learning speedup of action chunking Q-learning, and (2) the reactivity of standard, single-step policy. Furthermore, executing the first action (or more generally a partial chunk) of the original action chunk also brings practical benefits: it removes the need to explicitly train a policy predict the full action chunk all at once, which can be especially challenging when the chunk size grows big. Can we develop practical method that realizes such potential?
DECOUPLED Q-CHUNKING
In this section, we propose a new algorithm that enjoys the benefits of value backup speedup of critic chunking while avoiding the difficulty of learning an open-loop action chunking policy with large chunk size. Our core idea is to decouple the chunk size of the critic from that of the policy where the policy only predicts a partial action chunk. In particular, we train policy क (at=:th, st) to output an action chunk (with a size of ha < h)) using the following objective:
((x) = Banco (-(1 (s.) (Q [aratha d.4b,
(21)
where (an aethgttth represents the concatenation of two partial action chunks (size ha and size h - h.) into full action chunk ahu+h of size h, and aiangtta is the best 'second-half" of the action chunk that maximizes the critic value under Qø:
Githatth == arg max a teahhtthh Qu (Deethh 4+thae++))
(22)
7

---

Published as a conference paper at ICLR 2026
Essentially, want our policy to predict the partial action chunk (of size ha) within an optimal action chunk of size h, rather than the entire optimal action chunk. This lowers the policy expressivity requirement and hence the learning challenges associated with it.
However, directly optimizing the objective in Equation (21) does not lead to a new algorithm because taking the maximization over tt+hh, seemingly requires to learn a policy of the original chunk size anyways. To address this issue, we learn a separate partial critic Q which only takes in the partial action chunk (of size ha) as input, to approximate the maximum value this partial action chunk can achieve when it is extended to the full action chunk (of size h):
Q (St, at.thh.) z Qo(st. (auath., ++thatth).
(23)
To train Q we can use an implicit maximization loss function (as described in Equation (2)):
L ((b)= == Stap (Qe a+xthh) - Q (St, aesthh)))
(24)
where St, Our-+ h are sampled from the offline dataset D. As result, the partial critic, Qe is distilled from the original critic via optimistic regression, where its optimum Qi (8c; Ocst+h.) approximates Qs( [ag:+h.. ah xl) in Equation (21), conveniently removing the need for training a policy to predict the whole optimal action chunk entirely. This allows us to simplify the policy objective as
l(T) == - Bagyth, (=(-ss)))))))) [ (8t, Otst+ha]
(25)
In summary, DQC trains a policy to predict a partial chunk, Ger-+h. (of size ha), by hill climbing the value of a partial critic Q (se, Ot.t=h_) ) that is distilled from the original chunked critic Qø(s,, a::4hh) via an implicit maximization loss. This allows our policy to fully leverage the chunked critic Q+ (and thus the value speedup benefits associated with Q-chunking) without the need to predict the full action chunk (of size h), mitigating the learning challenge of an action chunking policy.
Practical considerations for offline RI. Finally, we describe several implementation details that we find to work well in the offline RL setting, which our experiments focus on. Our implementation draws inspiration from a prior method, IDQL (Hansen-Estruch et al., 2023).
We first train a behavior cloning flow policy Fy using standard flow-matching objective (Liu et al.. 2023) on the offline dataset D. Then, we approximate the policy optimization objective in Equation (25) using best-of-N sampling without explicitly modeling w:
Gi-thh. + arg max
(atttho) J.v. Q (8), dr.+hh)) where diith.." Ciithh ~ TT(- st), (26)
and ax is output of the policy that extract from O. for state st. Essentially, this sampling procedure is a test-time approximation of the objective in Equation (25), where it outputs an action (chunk) that maximizes Q subject to the behavior prior, as modeled by nT..
For TD learning of Qè, directly computing the TD backup target from either Q₂ or Qf is computation- ally expensive, as either requires samples from the current policy, which is approximated via the best- of-N sampling procedure as described above. Instead, we use the implicit value backup (Kostrikov et al., 2022) (i.e., as described in Equation (2)) to approximate the target:
L(E) fquantill (Q (sc,
(sc, ac.thh.)) -(()t))))))s))).
(27)
where we pick the quantile regression loss as the implicit maximization loss function. This is because the Q-value obtained from best-of-N sampling can be seen as the largest order statistic of a random batch (of size N) of the behavior Q-values. Such statistic estimates the behavior Q-value distribution's 's N-1 -quaniite,,, which is the same as (s(s) at the optimum of L(E) if we set Ai N practice, we use smaller Ke for numerical stability (see Table 7).
Finally, we pick the expectile regression loss for training the distilled partial critic Qf because prior work has found it to work the best among all implicit maximization loss functions FHstnshn-Estruch ct al., 2023). A summary of the algorithm is available in Section A, Algorithm
EXPERIMENTAL SETUP
We conduct experiments to evaluate the benefits of decoupling the policy chunk size and the critic chunk size OGBench (Park et al., 2025a) challenging long-horizon, goal-conditioned offline
8

---

Published as a conference paper at ICLR 2026
DQC
-0-
DQC-naive
-0-
QC
-
N5
05
cube-triple-100m
cube-quadruple-
cube-octuole-18
1.00
1.00
1.00
0.75
0.75
0.75
Overall Success Rake
0.25
0.25
0.00
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
numancidmaze giant
puzzle-4x5
puzzle 4x6:le-446:11
1B61
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.00
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Steps (x10%)
Figure 2: Offline goal-conditioned RL results (10 seeds). Our method (DQC) uses decoupled critic and policy chunk sizes. QC: Q-chunking (Li al., 2025b); NS: n-step return backup; OS: 1-step TD-backup; DQC-naive: same as QC but executes a partial action chunk.
Task
FBC
HFBC
IQL
HIQL
SHARSA
NS
DQC-naive
DQC
cohe-triple-- 100M
84 p4.000
56 66.000
66 68,00.7
35
80
47 11
93 g3
p0
27 p1sas
98
oube-quadruple 1000
34,00,
37
24.1.28
02.60
Oggor
27
35 35
40
912 p00.0)
aub-octtppee-18
28 20.200
20 17:23]
34 [s1.00]
9p.1a]
34 [as,as]
humanoidmaze-gt
Gual
3 32.5
24 22.20]
18 p0.200
95 94.97]
49 60.02
80 pmus
92 (x0.04)
pusale-425
0pn
20 (0.00)
0
10.9
19 rs
9X3
as
20 pouan)
33
96 pc
puzele-426-18
10.0
63.0
9
64 (00
19
91
64]
28 p0..0
33 ps.a9
83
Table 1: Comparisons with prior methods (10 seeds). Our method outperforms SHARSA (Park al., 2025b) (the previous state-of-the-art method on this benchmark) most environments.
RL benchmark consisting of a diverse set of environments (from manipulation to locomotion). We now describe our main comparisons. To start with, consider several direct ablation baselines where the same algorithm backbone is being used (i.e., implicit value backup and best-of-N sampling):
QC (Q-chunking (Li al., 2025b)) uses a single critic that has the same chunk length as that of the policy (i.e., h = h2)). This baseline tests whether having decoupled chunk sizes is important.
DQC-naive is naive attempt at decoupling the critic chunk size from the policy chunk size, where it takes the QC policy to predict full action chunks of size h but only execute the first ha actions.
NS: n-step return TD backup. This baseline uses single one-step critic (i.e., ( at)). Compared to DQC with h = n and ha = 1, this bascline tests whether using a chunked critic is important.
OS: Standard 1-step TD backup. This is the same NS but with n
We also consider a few strong baselines from prior work: FBC/HFBC (Park et al., 2025b), IQL/HIQL (Kostrikov et al., 2022; Park et al., 2023), and SHARSA (Park et al., 2025b).
In our ablations, we also consider an additional baseline, QC-NS, that uses the idea of decoupled policy chunking and critic chunking, but without using distilled critic. This baseline simply uses n-step return targets to directly train a critic with a chunk size of h2 without implicit maximization (Equation (24)). For all our experiments, we run 10 seeds for all methods, and report the means and the 95% confidence intervals.
7 RESULTS
In this section, we present our experimental results to answer the following three questions:
(Q1) Does DQC improve upon n-step return, Q-chunking? Figure compares DQC (ours) to both n-step and QC across six challenging long-horizon horizon GCRL environments, with our method performing on par or better across the board. Table 1 shows DQC also consistently outperforms the previous state-of-tt ofrthe-art method on this benchmark, SHARSA (Park et al., 2025b), on all environments. For each environment, we tune DQC (ours), QC, NS, and OS (see the tuning range in Table 8) and pick

---

Published as a conference paper at ICLR 2026
Overall Success Rate
cube-triple-100M
cube-quadruple-10m
cube-octuple-18
backup horizon
1.00
1.00
1.00
DOC (h-5,
NS (n 5)
0.75
backup herizon 2on12on-22,.000 25,0,
0.50
DGC (h-25
NS (n 25)
0.00
backup 25,
0.00
0.75
1.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
DGC (A) 25, -5
Training Steps (x10*)
QCAS (n- 25, N2
Figure 3: Distilled critic ablations (10 seeds). Each group in the legend contains DQC and its non- distilled counterpart with the same configuration. Our method (DQC) performs on par or better than the non-distilled counterpart across all configurations.
Overal Success Rate
Best-of-N Policy Extraction (N)
Implict Loss
Implicit Parameters (Ka Ka)
Batch Size
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.25
0.25
dall
0.25
-0- 6-
0.25
2924
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75 1.00
Training Stops (X10%)
Figure 4: Hyperparameter sensitivity analysis on cube-quadruple (10 seeds). Best-of-N: the number of action samples drawn from T.(- s) during policy evaluation; Implicit loss type: the implicit maximization loss function used for distillation and value backup; Batch size: the number of examples used in each gradient step.
the best configuration (Table 6) for hyperparameters used in Figure 2 and Table 1. For all baselines from prior work (SHARSA, HIQL, IQL, HFBC, FBC), we directly use their tuned hyperparameters and run with the same batch size (i.e., 4096) as used in our method and other baselines. See the complete result table for all combinations of h, n, ha in Section B.
(Q2) Is training separate distilled critic Q necessary? In Figure 3, we compare DQC DQC without using the distilled critic across three different (h, ha) configurations: (h 25, 5). (h = 25, h2 = 1), and (h = 5, h. = 1). For configurations with ha = 1. the baseline without using the distilled critic is the same as the n-step return baseline (with with ha = 5, it is the same as combining Q-chunking and n-step return (QC-NS). Across three configurations, DQC performs on par or better than its non-distilled counterpart. This highlights that the a separate distilled critic for the partial action chunk is necessary for the effectiveness of DQC.
h) and for the configuration
(Q3) How sensitive is DQC to its hyperparameters? Figure shows that our method is not sensitive to the implicit backup method (quantile or expectile), and somewhat sensitive to the implicit parameters Kb; Ka. In particular, DQC is still reasonably effective as long as some form of optimism is employed (i.e., either Ks # 0.5 Ka * 0.5). Using no optimism (N₃ Rdd 0.5) results in big performance drop. The other important hyperparameters are N in the best-of-N policy extraction and the batch size. Having large enough batch size (i.e., 4096) and N (e.g., N 32) is erucial for good performance, although increasing N further (e.g., 128) does not lead to better performance.
8 DISCUSSION
We provide a theoretical foundation for action chunking Q-learning and demonstrate how effec- tively extract policies from chunked critics. Theoretically, we provide a formal analysis of action chunking Q-learning. identifying the TD backup bias that arises from open-loop inconsistency and characterizing the conditions under which action chunking Q-learning is preferred over n-step re- turn learning and the conditions under which closed-loop execution the action chunking policy is near-optimal. Empirically, we develop a new technique that enables effective policy extraction from chunked critics with long action chunks, scaling up action chunking Q-learning to much harder environments. Together, these contributions advance the goal of tackling bootstrapping bias in TD- learning. Several challenges remain, indicating promising avenues for future research. For example, method relies fixed policy action chunk size ha and critic action chunk size h across all states, even though the optimal action chunk size may vary by state. Developing practical methods that can support flexible, state-dependent chunk sizes would be natural next step.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by DARPA ANSR and ONR N00014-25-1-2066 This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley. We would like to thank William Chen for discussions and inspiration, especially on the proof for Proposition 4. We would also like to thank Andrew Wagenmaker for suggestions and feedback on the theory (Theorems and 2 and Propositions and 3). We would also like to thank Dibya Ghosh for feedback on early version of the teaser figure and Ameesh Shah for writing feedback on an carly draft of the paper.
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline primitive discovery for accelerating offline reinforcement learning. International Conference on Learning Representations, 2021. URL httpt:tt/pppereeeeee http:////ppppppooooooooooooooa// net//forumn1de-VVB......111
Kamyar Azizzadenesheli, Alessandro Lazaric, and Animashree Anandkumar. Reinforcement learning of pomdps using spectral methods. In Conference on Learning Theory, pp. 193-256. PMLR, 2016.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In International Conference on Learning Representations, 2019.
Akhil Bagaria, Ben Abbatematteo, Omer Gottesman, Matt Corsaro, Sreehari Rammohan, and George Konidaris. Effectively learning initiation sets in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Leariing, pp. 1577-1594. PMLR, 2023.
Christian Bayer, Boualem Djehiche, Eliza Rezvanova, and Raul Fidel Tempone. Continu- ous time stochastic optimal control under discrete time partial observations. arXiv preprint arXiv:2407.18018, 2024.
Andrew Bennett and Nathan Kallus. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes. Operations Research, 12(3)_007- 2024.
Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. Off-policy evaluation in infinite- horizon reinforcement learning with latent confounders. In International Conference Artificial Intelligence and Statistics, pp. 1999-2007. PMLR, 2021.
Boyuan Chen, Chuning Zhu, Pulkit Agrawal, Kaiqing Zhang, and Abhishek Gupta. Self-supervised reinforcement learning that transfers using random features. Advances in Neural Information Processing Systems, 36, 2024.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Representations, 2021. URL htstps://openreeeeee wtw.renrrriieeeeee/rrrrmrumneeei aeet//oorum/dddAAYa22mmm
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, 17, 2004.
Imre Csiszár. On information-tyya measure of difference of probability distributions and indirect observations. Studia Sei. Math. Hungar., 2:299-318, 1967.
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 17(33)1-50,, 2016.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
11

---

Published as a conference paper at ICLR 2026
Kristopher De Asis, Hernandez-Gare G Holland, and Richard Sutton. Multi-step reinforcement learning: A unifying algorithm. In Proceedings of the AAAI conference on artificial intelligence. volume 32, 2018.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of artificial intelligence research, 13:227-303, 2000.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/ www./eeeeteid-ngeeeee 9aBBVJe.
Paul Dupuis and Hui Wang. Optimal stopping with random intervention times. Advances in Applied probability, 34(1):141-157, 2002.
Ishan P Durugkar, Clemens Rosenbaum, Stefan Dembach,, and Sridhar Mahadevan. Deep reinforce-
ment learning with macro-actions. arXiv preprint arXiv: 04615, 2016.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. CoRR, abs/1703.08294, 2017. URL http:: /ttttp:///iiiiiia org/ www.aos.////.....oooo...o.....rs wwtpaa//.../////////////////aaaa 002./499...00...
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot reinforcement learning via functional reward encodings. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the s1 International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 13927-13942. PMLR, 21-27 Jul 2024. URL https: httpps://proceedings mlr. www.ppessseeee wttesss///////// fww.s2aa.aa2..aa/aa html.
Zuyue Fu, Zhengling Qi, Zhaoran Wang, Zhuoran Yang. Yanxun Xu, and Michael R Kosorok. Offline reinforcement learning with instrumental variables in confounded markov decision processes. arXiv preprint arXiv:2209.0866 2022.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. Advances in Neural Information Processing Systems, 34:11553-11564 2021,
Philippe Hansen Hansen-Estruch,, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Hao Hu, Yiqin Yang, Jianing Ye, Ziqing Mai, and Chongjie Zhang. Unsupervised behavior extraction via random intent priors. In Seirth- Conference on Neural Information Processing Systems, htt.///mmiiddvvvoo//
2023. URL https: / /oppreeeeeeeeeeeeeee
Edward L Ionides. Truncated importance sampling. Journal of Computational and Graphical Statistics, 17(2):295-311 2008
Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dynamic programming algorithms. Advances in neural information processing systems, 6, 1993.
Jihwan Jeong, Xiaoyu Wang, Michael Gimelfarb, Hyunwoo Kim, Baher abdulhai, and Scott Sanner. Conservative bayesian model-based value expansion for offline policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https: https: /ttp:////////eeeie.//mmom/ /eeteeerreeeeeeeeeeeeeeee net/ forum?id=dW@qz2 foruunid-dMMq2Zeyjyc.
Nathan Kallus and Angela Zhou. Confounding-robus policy evaluation in infinite-horizon reinforce- ment learning. Advances in neural information processing systems, 33:22293-22304 2020.
Nathan Kallus and Angela Zhou, Minimax-optimml policy learning under unobserved confounding. Management Science, (75)) 2021.
Chinmaya Kausik, Yangyi Lu, Kevin Tan, Maggie Makar, Yixin Wang, and Ambuj Tewari. Offline policy evaluation and optimization under confounding. In International Conference on Artificial Intelligence and Statistics, pp. 1459-1467. PMLR, 2024.
12

---

Published as a conference paper at ICLR 2026
Teun Kloek and Herman K Van Dijk. Bayesian estimates of equation system parameters: application of integration by monte carlo. Econometrica: Journal of the Econometric Society, pp. 1-19, 1978.
Anita De Mello Koch, Akhil Bagaria, Bingnan Huo, Cameron Allen, Zhiyuan Zhou, and George Konidaris. Learning transferable sub-goals by hypothesizing generalizing features, 2025, URL httpp://openrrevir. wttp:eeviwiieeeeefreeeeeeeeeeeee www.orrmmnii--000030000.
George Konidaris, Scott Nickum, and Philip Thomas. TD,: Re-evaluating complex backups in temporal difference learning. Advances in Neural Information Processing Systems, 24, 2011.
George Dimitri Konidaris. Autonomous robot skill acquisition. University of Massachusetts Amherst, 2011.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https:// // openreview aee//foorum/ddd6B22s22....
Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 18:1179-1191, 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint http::200000064445 2020.
Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, and Gerhard Neumann. TOP-ERL: Transformer-biie based off-policy episodic reinforcement learning. The Thirteenth International Conference Learning Representations, 2025a. URL https::t/tttteeeeeeeeee http://ppppeeeeeeeeeeeeeeee net/ wwt.:nnunnrr/r...oe N4NhVN30ph.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025b. URL wwwpeeewwwiweeeeeeeeeeeeeeeeeeee f
Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforcement learning not scary? In Conference on Learning Theory, pp. 5175-5221 5220. PMLR, 2022.
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023. URL https: htttps://ppettt /ttp///eep//eeeee//////ee/ee/eee nwt/fformmidddXVV .ett/
Amy McGovern and Richard Sutton. Macro-actions in reinforcement learning: An empirical analysis. 1998.
Ishai Menache, Shie Mannor, and Nahum Shimkin. dyut-iynamic discovery of sub-goals in reinforcement learning. Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13, pp. 295-306. Springer, 2002.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations, 2019. URL https: /ttttp:://opeee /ptp//eeepeeeeeeeeeee/eeeeeeeeee net/ forum/id-BB16Tjjea
Rui Miao, Zhengling Qi, and Xiaoke Zhang. Off-policy evaluation for episodic partially observable markov decision processes under non- parammeectiic models. Advances in Neural Information Pro- cessing Systems, 35:593-606, 2022.
13

---

Published as a conference paper at ICLR 2026
Prabhat K. Mishra, Debasish Chatterjee, and Daniel E Quevedo. Stochastic predictive control under intermittent observations and unreliable actions. Automatica, 118: 118:109012, 2020.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off- off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36, 2024.
Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off- policy policy evaluation for sequential decisions under unobserved confounding. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 18819-18831. Curran Associates, Inc., 2020. URL https: p/ppoeeed//oooeiinggs cwurips.cc/papeee. files/ Sataper_filessppeee/////0 ppppp///22//f//l// pdf.
Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning, 2022.
Kei Noba and Kazutoshi Yamazaki. On stochastic control under poisson observations: optimality of barrier strategy general I, evy model. arXiv preprint arXivv2210.00501, 2022.
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic movement primitives. Advances in neural information processing systems, 26, 2013.
Kwanyoung Park and Youngwoon Lee. Model-based offline reinforcement learning with lower
expectile g-learning. In The Thirteenth International Conference on Learning Representations, 2025. URL
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline goal-
conditioned RL with latent states as actions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL TtLLhttss//pppereiiiie http://ww....eeotrmmteeeeeeaaao-
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In Forty-first International Conference on Machine Learning. 2024. URL /tsttss:/oppervviii nett/forum?.de-LNleeal....
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking
offline goal-conditioned RL. In The Thirleenth International Conference on Learning Representa- fions, 2025a. URL https: /ttp////ppeeeeeeeeeeeeeeeeeeeeee net/ feet/forum?? (eerum?id=M992mmm d=M992mjgKZI..
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes RL scalable. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL htttpp:://ppereeeeeeeee www.eewwweee..otteeeeeee cottt/fommmiid. hguaupzLCU.
Jing Peng and Ronald J Williams. Incremental multi-step Q-leaming. In Machine Learning Proceed- ings 1994, pp. 226-232. Elsevier, 1994.
Xue Bin Peng, Glen Berseth. KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hicrarchical deep reinforcement learning. Acm transactions on graphics (tog), 3614) 2017.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-204, PMLR, 2021.
Doina Precup, Richard Sutton, and Satinder Singh. Eligibility traces for off- policy evaluation. In ICML, volume 2000, pp. 759-766. Citeseer, 2000.
14

---

Published as a conference paper at ICLR 2026
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference machine learning, pp. 4344 4353..
PMLR, 2018.
Mark Rowland. Will Dabney, and Rémi Munos. Adaptive trade-offs in off-policy learning. In International Conference on Artificial Intelligence and Statistics, pp. 34-44. PMLR, 2020.
Younggyo Seo and Pieter Abbeel. Coarse-to-fin q-network with action sequence for data-efficient reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https::tt/// /ttp::////e/eeeeeeeeeeeee
Younggyo Seo, Jafar Uruç, and Stephen James. Continuous control with coarse-to-fine fine reinforcement learning. In 8th Annual Conference Robot Learning, 2024. wtw.ttttt:///peeeeeeeeeee net/
Forum?id=sjjDR4e1330
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pp. 8624-8633, PMLR, 2020.
Chengchun Shi, Masatoshi Uchara, Jiawei Huang, and Nan Jiang. A minimax learning approach off- policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning, pp. 20057-20094. PMLR, 2022.
Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, (9(555):273--288 2024.
Max Simehowitz, Daniel Pfrommer, and Ali Jadbabaie. The pitfalls of imitation learning when actions are continuous. In Nika Haghtalab and Ankur Moitra (eds.), Proceedings of Thirty Eighth Conference on Learning Theory. volume 291 of Proceedings of Machine Learning Research, pp. 5248-5351. PMLR, 30 Jun-04 Jul 2025. URL htttps://proceediine mlr simchowitz25a html.
wttp://s...m.....s/22ss0// press/v291/
Özgür Şimşek and Andrew G. Barto. Betweenness centrality basis for forming skills, Working- paper, University of Massachusetts Amherst, April 2007.
Aravind Srinivas, Ramnandan Krishnamurthy, Peeyush Kumar, and Balaraman Ravindran. Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv preprint 1505.0599 2016.
Richard Sutton, Andrew Barto, et al. Reinforcement learning; An introduction, volume MIT press Cambridge, 1998.
Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)181-211, 1999.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach offline reinforeement learning. Advances in Neural Informaion Processing Systems, 36, 2024.
Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environ- ments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10276- 10283, 2020.
Philip S Thomas, Scott Niekum, Georgios Theocharous, and George Konidaris. Policy evaluation using the Ω-return. Advances in Neural Information Processing Systems, 28, 2015.
Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann. Chunking the critic: A transformer-hased soft actor-critic with N-step returns. arXiv preprint arXiv:2503.03660, 2025.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In
The Eleventh International Conference on Learning Representations, 2022.
15

---

Published as a conference paper at ICLR 2026
Stephen Tu, Alexander Robey, Tingnan Zhang, and Nikolai Matni. On the sample complexity of stability constrained imitation learning. In Learning for Dynamics and Control Conference, pp. 180-191. PMLR, 2022.
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions. Advances in neural information processing systems, 29, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning, In International conference on machine learning, pp. 3540-3549, PMLR, 2017.
Hui Wang. Some control problems with random intervention times. Advances in Applied Probability, 33(2):404 422, 2001.
Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine. Leveraging skills from unlabeled prior data for efficient online exploration. In International Conference on Machine Learning (ICML), 2025.
Yihong Wu. Lecture notes on information-theoreti methods for high-dimensional statistics. Lecture Notes for ECE598YW (UIUC), 16:15, 2017.
Kevin Xie, Homanga Bharadhwaj. Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. In International Conference on Learning Representations, 2021. URL URL.https://ogenrevviee
Shuhao Yan, Mark Cannon, and Paul J Goulart. Stochastic output feedback MPC with intermittent observations. Automatica, 141:1 10282, 2022.
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 2.5607/RSS.2023.XXX..
16

---

Published as a conference paper at ICLR 2026
Algorithm I Decoupled Q-chunking (DQC).
Given: D, (ot tn)+h): f (8t; (n:thha), Ve(St); Ta(areth
1. Agent Update:
(stc+++11 Ourto (1a4h) D.
sample trajectory chunk from the offline dataset
Optimize Qs with L(Ø) = (Qolse, aeeen))))))))))))))))))))))))a - -n 2 r - -
Optimize 2 with L(b) Sespectile (((s)), -
arothaa) (sr,
Optimize V/g with [(E) S quamiite (Qg (St, (t:ttha)) (s():
2. Policy Extraction:
hh Tz(. 8t)
> sample N actions from behavior policy
ghttho arg max
1x (s1, aratha)
>> take the action with the highest Q-value
(*) ha Jim1
A ALGORITHM PSEUDOCODE
B FULL RESULTS
Table 2 reports the performance our method (DQC) and baselines for all hyperparameter configu- rations. All of them use the same hyperparameters in Table 4 with the only exception that SHARSA, HIQL, IQL, FBC, and HFBC handle goal- sampling for training behavior cloning policies differently. We discuss this in more details in Section D.
Method
c3-100M
c4-1000 14-100M
c8-1B
hg
p45
p46-1B 1B
DQC
h 25
h,
76 [3.80]]
45 [9]
10 [s.)111]
92 [90994]
91 [69.92]
83 [00866]
DQC
h 25
h, 5
98 p8.09]
92 [90.98]
34 (33.96)
51 [48,64]
96 95.97
68 (66,7]
DQC
h 5
he
p4.97]
84 133, so]
0g0,0]
19 p,.22]
90p88,92)
44 (42,477
DQC-naïve
25
Re
14 [8,22]
16 99225
[0.2]
22 [e0,24]
32 [28.66]
33 20.97]
DQC-naive
h 25
ha
27 [1..38]
27 [1.39
Ojo.aj
33 p29
33 p8.38]
DQC-naive
h 5
he
16 [7,30]
40
29.400 49]
0o.00
80 [77,83]
20 (20.20]
26 ps.28]
QC
h 25
he 25
21 [13,31]
12
(7,88]
Drooo]
30 30 [27.33]
37 [33
QC
h 5
he
20 [7,36]
35
(28, 43
(0.0
48
[45.52]
20 pp0.20]
28 (27
QC-NS
n = 25
he =
[22,800 80]
53 22. 77]
18 [1.255]
60 (8.61]
95 99.96]
95 (93.97
NS
n 25
ha
30 [20
19
[8]
9
95
94.97]
89 [87,91]
91 [SG,94]
NS
5
ho 93 (1.94]
27
[11443]
1o0.3]
89
[87.91]
93 [1.96]
56 (8.66)
OS
ho
47 [11,3]]
[.,0]
Oio,o|
19
[18119]
19 [19200]
FBC
34 [1.56]]
[2.37]]
[1.2]
pooo
[o.1]
HFBC
56 63,
37 [3444]]
28 (66229)
614.8]
Op.o
49.66
IQL
66
53 (2, 65]
O
3
20
63,0
HIQL
35 [1.39]
24 21.28
20 17.20)
24 24
0
[,13]
SHARSA
83 [1185]
64 (22 68)
34 (31,36)
19 [0.233
1
64
(0008)]
Table Complete results for all hyperparameter configurations across differ- ent combinations of h, n and ha (10 seeds). tions: c3-cube-triple, di-cube-quadrupe c8-cube-octuple, hg=humanoidmaze -iiatt,, p45=puzzle-Ge 4x5, 116--uuzzee-4x.. The hyperparameters used are specified in Tables 6 and 7.
We adopt the following abbrevia-
C ENVIRONMENTS AND DATASETS
To evaluate our method, we consider goal-conditioned environments OGBench with varying difficulties (Figure 6). The dataset size, episode length, and the action dimension for each environment available in Table 3. We describe each of the environments and the datasets we use as follows.
Environment cube-*: We consider three cube environments (cube-triph) cube-quade cubbe-quadrupp... cube cube-octuple). As the names suggest, the goal these environments involve using a robot arm to manipulate 3/4/8 cubes from some initial configuration some specified goal configuration. We
17

---

Published as a conference paper at ICLR 2026
4096 (ours)
1024
256
cube-triple-100M
cube-quadruple-100m
cube-octuple 1B
1.00
0.75
0.75
0.75
Overall Success Rate
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
humanoidmaze-ginn
puzzle-4x5
muzzle-4x6-16
1.00
1.00
1.00
0.75
0.75
0.75
0.25
0.25
0.25
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Steps (x10%)
Figure 5: Batch size sensitivity (10 seeds). Large batch size is crucial for DQC's performance especially hard tasks (e.g., wwwe-uuuuucceeeeeee qube-quadruple, cube-octuple, puzzle-4x5 and puzzle-4x0l
use the same five evaluation tasks used in OGBench (Park al., 2025a) for cube- tubpetriple and cube-quadrun quaadruua and the same five evaluation tasks used in Park et al. (2025b) for cube-octuple. We refer the environment detail to the corresponding references.
Environment
Dataset Size
Episode Length
Action Dimension (A)
tibbetrripe-1001
100M
1000
5
cube- uube-quadruple-100 100M
100M
1000
5
cube-octuple-18
1B
1500
5
humanoidmaze-giian
4M (default)
4000
21
puzzle-4x8
3M (default)
1000
5
puzzle-4x6-1B
1B
1000
5
Table Environment metadata. For both humano dmaze-giant and puzzle-4x5, we use the default dataset that is released in the original OGBench benchmark (Park et al., 2025a). For the other environments, we use larger datasets as we find them to be essential for achieving good performances on these environments.
Environment humanoi -:aaee We also consider the hardest locomotion environment available in OGBench. The goal of the environment is to control and navigate a humanoid agent from some initial location to some specified goal location in 16 12 maze. This environment also has the longest episode length (4000, more than twice as long as the second longest episode length as used in cube-octuple). We refer the environment detail Park al. (2025a).
Environment puzzle-+: Finally, we consider two environments that involve solving combinatorial puzzle with robot arm. The puzzle consists of a board of x 5 or 4 buttons, organized regular grid (4 rows and 5 or 6 columns). Each button has a binary state. Whenever the end-effector of the arm touches button, the button and all adjacent four buttons (three or two the button the edge of the grid or in the corner) flip its binary state. The goal of the environment is to transform the board from some initial state to some specified goal state. We refer the environment detail to Park al. (2025b).
At the restttime/vvlluation time, the goal-conditioned agent is tested on five evaluation tasks for each of the six environments consider. The overall success rate is the average over 5 tasks with 50 evaluation trials each. For the prior baselines, SHARSA, HIQL, IQL, HFBC and FBC, we run 15 evaluation trials for each task, following Park et al. (2025b).
18

---

and learn Published as a conference paper at ICLR 2026
Datasets. We use play datasets for all cube-* * and puzzle-* environments and navigate dataset for humanoidmazer
We use the original datasets available for humanoidmaze- gnaat and puzzle-4x5 because they sufficient for solving the environments, Using larger datasets these environments do not help differentiate among different methooss/aasllins. For each the other envi- ronments, the largest dataset available from Park al. (2025b) as we find it to be necessary solve these environments (or achieve non-trivial performance on sube-octuple),
cube- trrple
cube- quadrupua
cube octupee
TAC
humanoidmaze dwwaa-aiaaanntttteee
puzzle e4x5
puzzle-4x6
Figure Environments used in our experiments.
D HYPERPARAMETERS AND IMPLEMENTATION DETAILS
Hyperparameters. Table describes the common hyperparameters used in all our experiments. Tables 6 and 7 describe the environment-specife hyperparameters and Table 8 describes the range of hyperparameters we use for tuning each method.
Goal-conditioned RL implementation details. While we have described in the main body of the paper how DQC works as a general RL algorithm, we have not touched on how DQC and similarly all our baselines works with the conl-conditioned RL (GCRL) setting. We consider the setting where we have access to an oracle goal representation V S - G where G is the goal space (see Table for the oracle goal representation description for each environment). The goal-conditioned reward function (s, g) + (=)== is binary reward function where its output is 1 if the goal g is reached by the current state We can treat y as part of extended state [s, E $ value functions (e.g., Q (8, a)) normally with such extended state.
common practical trick in the GCRL setting is goal relabeling. That is, during training for each (s, a) pair in the training batch, a goal is sampled from some distribution (i.e., (- s, a)) and the reward of the transition is relabeled with the goal-conditioned reward function. Following Park et al. (2025b), the goal distribution (- 8, s, S x A + Ag is a mixture of four distributions, conditioned on the training state-action example:
p9 = Weur cur
P2 + ggoom geom + Wtraj p9 + Wrand P9 traj
Pg
(28)
rand!
19

---

Published as a conference paper at ICLR 2026
Parameter
Value
Batch size
4096
Discount factor (?)
0.999
Optimizer
Adam
Learning rate
3 x 10
Target network update rate (A)
5 10-3
Critic ensemble size (K)
2
min Q2) for cube-*
Critic target
(Q1 + 22 for puzzle-* and humanoid-* binary cross entropy
Value loss type
Best-of-N sampling (N)
32
Number of flow steps
10
Number of training steps
106
Network width Network depth
1024
4 hidden layers
Value goal sampling (w w goom Waraj (rand)
(0.2, 0.5,
DOC/OC/NS/OS: n3 is not goal-conditioned SHARSA (cube): (0, 1, 0)
Actor goal sampling (w up geoms w
Wfraj
w (andd)
SHARSA (puzzle): (0, 1,0) SHARSA (humano dmaze): (0, 0, 1,
Table 4: Common hyperparameters. For the GCRL goal-sampling distribution we follow the same hyperparameters used Park al. (2025b).
Environment
Goal Representation (
Goal Domain (g)
cube-tripe triple
(r,, 2. of three cubes (rel. to center)
R
cube-quadruple
(x, y, four cubes (rel. to center)
R12
cube-octuple
(z, of eight cubes (rel. to center)
R21
Humanoidmaze--ee -giant
(æ, y) of the humanoid
R
puzzle-4x puzzle-4x6
the binary state for each button
{0, 1 20
the binary state for each button
{0, 1] 24
Table 5: Oracle goal representation description for each environment. Following Park et al. (2025b), we assume access an oracle goal representation for each environment. More detailed definition of these oracle goal representations is available in OGBench (Park et al., 2025a).
where
1. (. ()) = ow):):
a)
the goal is the same as the current state;
2. pg grom ( a.): geometric distribution over the future states in the same trajectory that (s, a)
is from;
3. (- a): uniform distribution over the future states in the same trajectory that (s, a) is
from; and finally
4. pg (: 8, ((LD())): uniform distribution over the dataset (D(s) is the distribution of states in the dataset).
and Weur: Wgeom, Wraj: Wrand > 0 are the corresponding weights for each of the distribution compo- nents with waur Wgeom + Wtraj Wrand
In practice, it has been found to be beneficial to use a separate set of goal sampling weights for TD backup (Park al., 2025a) (i.e., (wire> w/ Wg Whrajj (Yand)) )) and for policy learning (iee. (wPrr u.poom
"ggnm "raa (Pan))) However, in our implementation DQC/QC/NS/OS, we do train coal-conditioned policy, our policy extraction done entirely test-time by best-of-N sampling
20

---

Published as a conference paper at ICLR 2026
Enviromment
(h, ha, Ne. Ka)
DQC
DQC-anive
(b, has 6x)
(h, Ra. Ko)
(h Nu. ma)
QC
(o, no)
(no)
SHARSA (8)
(h n. o)
HIQI.
(a)
(b)
cube-triple- 100M
(25, 5. 0.94, 0.8)
(25, 5,098)
(25. 5, 0.335)
(5. 0.33)
(5.0.5
0.5
25
(25,0.5,100 10)
3
25
cube aibe-quadrrppa 100M
625.6.0.90.00 0.5.0.9..00 0.8)
[5. (5.1.0.00)
0.395.
0.93)
(5.0.7)
0.7
25
(25,0.5,11
octuple
(25.5,0.98)
(35.0.03)
0.97
huzanoi dmaze dnaze-nzzeee-nee glant
(25, 0.5, 0.8)
1.0.9)
125. 5.0.5
(5.0.5)
125,0.3
0.5
30
(50,0.3.3) 81
puzzel 4-45
125. 0.8, 0.5)
(25, 6.99)
(26. 5,0.7)
(6.0.0)
(25 0.77
0.7
50
(25.0.7.3)
25
pusse 18
1,..,.07
(25,5,0.7)
(36, 0,0.5)
(6.0.7)
(25, 0.5)
0.7
50
(25,0.7.))
25
Table Environment-ppecffc hyperparameters for DQC, QC, NS, OS, SHARSA, HIQ,, IQL, and HFBC. For SHARSA, HIQL, IQL, and HFBC, we follow the hyperparameters in the original paper (Park et al., 2025b).
DQC
DQC
DQC
QC-N8
NS
QC
QC
OS
Enviromment
A 25, Kg)
25, ha
h, =
25, h,
25
25
(Kg. Kg)
cube-triple- 100M
0.8)
0.8)
(0.5.088)
cube quadruple- 100M
[0.93) 0.8)
(0.93, 0.8)
(0.5.0.5)
0.93
01.
0.7
0.93
0.90
0.7
cubs-octugle--i a-19
(0.93.0.5)
(0.93.0.5)
(0.93.05:
0.97
0.93
0.7
humanoidasze- giant
(0.5, 0.8)
0..5
(0.5.0.5)
0.5
0.5
0.5
0.5
puzzl 4xx5
(0.9) 0.5)
(..9.0.5)
(..5.055)
0.7
0.7
0.5
0.9
0.9
0.7
puszle-1x6-11
(0.7.0.5)
(0.7,0.5)
(0.5,0.5)
Table 7: Environment-speeffic hyperparameters under different h, n. ha configurations for DQC, QC, NS, OS. For DaC-naive, we use the same hyperparameter as the corresponding QC baseline.
from an unconditional (i.e., not goal-conditioned) behavior policy ти. In particular, ditioned flow policy T 8) that is parameterized by velocity field Uy S x R that is trained with the standard flow-matching objective:
an
[0, 1
RA
(BM = LE
[[lvs(s, (1 - u)= wa, u) - a
(29)
the official implementation where both flow policies (high-level and low- level) are goal-conditioned (and thus are trained with the goal distribution mixture specified by weurs wP geom "raj w (aaa)). The goal sampling distribution for training the value networks (for all methods) and the goal sampling distribution for the policy networks (for SHARSA only) provided in Table 4.
For SHARSA,
21

---

Published as a conference paper at ICLR 2026
Backup Quantile (Ka)
Distillation Expectile (Ka)
Backup Horizon (h) or (1)
Policy Chunk Size (ha)
Environment
cube-*
(0.5, 0.7,0.7, 0.0.0.9.000 0,93, 0.95, 0.97, 0.99)
(0.5, 0.8)
(5. 25
(1. [1,5. 25
Others
{0.5, 0.7,0. 0.9}
{0.5.0.8.
{5.25]
{1,5, 25}
Table 8: Hyperparameter tuning range for all methods. For NS, we only tune and n because the policy chunk size is always I and there is no distilled critic. Similarly, for QC, we only tune K and h = ha because the policy chunk size is the same as the critic chunk size and there is no distilled critic. For OS, we only tune Kig.
E ADDITIONAL RELATED WORK
Theory of Action Chunking Existing analyses for action chunking focus exclusively on the imitation learning setting (Tu et al., 2022; Simchowitz et al., 2025). While they laid out the theoretical foundation of action chunking policies for imitation learning, formal guarantees of action chunking RL are still open problem. In the adjacent field of stochastic optimal control (SOC), action chunking is related to control under intermittent observations where the observation inputs to the controller either unreliable (e.g., with Poissonian model (Wang, 2001; Dupuis & Wang, 2002)), or partially missing (Mishra et al., 2020; Yan et al., 2022; Noba & Yamazaki, 2022: Bayer et al., 2024). While conceptually related, these analyses are in the continuous-tin time setting in contrast to discrete- time transitions. To the best of our knowledge, we are the first to provide a formal analysis of action chunking in Q-learning. In particular, we identify the key open-loop consistency condition under which we quantify the exact worst-case Q-learning sub-optimality.
Theoretical analysis for reinforcement learning under unobserved confounding variables. RL. with action chunking policies can be seen as special case of RL. under unobserved confounding variables (Kallus& & Zhou, 2021) as the action chunking policies ignore the intermediate states during the execution of an action chunk. Prior analyses are based off either
iealll--neeeeneinnseeee sensitivity models (Kallus & Zhou, 2020; Namkoong et al., 2020; Kausik et al., 2024), confounded MDP models (Bennett et al., 2021; Fu et al., 2022; Shi et al., 2024), more general partially observable MDP (POMDP) models (Tennenholtz et al., 2020; Miao et al., 2022; Shi et al., 2022; Bennett & Kallus, 2024) where the confounding variables are modeled observable states. These models largely focus on characterizing either how much confounding variables affect the policy behavior (e.g., bounded odds-ratio between the policy with or without conditioning on the confounding variables (Kallus & Zhou, 2020)) or how much the observations reveal the confounding variables (e.g., the full-rank emission matrix assumption (Azizzadenesheli et al., 2016) and the weak revealing assumption (Liu et al., 2022) in POMDP). In contrast, our analysis specializes in action chunking policies where the unobserved variables are the intermediate states during an action chunk. This allows us to establish more specialized (and thus distinct) open- loop consistency condition under which we can identify the exact worst case bias (i.e., with matching lower and upper-bound to the exact value) for both behavioral value estimation and sub-optimality gap of the fixed-point for bellman optimality iteration, which are usually unknown under the more general models/assumptionn in the literature.
part of the partially
Hierarchical reinforcement learning methods (Dayan & Hinton, 1992; Dietterich, 2000; Peng et al., 2017; Riedmiller et al., 2018; Shankar & Gupta, 2020; Pertsch et al., 2021; Gehring et al., 2021; Xie et al., 2021) solve tasks by typically leveraging bi-level structure: a set of low-level/skill policies that directly interact with the environment and high-level policy that selects among low-level policies. The low-level policies can also be learned via online RL (Kulkarni et al., 2016; Vezhnevets et al., 2016; 2017; Nachum et al., 2018) or offline pre- pre-training on prior dataset (Paraschos al., 2013; Merel et al., 2019; Ajay et al., 2021; Pertsch et al.. 2021; Touati et al., 2022; Nasiriany et al., 2022; Hu et al., 2023; Frans et al., 2024; Chen et al., 2024; Park et al., 2024). In the options framework, these low-level policies are often additionally associated with initiation and termination conditions that specify when and for how long these actions can be used (Sutton et al., 1999; Menache et al., 2002; Chentanez et al., 2004; Şimşek & Barto, 2007; Konidaris, 1; Daniel et al., 2016; Srinivas et al., 2016; Fox et al., 2017; Bacon et al., 2017; Bagaria & Konidaris, 2019; Bagaria et al., 2024; Koch et al., 2025). A long-lasting challenge in HRL is optimization stability because the high- level policy needs to optimize for an objective that is shaped by the constantly changing lowelevel policies (Nachum et al., 2018). Prior work (Ajay al., 2021; Pertsch et al., 2021; Wilcoxson et al.,
22

---

Published as a conference paper at ICLR 2026
2025) avoided this by first pre-training low-level policies and then keeping them frozen during the optimization of the high-level policy. Macro-actions (McGovern Sutton, 1998; Durugkar et al., 2016), action chunking (Zhao et al., 2023) is another form of temporally extended action, special case of the low-level policies often considered in HRI., options literature, where a short horizon of actions is predicted all at once and executed in open loop. Such approach collapses the bi-level structure, conveniently side-stepping optimization instability, and when combined with Q-learning, has shown great empirical successes oflline-t--online RL setting (Sco al., 2024; Li al., 2025b). Action chunking policies need to predict multiple actions open-loop, which can be difficult to learn and sacrifice reactivity. Our approach regains policy reactivity by predicting and executing only partial action chunk, while still learning with the fully chunked critic for TD-backup. This design preserves the value propagation benefits of chunked critic without relying on fully open-loop action chunking policies, allowing our approach to work well on wider range of tasks.
23

---

Published as a conference paper at ICLR 2026
F WORST-CASE ANALYSIS
F.1 WORST-CASE ANALYSIS OF THEOREM
Theorem 4 (Worst-case AC Value Bias) For any h 1,7 E f), E [0, 1/2.. there exists an MDP M and weakly en-open-loop consistent 'D such that for some 8t E Supp(Po(s:)
(sc( - (sc (1 (1 =n))
7ea
(30)
Similarly, there exists M and E open--oo consistent D such that for some E supp((o(ss))
(5t) - (5c( = (1 7)(11 - (1 En)T
7EN
(31)
Proof is available in Section H.3.
F.2 WORST-CASE ANALYSIS OF COROLLARY
Corollary 2 (Worse-case Optimality Gap for Action Chunking Policy) For any h > 1, E [0, 1) [0, 1/2], there exists MDP M whose optimal policy induces a data distribution D' that is weakly ε -n-open-ooop consistent, such that for some 8, E supp( (st)).
(8t) Vic (8t) 7)(1 - (1 -ea)r))
7Eh
(32)
Proof is available in Section H.5.
F.3 WORST-CASE ANALYSIS OF ACTION CHUNKING Q-LEARNING UNDER WEAK OLC
Proposition 2 (AC Q-Learning under Weak OL.) For any > 1, E [0, 1), E [0.1/2) E 10,1,2), there exists an MDP M, weakly сΛ oo-open-loop consistent D and DD' with supp(Pp(ss, (ra+))) 2 supp(Pp (8t, (t)++)),, such that for some 8, E supp(Po (s:)).
V V*(st) vst(()) (st) Vat (8t)
7/
(33)
7-7
Proof available in Section H.6.
F.4 WORST-CASE ANALYSIS OF THEOREM 2
Theorem 5 (Worst-case Analysis of Q-Learning with Action Chunking Policy Off-policy Data) For any > 1, E (0, 1), e (0,1/5), E (0, cx/2), and Co E (0, 25, there exists an MDP M and strongly on-open-loop consistent data distributions D and D* with (upp(Pp(s1, (1:++)) 2 supp Pp- (52, (r:++))) such that for some 8, E supp( (s)),,
V*((s) Vit (st) (1 - (1 (1 2)
28 02
ENY
(34)
(1- 71 (1 En a)
where V* the value of an optimal policy and Vat the true value of - As c1, 02 +
V*(s:) - vot (st)
En?
[ (1 - 25 + 1- (1 sw
(35)
Proof is available in Section H.8.
F.5 WORST-CASE ANALYSIS OF THEOREM 3
is worth noting that although the global optimality variability condition looks similar to the strong open-loop consistency condition, they have completely different properties. For instance, nearly
24

---

Published as a conference paper at ICLR 2026
strong open-loop consistent data distribution D can have unbounded global optimality variability and a data distribution that exhibits zero optimality variability can also have large open-loop inconsistency. The implication of this is that while the closed-loop execution of an action chunking policy can be near-optimal, the same action chunking policy executed in chunks can be sub-optimal. We formalize this intuition the worse-case result below:
Theorem 6 (Worst-case Closed-loop AC Policy under BOV) For any h > 1, E (0, 1), 0% 0 E (0.
y-2" T[1- E 0, 7-yh 4(1-7") , E O, min( 0g (.00)) there exists M and I-y
satisfying the assumptions in Theorem such that there exists 8, E supp PD. (s+)), where
oh + 7 min 0 02) (1 7) 7*) a, (*) - Vac (x)) 2
(*)) V = h
C
(36)
The examples the proof of Theorem 6 (available Section H.13) serve as dual purpose- they not only show that our upper-bound in Theorem 3 is tight (since we can make o - 0), but also show that the sub-optimality of the action chunking policy can be made arbitrarily large. Furthermore, both the local optimality (0%) condition and the global optimality (0%) are necessary guarantee n being near-optimal. When any of them is large, Theorem 6 implies that there exists an MDP guarantee x' be near-optimal with alternative
where π is sub-optimal. As side note, "stochastic shortcut' assumption (a weaker form the global optimality variability assumption) and slightly stronger data mixing assumption. We refer the readers to Section G.4 for the formal results under this alternative assumption.
25

---

Published as a conference paper at ICLR 2026
G ADDITIONAL THEORETICAL RESULTS
G.1 COMPARING TO R-STEP RETURN Q-LEARNING
We now characterize the condition when action chunking Q-learning should be preferred over the standard nt-tep return backup. We start by introducing notion of sub-optimality:
Definition 4 (Sub-optimal Data) is 8, S---ubboptimal for a backup horizon length of n > 1 if
(s,, - (pp()) [Bestn + V (st+a)) 2 6n vs,, az E supp(Pp(s;,,a)). (37)
Intuitively, 5n captures how much worse the n-step return policy get compared the optimal policy incurred by the backup bias. Under such condition, we can show that the action chunking policy is provably better than the n-step return policy as long 6, is large.
Proposition 3 (Comparing action chunking backup and n-step return backup) Let D be strongly En Ex-open-loop consistent and 8, o,-subboptimal, and supp(Po(ss) 2 oupp(Po- (st)). Let पे. St + arg max,, Q (st, az) be the policy learned from D, via n-step return backup:
Q (st, E
[Reann + n (sttns क. (Sttn))
(38)
Then, for all 8, E supp( Po. (st)) (and with Hn = 1/(1 )")
Ve (st) - n (st) 2 on
ENY
2
n" 1-7
1 - - 22x)7h 1- (1 en Ex)T
(39)
2 6 Hn - 33, H.
The proof of Proposition 3 is available in Section H.10. Notably, for h, long as D is more than (3sa H) sub-ooptiaall, the value of the action chunking policy is provably better than the value of the n-step return policy. It worth noting that Proposition 3 uses the nominal value of the n-step return, which may be lower than its actual value. We refer the readers to Section G.3 for examples where the n-step return policy is provably worse than the action chunking policy.
G.2 E-DETERMINISTC DYNAMICS IS WEAKLY OPEN-LOOP CONSISTENT
To provide some intuitions what this open-loop consistency implies, we discuss concrete family of MDPs where any data distribution from these MDPs is (weakly) on-onen- consistent (Proposition 4, with proof available in Section H.15).
Definition 5 deteriittriiiiitccc Dynamics) A transition dynamics T is a-deterministic if there exists a deterministic transition dynamics represented by function f : S A - S and another arbitrary transition dynamics T x A + As, and T is a combination of f and T:
T(s' a) = (1 - =). =T(s'' s, as, S,
(40)
Proposition 4 (Deterministic Dynamics are Weakly Open-loop Consistent) If transition dynamics M is s-deterministic, then any data 'D collected from M is weakly En -p-en-olo consistent with respect M for any h E N+ long as EY 2 3(1 (1 e)--1) 1).
e-deterministic dynamics acts like a deterministic one most of the time (with - probability) and non-deterministic one occasionally (with e probability). This bounded stochasticity allows the results of taking an action sequence (of length ht) open-loop to be deterministically determined the event that the deterministic dynamics is 'triggered' (with joint (1 =)^ probability across h time steps). is clear that under such event, there is no gap between the 'replayed' open-loop data PS and the original data distribution Pp. and as result there is also value estimation bias under this event, and thus intuitively we can bound the value estimation error by a function of the probability that the stochastic dynamics is 'triggered" (i.e., with
- probability).
26

---

Published as a conference paper at ICLR 2026
G.3 CONDITIONS WHEN 7-STEP RETURN POLICIES ARE PROVABLY SUB-OPTIMAL
Definition 6 (Near Optimal Data)
We say D 8, optimal for backup horizon length a E N+ if (s, az) - (p. [Reann + 7 - 8n. Vst, supp(Pp(s),a))). (41)
In Proposition 3, we have shown that the value the learned action chunking policy better than the nominal value of n-step return policy with value gap of 3e, H. However, the actual value of the n-step return policy maybe better. Here, we analyze the worst-case performance of n-step return policies.
Proposition (Worst-case analysis of n-step return backup) For any n N+ 8 E (0, 7 177-7" -")" and (1, (1-)) there exists MDP M, and opttmmal data distribu- tion D with supp(Pp(s;, a;)) 2 supp( (st, az)) such that for some supp(Po- (s:)),
(s) Vs)(s) = 8n
(42)
1-7
and for all 8 supp(Po>(st))
(*(s) (s).
(43)
The proof (available in Section H4) provides concrete examples where n-step return policies are worse than action chunking policies. The implication of this result is that the sub-optimality of the data distribution (as characterized by 8, and 8,) is generally independent from the open-loop consistency (as characterized by En).
G.4 CLOSED-LOOOP EXECUTION WITHOUT STOCHASTIC, SHORTCUTS
In this section, we provide alternative way of bounding the sub-optimality of n the closed-loop exccution of the learned action chunking policy t In particular, we characterize two conditions when closed-loop execution of an action chunking policy can help mitigate open-loop biases.
Our first condition is based on the key observation that only a certain type of value overestimation harmful for closed-loop execution of the action chunking policy. The source of this type of value overestimation comes from stochastic shortcuts:
Definition 7 (Stochastic Shorteuts) We say M free of ún -toshhcaii shortcuts for horizon
h if
V*((++h) + REtth - V*( S On,
(44)
V8sa41411 Ouith
II
P(s+4441 St42, O,
km
where V* is the value function of optimal policy in M.
Intuitively, stochastic shortcuts are low-probability (but plausible) paths (i.e,, 8t, at, a....... .14+k) h in the MDP that lead to returns that are much higher than the optimal expected value (i.e., V*). These stochastic shortcuts are particularly problematic for action chunking value backup because the chunked critic/Q-functien cannot distinguish between a low- cow-probability stochastic shorteut and an optimal (or near-optimal) closed-loop trajectory, leading it to erroncously favor the shorteut.
Our second condition is that our data distribution is mixture of some data distribution that is collected by some optimal closed-loop policy (D*) and some data distribution that is collected by an open-loop policy (Dº, and thus is open-loop consistent). Intuitively, this condition makes sure that any non-optimal trajectory
be accurately estimated by the action chunking value function V and the bounded mixing ratio restricts the amount of bias that the V has on the estimation of the optimal trajectories when the open-loop action chunks (e.g., in DD) coincide with the action chunks the optimal data (e.g. in D*), We formally define the second condition as follows:
27

---

Published as a conference paper at ICLR 2026
Definition 8 (Open-loop Data Mix) We say D is a-open-loop mixed for some / E [0, 1), can be decomposed into two data distributions D*, D as
Po(- 8t) = 3P (: 8e) + (1 3) PD 8t),
(45)
where D any data distribution collected by an optimal closed- loop policy nt and D is any strongly open-loop consistent data distribution, and
ag
PDo [oEL+h E
8e)) 8t) S (1 a) 3)
(46)
With such data mixing assumption and in the absence stochastic shorteuts, we can show that closed- loop execution of the action chunking policy (i.e., only executing the first action of the action chunk) recovers a near-optimal closed-loop policy:
Theorem 7 (Closed-loop Execution in the Absence of Stochastic Shortcuts) D is a a-open-loop mixed and M is free of 8n -toschchat shorteut, the value (V*) of the one-step policy (л*) as a result of the closed-loop execution of the action chunking policy nhe learned from D admits the following bound for all 8, E supp PD (st)):
(*(()) - (+)) S
a (1 7) - 7hhh al)aa) a)) + (1 7) -h)
(47)
proof is available in Section H. Intuitively, the second condition measures how much percentage of the open-loop data has overlapping support as the optimal data. With some algebraic manipulating, assuming the worst case of Equation (46), we can rewrite the data mixture as
D = - a1)- + aDD + -ByDo
(48)
where 3 Laa supp f.. (.1 st)) S supp( Pa- (.1 8t)) and (upp(Pre... (. 8t)) n supp(Pp- (- st)) = Ø. As the bound is independent of 3, it becomes clear that D plays no contribution to the optimality of action chunking policy learning. The only harmful portion of the open-loop data distribution is Di, as the action chunking Q-function cannot differentiate these open-loo actions in Di from the closed-loop optimal actions in D* This is reflected as the first term in our bound. The implication is that even if the data D is arbitrarily sub-optimal (with 3 O, and hence arbitrarily bad for n-step return policies), π remains near-optimal as long as the 'in-distribution open-loop data Dí is relatively low in density compared the optimal closed-loop data D* (i.e., is small).
Furthermore, our bound is independent of the open-loop consistency of the data D. As a, 0 - O, closed-loop execution of the action chunking policy exactly recovers the optimal policy. In contrast, even when a, & + O, open-loop execution of the original action chunking policy (i.e., THt can suffer from the open-loop inconsistency the data D: its value error can only bounded by (I-7)(1-"))
(as shown in Theorem 1), a function of E,, (the strong open-loop consistency of D).
28

---

Published as a conference paper at ICLR 2026
H PROOFS OF MAIN RESULTS
H.1 UTILITY LEMMATA
Lemma (Mean value theorem for conditional probabilities) Let P1, P2 E Axxy and P(r,y) y) == P1 v) + (1 (z y) and there exists a > such that ( -
a, a, E V Then, there exists E y and ā S a such that
P(- v) - & v) + (1-))22
(49)
Proof.
P(x,V) (()) (1-
P(y)
(1 (L) Pz (z 8) + (1- Pz) v)
(50)
where 8(v) = a(r) Po)) &
Pa()) We now prove By V ā S a for Equation (49) to hold by
contradiction.
We first assume = B(y) > a, Vy y. Now, substitute Byy) and integrate both side by y to obtain
Pl > ( a(1 [1 (z))) a(y) aá(y) + -dlly) a,
(51)
(52)
which is contradiction to the condition d(y) a. Therefore, there must exist E y with à S a such that Equation (49) holds.
Lemma 2 (Expectation difference for bounded function and TV) For two distributions P, A and two bounded functions f.g. X + [0, 1]. if the TV distance between P and Q is no larger than and || oo I ê under supp(P) n supp(Q), then
[E>+)](())] - Ee=o)o(=)]] < (1 =)8 +
(53)
Proof.
Let's decompose the probability mass of P and Q in terms of de, drq, do X - R2 as the following:
P(x) = dr(x) + dro (x);
(54)
Q(x) d=Q(=)+ (2
(55)
The f dp(x)dd maximizing solution is
dr(x) max max (P(x),((x)) Q(x)
(56)
do(x) max(((z),((()) P(z)
(57)
drQ(x) P(x) + Q(z)) max(P(=).Q(z)).
(58)
It is clear that under this decomposition,
/ |do(s)de= / /dg(x)ddo d = ( ddr(())dr=
(59)
1 11g(2)d==1-E27
(60)
29

---

Published as a conference paper at ICLR 2026
Now are ready to bound the expectation difference:
p(((e) - Ex~olo(=)]
(J dp(x)f(o)dr - de(w))(())x)) + (elaa))+((((araa ((F((=)(((()-(())))
/ dr(x)f(z)dr / do(x))(2)dl)) + (()()))(()) - g(x))dx
sup f()) / - -oeddd-innfoo inf gnf)(())))))))((())) / (((r)d / dz())dx-inf inf f(x) / dd(())d))
max
sup [f(x) - ((x)] / dpo(t)do (x)p@(x)00)
+
sup sEsupp(P)naupp(()
1f(x) - s(w)1 (1-E)
+ |[f gll
I É(1 - 6) +
s(1 - 6) + d
(1 8 +
(61)
as desired.
Lemma 3 (Total variation under event conditioning) Y E Dy and any E V.
For random variables X
Ax and
DPv(P(X = P(X)) S P(Y = v)
(62)
Proof. Let P(Y y)
Drv(P(X Y = P(X))
= 2 IP(x) -P(x 1))dd
1
/ P((x Y
Y =
= =)) uP + P(x Y * P(V # y
2
(63)
1 1-p
/ (P(z Y 5y) Y yy))|dr
2
= (1 p) Drv(P(X y= P(X Y + v))
< p
Lemma 4 (Data Processing Inequality for /-divergence (Csiszár, 1967)) For two random variables A, Ax and deterministic function f X + V, and == g(A) = (BB)
(PA PB) Z (Pc Pb).
(64)
Since TV-distance is a f-divergence with a 11, have
Drv Pв) 2 Drv || Pp).
(65)
30

---

Published as a conference paper at ICLR 2026
Proof from Wu (2017).
Dy || PB) = ER~Pa [[PA(z)/PB(t)]
EPDD (S PBD]]
(a.y))P [EPsI (F
2 Ry-Po [ʃ (Ex-Papee Pac((:.))/ano(evl)]]
(66)
Ey~Po [f (Be~Palas, (Pc(W)/PP()))
Eyy-r [f (Pc(W)/Pb(w))
= (Pc || Pp).
31

---

Published as a conference paper at ICLR 2026
H.2 PROOF OF THEOREM 1
Theorem (AC Value Bias) Let Vac :S - [0. 1/(1 7)] be solution of
Vac(St) = Es Bes
[Resst + 7 Vac($e+h)
(11)
with Beath Eie tssp If 'D is weakly en en-oee-looop consistent, then for all 8, supp( (s:).)).
, ar) and Vae is the true value of Tac 8t Po(orath
st).
(ac(se) (sc(st) -
7Eh
S EnHHH.
(12)
Since is op-open-o consistent in state-action for Ní' < h, the state-action distribution leading up to step h admits the following bound:
Proof.
DTv Or4h ss) || (S acth st)) S En
(67)
Let Rec+h Ek=o ff (s ac+x) be the h-step reward distribution. Then the difference in h-step reward is bounded by
k=
[Po(1ss) [Reet]] Bpg((1.)) (Bettn))
M Epp B a(H))) Epp (ww (68)
N Eke
k=1 where the first inequality uses Lemma 2 and the fact that TV distance is bounded (Equation (67)).
Since D is E ope---ooo consistent for h in state, we have DTv (Po(s 8x) Ps(s-- 8x)) & En:
(69)
which can then be used bound the estimation error using Lemma 2:
E Batth ~ Pro(stalls))
E Exsss Pggoreala)) [Vo(()+))]]
(70)
1 y + [1-
sup #
((c(5)) - Vac (Sc
For all 8t E supp(Po(st)),
(st) (sc(
E (Polle)lReett - Bpg(t1e) -Baaea)--Bgrraa)) [Reata]
t BenarPo(aaull (oc(oe44))
Eest [[o())+))]
(71)
h-1
( Eh + eeen
1-7 + * -nh)
[oo(se4h) - Vao ((44)))))]]
sup 41l Esupp( Po (o)
Since the support of 8t- st subset of the support for 8t by Assumption 1, we can recursively apply the inequality to obtain,
(sc) Vac(st)
1
MMh --77
eh
(1 k
W=1
(72)
7Eh
(1 (1- (1 - =))
as desired,
32

---

Published as a conference paper at ICLR 2026
0 m
6 1r-1
Xa
Xi
Figure 7: A 2h-state MDP that is constructed to meet the upper-bound in Theorem 1. The data distribution D that achieves such an upper bound collected by the optimal policy: <(X;) = 1. (X,,) = O.
H.3 PROOF OF THEOREM 4
Theorem 4 (Worst-case AC Value Bias) For any h h> 1, E 1), [0. 1/2]. there exists
MDP M and weakly en-open-loop consistent D such that for some 8, supp(Pr(s:)),
(x)( (8c)
7Eh (1-=))))))
(30)
(1 (1->)(1
Similarly, there exists M and E -n-o---oo consistent D such that for some E supp((pp(s))
Vac(st) - Vac(st) = (1 (1-7)(1- - (1-11-(1)))4))
7Eh
(31)
Proof. Let ô E 1] be any value that satisfies En = 26(1 - 5). must exist because En E [0, 1/2]. Let us define a MDP that has S = 2h states, = [Xu,X,, X1, actions, A (0, 1), and the following transition function T and reward function (see diagram in
1. 2). and A
Figure 7):
T(X#+) Xi,a) T(X Xi,a)
Va {0,1), E (I, ... -h 2} Va E {0.1), E (..... h 2}
T(X+11 (x,a) = T(1++1 X, 1166 -, 8,
T(Z Xn-1, 1) = ( a 0)
T(X, X 0) T(X, a =1
(73)
r(x,, = 0) = r(X,,
1,
Vi t....... ... 10,.,,, 1
18, = 1) = r(Xi, x
= 0) 0,
Vi E (0, ... .0,...,h. 1}
r(Z,a 1) rr(,, = =0
T(z Z, a 0) = T(2 Z,a = 1)
Now, we assume that the data D is collected by the optimal closed-loop policy where
T(x) 1, +(x)) O.
(74)
First, we check D is ед En-open-loop consistent.
We can show that by computing the distribution for Pp Ceti 8t Xx)) and Pe (S ari 8, Xn) as follows:
Po(er = X =0 0 Xo)
Po(A11: = 84. =1 Xo)
0
Polsis = Xi,octi =0 - Xx)
Po(nci Xi, =1 Xn)
8
(75)
82
(1 (1-8)0 8)0
P8(4 = X 0 Xx)
PB = 8 Xx) Pa X, Xp]
Pa X2.9 [
8(1 8)
(1 - 8)
33

---

Published as a conference paper at ICLR 2026
From the calculation above, it is clear that
Drv ( ar-i st) PD(stti: acti 8t)) = Ehs Vi (1,2, ... , ,hh 1}. 1}. From the computed values of (s (st+n = 8t = Xo) = 0, PB (st+A~ = Z Xo) =2(1 -8)8 En.
(76)
st) and 111--4h-,,04hh1 8t), we can derive
(77)
From the calculation above, it is clear that
Drv (rv(PP(ssH)) (8+41 8x) PD 8x)) = En.
(78)
Up now, we have checked that D is Ey op-ooenloo consistent. Now, we are left with analyzing Vac and Vac- With some calculations, we can obtain the following:
E Bpg Rea+h] 1 - -x)(7--9
1-7
1
(79)
(Xo) 1 1-9
(2) = O.
Now, we ready compute Var (Xo):
(Xo) (1 70) cn(7 - 7/1) + [(1 9n (a) ENN
(1 7)
(80)
n1 En (7 7h))
(1 71 7m =N))
Finally, with No E supp(D), we obtain the desired value difference
(Xo) Vac (1 (1--)) 11-7)(1mm 1h En))
E
(81)
By symmetry, flip the reward value (i.e., 0 1 and + 0) to construct the example such that
(Xo) - (Xo) (1 (1-7)0 (1-7)(11 71 =k))
ENT
(82)
34

---

Published as a conference paper at ICLR 2026
H.4 PROOF OF COROLLARY
Corollary (Optimality Gap for AC Policy) Let D be the data collected by any optimal policy x². If D is weakly En Ex-open-ooop consistent, then for all sn E supp PD. (s,)),
PEh (st)) (st) S (s:) (si(s:) S (1 7)(1 (1 (--=))))))
S HHHH
(13)
where V* is the value of the optimal policy n², Vae is the true value of the optimal action chunking policy, and Vn is the true value of the action chunking policy from cloning the data D*:
(as:th (t)) st + Po. st).
(14)
Proof. Let Vac be the fixed point of the following equation:
Vac(s:) = E B (-Ix)) (Berth + +7 (ac(se+h)
(83)
where again Recth = EV' nth *r(ss,ap), The value of the optimal policy is the fixed point of the following equation:
(si) =
E, 1. (1a)) [s and + (s
Pr (H) (r)s as) + 77(5411,4411) + (V
(84)
Beath + (sc-nn)
which is equivalent to fixed-point equation for Vac. Therefore Vac = V* By Theorem 1, we know that the true value Vae of the action chunking policy #ae that clones DD is close to Vnc. More specifically, for all 8 E supp(DD))
PEA Vao(se) - Vac(se) 5 (1 (1-7)(1
(85)
(1-))))1)
which means that
V*())) - (xc) 5
7EA
(86)
(1 - 71 =11
where we can remove the absolute value operator because V*+(s; is by definition always at least as large as (st). Since the optimal action chunking policy, by definition, attains equally good or better values (over S) represented by Vac, and the optimal policy x* also attains equally good or better value (i.e., V*) compared to that of the optimal action chunking policy a (i.e., V v the following inequality holds for all 8 supp(D*):
V*(s:) 2 (xt()) 2 (ac
(87)
Therefore,
V (st) (V.))-V(ss)) < Vac(st) (*))s)
(88)
(1 (1- 71 (1 k))
as desired.
35

---

Published as a conference paper at ICLR 2026
H.5 PROOF OF COROLLARY
Corollary 2 (Worse-case Optimality Gap for Action Chunking Policy) For any h > y
[0, En [0, 1/2]. there exists an MDP M whose optimal policy π induces a data distribution
D that is weakly En -open-loop consistent, such that for some 8, E supp(Pp> (s,)),
(*(s) Vic (st)
7
(32)
- (1-7)(1 -x))²)
Proof. To show this, we need a slightly more complicated MDP (compared to the 2h-state MDP use in the proof Section H.3). The MDP we construct for this proof is (3h 1)-state MDP as illustrated Figure 8.
020
1021
X4-k
Be
Figure 8: A (3h- 1)-state MDP that is constructed to meet the upper-bound in Corollary 1.
The optimal policy we pick described as follows:
T (a X2)) 1/2
ा (a X2) = 1/2
(89)
T' (a A,) =1
n (a = 0 B;) = 1/2
This induces the following state distribution.
(5 = A; | = X,) = Pp- (Stti B; 1//1: 8t =Xo)
Pf. (s A4 st Xo) Pp. (st+s B, St = Xxo) 8/2,
(90)
Ppr (st+i = Xx st X.) = P (stti X; = Xo) 11 8,
and fully factorized distribution for the action chunk,
Pg (ar4: 0 0100)) s.)
Po. (ac+i St, (nitee)
(6am0 + 8am1).
(91)
36

---

Published as a conference paper at ICLR 2026
Now, derive the condition on ò when the optimal data D* is EY -o-open-loop consistent. We start by calculating the TV distance discrepancy for the future state-action distribution:
Drv Popen
(se+is at+i |
Pb. (st+s; Oc+i st))
0
6/2
6/4
6/4
1
(1 80/2
(1
61-6
82/2
(1 81-
(92)
2
8/2
0
6/4
6/4
1,1
6/2.
In the second line of the equations above, each row in the matrix corresponds to a distinct action ar+i E {0,1) and each row in the matrix corresponds distinct state st+4 {Ai,, X,, B.}.
Next, we calculate the TV distance discrepancy for S1tAi
Drv
popen D (8tth 8t) || Pb. (8eth 8t))
|[ 0 - -6622 8/2]1h
(93)
2
5/2.
In the second line of the equations above, each element in the vector corresponds to distinct state
814h {Xo, 2}. Up now, we have concluded that 7 is (6/2)-open-loop consistent.
Due to the symmetric structure of this MDP, it is clear that any action chunking policy Tao (Xo) aut+h with aerthh E {0, 1} optimal and achieves the following value:
A 1-7 + + (NV
V (Xa) = 1 + (1 6/2)
(94)
(1 - + (1-5//2(7 - 711)
- (1-
The optimal closed-loop policy can achieve the maximum possible return
(*xo
(95)
Therefore, with -//2, the optimality gap achieved by this (3h 1)-tte MDP is
(*X) Vi (Xo) = (1 (1-7)) (1 (1 =x))4)²
(96)
as desired.
37

---

Published as a conference paper at ICLR 2026
H.6 PROOF OF PROPOSITION 2
Proposition 2 (AC Q-Learning under Weak OLC) For any h > 1, 10, E [0, 1/2),
En E (0, 1/2), there exists an MDP M, weakly En En-open-loop consistent D and D' with supp(Po($1, (ra+h)) 2 supp( (DD (8t, (tzt+)),, such that for some 8 E supp( PD (8:)),
V*(st) Vit (st)
Va (s;) VaL (st)
(33)
1-7
Proof. To prove this theorem, show example where the optimal action chunking policy can be highly sub-optimal in the absence of the strong open-loop consistency condition.
We define MDP follows. Let S B, D, E, Z) and A {0, 1). Define the transition dynamics and reward function as shown in the diagram below (Figure 9):
=
D
B
=
=1
r=0
1-8
A
Z
r=0
C
0
a
E
Figure 9: A 6-state MDP that is constructed to illustrate the pathological failure mode of action chunking Q-learning under weak open-loop consistency.
where 8, ē E [0,1) 1) are real numbers and dotted lines denote stochastic transitions. The reward function depends only states (r(A) r (B) r(() = r(F) = 0, r(D) 1, and r(G) = c). Assume that the dataset collected by a policy To defined as TD(A) = (with probability 0) or 1 (with probability (1 0)), (B) = 0 (with probability 1), (C²) = 1 (with probability 1), and (D) = (Z) = (E) = (with probability 1).
Let π² be the deterministic policy with x (AA 1, (C) = (E) 0. Then, it is clear that
supp(Pp(s,, (s.++))) 2 supp(Pp- (5t, dm:++))) = {(s = A, a = 1, 1,4+1 = 1, 1.4th21th = 0)}.
Now are left to show that D is EY En-oon--ooo consistent.
Since D, E, 2 are all self-loops, we only need to analyze the first two actions in the action chunk; the rest is all 04+2:t+ 0 and does not affect the value.
We can compute the following:
PA(A, (0.0)) = D, R(A, (0,0)) = 7
Pb(A, (0, E, R(A, ((A,(0,1) 1)) 207
(97)
PA (1, = E, R(A, (1, = 207
where we denote action chunks tuple and slightly abuse notation denote deterministic outputs of Pp(- 50, 40.2) (e.g., Po A, (0, 0)) = D indicates that all length-2 trajectories in D from state A with dg 01 have 82 D with probability 1).
We can similarly compute the marginal state distributions as follows:
(D A)
06
(z
A)
=
0
(98)
Pp(E
A)
1 06
38

---

Published as a conference paper at ICLR 2026
and
PD(s141 = B.a: a, = 0 5 = A) Po(s141 C,a. = 0 A)
Po(s--1 = B, = 1| 81 = A)
08
0
(99)
PD(5441 = C,a.
A)
0
08
The marginal probability distribution of the action chunks is
Po (0,0) A) 06,
Pa(a0.1 = (0, 1) A) (1 8)0,
(100)
Po(ao. (1,1) A) 1 0.
The induced Pe is then
P S5thh D 8t A)
082
P Se+h = Z St A)
(1 11-000
(101)
P (stth E
8t A).
(1 8)20 + (1 0)
and
Pg (s
B, 3,0+11 =0 0 8t A)
p8 = B,04+1 = =1|8 8t A) p = C,, =1
P (3 C,a,+1 = 8: = A)
8: A)
(102)
8²0
(1 -1- 0)60
810
(1-
We can then compute
DTv Drv(PB(su41, 1441 8, A) || Pp(S++1, ac-1 8; A)) 205(1 6)
3
(103)
DTv(P2(S(4h St A) Po(s44n |8.= A))
01 6) 2
Note that we only need to check + time step for the state-action distribution because
Drv (PD a-- St = A) || PD(st4N' a++h 8t =A))
(104)
(Po St A) PD(st+h St A)), Vh > 1,
coming from the fact that the rest the action chunks are all constant 0 (i.e., a+42:th = 0).
Now, set 8 to be solution of En 2(1 8)88, such that D is En oppe--ooo consistent as required by the assumption.
Next, analyze the action chunking policy at learned from this e op-n--ooo consistent data distribution. We start by calculating Qt follows:
(0,0)) 0))
7 1 ?
(A, (0, 1)) 1-
(105)
2t, (A, (1, 1))
The optimal action chunking policy is then * (0, 0) because 1. The true value of this action chunking policy is
8y -
vtt (A)
(106)
As long as ē 8, the optimal strategy in this MDP is to always choose (ag, a;) (1, (), in which case the agent receives a constant return:
V*(A) -PAA)VV -
(107)
39

---

Published as a conference paper at ICLR 2026
The optimality gap in this example is therefore
V (A) Vat (A) (2 - 6) 7 7
(108)
Finally, we solve en =2(1 - 5)60 and pick the smaller solution:
2
8= 2
(109)
If we set / = 25 and ē = c + 2. then we get
07 (* Vac (A)
(110)
7²
as desired.
As small extra note, the last step is where the assumption &q 0 becomes necessary, since otherwise the term / 0 (with @ = 28a) would be undefined.
40

---

Published as a conference paper at ICLR 2026
H.7 PROOF OF THEOREM 2
Theorem 2 (AC Q-Learning under Strong OLC) D and D' are strongly ex-open-loop con- sistent and supp( (P (rs+n)) 2 supp( PD (8t, (m)+h)), then for all ot supp D (s))),
(*()) - Vac (5t)
2 Σ 1-7 1- (1-11- 2
E
+ - (1 - En)r S HH,
(16)
where V* is the value of a closed-loop optimal policy and Va is the true value of m
Proof of Theorem 2. We start by constructing a bound between Qte and Qie the solution of the following bellman equation:
(Sc Orse4h)
Reath + +
mnxx ar4hi542h
Qze (St4hr ac4hat42))
(111)
Intuitively, Qic is the Q-function of the optimal action chunking policy whe that can be learned from D. Because (D) 2 supp(D"), Tae is at least as good as Tac, the action chunking policy obtained by behavior cloning D Bounding the difference between Q and Q allows us to leverage the bound in Corollary to form bound between Vut and V*.
Since D is strongly En opee--ooo consistent,
Dry (T(s44
Shn E (1,.., 1}. St, 0c:+4:) PD(s++ (1, En, Wh'
(112)
Since D* is also strongly ex-open-loop consistent,
Drv(T 8t, (1:at))))) || Pp- ($14k/ sts aee+h)) S En, Vk (1,".,,h 1}. (113)
Using the transitive property of TV distance, we have
(Tv((D(S St, aeath) || Po- ($141) Sts (mc+t)) S ZEh, Vh' (1.-.. , 1}. (114)
Now, for the h-step reward, we have
Erot-baaaa [Reeta]
- Epp (looeees)) [Rettull
Y W' DTv((P(S04N' se: (aath) Po- (St4M) Sts (mt+4))
(115)
W=1
2(7 Men
1-7
Similarly, for the value h-step into the future, we can use Lemma 2 to obtain the following bound:
EEL PPL_A-Polocsll)) [V*44)))] E (susalse) Vit (8c4m)) < 2cn + (1 - 28n)
(116)
(((++))) Vit (#242)
sup
We define (st. (+tth) to be
(Strart = Eppr (111) [Reesh 2 AV (st4n).
(117)
is clear that
V*(ss) EautsaaPr 1Q' (te)
(118)
41

---

Published as a conference paper at ICLR 2026
Combining the bound for the histep reward and the bound the value for Sith, for all st. acith supp(Po- (st, (1:4+h)),
(sts thethh) = Q"(((: (r:eth) (Sc (ea+hh)
2EAT + 2(7 ) + (1 2End"" (v*th))) Vat (sLth))
1-7
2ENY
+ (1 - -2En)"
Epp (s1+h ach442h))+++2) sup acthie+2h
Quc(SLth. acthet2a))
1 1-7
2EN7 + (1 -2xn)
Epp (ettsss As
Q (s acthe+ah)))))
1-7
sup
arthin42h
2ENY
$ 1-7 + - -2nnm
sup #o4h.24
((s a++he+2a))
(119)
which can be recursively expanded to obtain
2047
(*+))) - (st) I
(120)
(1 (1- - -1- k
By Theorem 1, for all 8 E supp(D).
(se) - Vat (8t)
S (1 - 71 - =xh
E
(121)
Combining the two inequalities above, for all 8t E supp(D"),
V*(x)) - Vat (8t) < 1-7
SNY
2
1
(122)
1 (1 -2 )/
(1
42

---

Published as a conference paper at ICLR 2026
H.8 PROOF OF THEOREM 5
Theorem 5 (Worst-case Analysis of Q-Learning with Action Chunking Policy on Off-policy
Data) For any h > 1, 7 (0, e), (0,1/5) e0,en/2), and C2 E (0, 2), there exists an MDP M and strongly En oo-ep--oo consistent data distributions D and D with supp(Pp(s;, (1:++)) 2 supp(Pp (pop(Pp->ae (5t, amc++)), such that for some St supp(Pp- (s:)).
(*(()) Vat (st)
ENY
(34)
(1 () (1 28h )²x)7²)
(1-
(1 En (1)7)²²
where V is the value of an optimal policy and Val is the tee value of t As 01.02 +
2
(*(ss) (st
(35)
-1- En)
The examples in the following proof of Theorem 5 (available in Section H.8) provide insights on the factor V Vt S 3e, H (with H = 1/(1 7)), H 1/(1 1/(1-7") is necessary. In particular, the worst can be roughly seen combination of the two main results that we have presented so far:
1. V Vie x En HH (Corollary 1, Corollary 2):
the optimal action chunking policy is (en (1? --ud)-)--uoppimmaa due to its inability react to environment stochasticity, quantified by the strongly-E, open-loop consistency of DD*
2. V - -v. E (a transformation of Theorem 1 and Theorem 4 on the optimal action chunking policy vac): the value under-estimation bias can incur another factor of e H H bringing up the sub-optimality of V to at most 2 H, and finally,
3. V+ Vat as H (Theorem 1, Theorem 4): the action chunking value function may ac prefer overestimated action chunking policy ग where its actual value is again H from its estimated value, resulting in a total sub-optimality of H. H.
Our construction (in the proof of Theorem 5) directly builds on the above insights by using a 2-part MDP where the first part corresponds to an -undereeitttt action chunking policy that has a (e,, -optimmllyyyyy gap from the optimal closed-loop policy and the second part corresponds to an action chunking policy that has a (3s, H H) -opiimally gap that is preferred by the value function.
(Er,
Before start main proof, we first introduce Lemma that helps simplifies the inequalities.
Lemma 5 (Optimality gap comparator) For any 7 E [0, 1) and 0 < E1 < < 1,
E1
E2 1-
(123)
1 1- - 01)7
22
Proof.
0 < (1 ()2 61)
82 82 - E1 + f6
(124)
E2 eхî + 81627 - s1 + + 21122
E2 (1 - E)) - 51 - - =2)
Since 1 - (1 -1i)9 0 and - (2) to get
1 (1 22) 0, we can divide both sides by (1 -1-1111111 -1--
82
E3
0 < 1 1- (1 32)
(125)
1- (1 -
as desired.
Now, we begin the main proof follows.
43

---

Published as a conference paper at ICLR 2026
Proof of Theorem S. We prove by constructing the following (2h + statt MDP where the agent can take any of the three actions {0, 1, 2 at each state (see diagram in Figure 10).
Notations: start by introducing some abbreviations for all action chunks that appear in this proof:
ai (0, 0,0, 0)
at:++A (0, 1,,, 00,00,000000...00000000.0 0)
Caa+ 20,2,0, 0)
(126)
A (1,1,1,...,
Lit+ (1,
autth (1, 11,2,1.....
The first three action chunks die+hh: Gltt Curth are only possible in the top branch and the last three action chunks dz G C2a4+h are only possible in the bottom branch because the first action in the action chunk deterministically divides it into the two branches.
Among these action chunks, it is clear by inspection that Tac (Xo) (0, ...0....)).... 0) is the optimal action chunking policy, and thus we directly use 'x" to denote a = (0, 0, . 0). a is also of great importance:
will show later, go ( (X)) atx4hh- The actual values and nominallestimated values for these action chunks are (V ac, V ac Ve, Vix) and (Vat, Vo, V V.A acc Vax) respectively. Much of the focus of this proof is calculate the optimality gap, which is the difference between the optimal closed-loop value and the action chunking policy value (either estimated or actual):
actual optimality gap: V*(X)) - (Xo
(127)
nominal optimality gap: V*(X)) VLl (Xo)
(128)
High-level proof sketch: The MDP contains two branches: top branch where (as we will show) both the optimal policy x² and the optimal action chunking policy ahe take, and bottom branch where (as we will also show) the learned action chunking policy m takes. The key idea of the construction is that for the top branch, we have
2E (Xx) - (Xoo ~ (1- (1-7)(1 (1 22x)
(129)
and for the bottom branch, we have
(Xoo < Vat x (Xo) + (1- 71 -1
ENY
(130)
Combining these two together gives
v*(x)) Vit (Xo)
ENY
2
+ 1 (1 En)h
(131)
1-7
(1 2
We use "sg" because the equalities are not strictly achievable but (as we will show) can be made arbitrarily close.
The proof can be roughly divided into the following steps (we 'es' to help illustrate the high-level idea below and use more precise argument in the actual proof):
1. MDP description: we formally describe the transition dynamics T and the reward function r for each state-action pair for both the top and the bottom branches.
2. Strong ox-open-loop consistency of D* we then check the strong open-loop con- sistency assumption for D*
3. Data distribution Deop for the top branch: we use a mixture data distribution from policies to construct Drop-
44

---

Published as a conference paper at ICLR 2026
4. Strong En on-open-loop consistency of Dtop: then check that the constructed data distribution of the top branch satisfies the strongly open-loop consistency assumption. Note that can do separately for the top and the bottom because these two distributions have non-overlapping support in death
5. The optimality gap and value estimation error for the top branch: we
prove that (Xx) V (Xo) = (I- (1-22)))))))) and (*(x)) V (Xp)
2 (1-7)(1-1125aa))) 1yh) and the other two possible action chunks Ggtth = (0, 1, 0, ...) and
a Gor4h = (0,2, 0, ...)) both admit lower estimated values compared to dz Vo (Xo) < (Xa).
(Xo) and Vie
6. Data distribution Dhottom for the bottom branch: we again use a mixture data distribution from two different policies to construct Dootoom
7. Strong AΛ op-open-ooop consistency of Dotttmm then check that the constructed data distribution of the bottom branch satisfies the strongly open-loop consistency assump- tion.
8. The optimality gap and value estimation error for the bottom branch: 2 we prove that (*xxx) (Xp) (1--) and VIA VA (Xp) (1~7)([[[--))))) and the other two possible action chunks air4h = (1.0,0,...)) FLY
and a (1,2,0,....) both admit lower estimated values compared to o (X,) < V.A (X₂) and V (Xo) < V.A (Xa). Moreover diith also admits lower esti- A mated value compared "Etth" Vi V.A (Xo) which proves t (Xo) aes+a and thus concluding our proof: (*x))) Vaa (Xp) =
Fx [--1(- (1->)11
25
Now we begin our proof as follows.
Step 1. MDP description (Figure 10).
The transition function 7 of the MDP is defined as follows (from left to right):
T(Z Z, a) T G,a)
Va,
T(Z s,a = 2) = 1. Va, vs $ + G
T(X1 Xo, = 0) =1 22h
T(X, Xo. a = 0) = =Eh
T(C Xo, a 0) Eh
T(V Xo, a = 1) = 1 Eh - C1
T(h Xo, a 1) = Eh
T(G Xo, a = 1) = C1
T(X, X1, 0) = 1
T(X₂ 81,0 = 1) 1
(132)
T(X, C,a= 1) = 1
T(2|X1, = 1) 1
T(2 C,a= = 0) = 1
T(G | 0) 1
T(Y; Yi,a = 1) = 1
7(2, "1,a 0) = 1
T(Z Yi,0 = 0) 1
T(2 a;a= [.a=11
T(X4+1 X2, E (0,1)) T(Y.1 Y, E (0,1)) =1,
Wi (2....,h-
T(Xo Xx-1, (0,1)) = T Y E {0,1)) =1
45

---

Published as a conference paper at ICLR 2026
+r=0
X1
H
1-26h
Xn
X1
X
+==
X0
1=1-6
==
1r=-
1-
Yb-1
Y1
Figure 10: A (2h + 4)-state MDP that is constructed to illustrate the MDP constructed to meet the exact upper-bound optimality gap in Theorem 2. We redraw the same states Z, G, Xo in multiple locations in the diagram above for better clarity.
The reward function defined as
r(Z,a) = 0, Va
r(G,a) Va
r( = O,
:s: G
0)
=
r(C,a= 10 rx, 0)= - (0,1)) = 1,
(133)
r(C,a 0) (Xxi
r(Yi,a = 1) = a(Nia== =0==0 0) = 1- ò
r(Yia 0) r(
0,
r(X,, E (0,1])
Vi {2. , 1h11 r(Yi,a E (0.1}) = -1-6 6, Vi {2, ... 1}
Notably, there are some special states:
State Z: self-looping "black hole" state that always gets reward at each time step and thus has constant value of O.
46

---

Published as a conference paper at ICLR 2026
State G: self-looping "black hole" state that always gets reward at each time step and thus has a constant value of 1/(1-7). 7)).
State Xo: the special state that branches out based on the action taken. The agent periodically visit this state every h steps unless has been trapped in either Z or G. As we proceed in the proof, we will encounter factors in the form of i-by in the calculation of the optimality gap. These factors come from the agent looping around and revisiting Xo with b-probability each cycle.
These two absorbing states are important because their values sit at the boundary of the value range of our value function V(s) E [0, 1/(1 -]]. Shifting the reaching probability from Z to G or the other way around results in the biggest possible difference in the policy value. Our construction hinges on the constructing D such that
1. Pp(- $t, st, (s:)) and T(. st, (st)) differs by only En (in TV distance required by the strongly open-loop consistency assumption) where precisely en probability mass is moved from reaching state Z to reaching state G. This causes the V to precisely underestimates is worth noting that we cannot make the 2 in
the value of Vi by (1-))[[[1-2aa>))) the denominator En because ac needs to simultaneously maintain a value gap with V*.
CxT we were to construct an example where (Xo) (Xo) V be (T=>)[1[(1=aa>>] it would enforce (Xo) = V*(Xo) because there would be no probability mass left to create the gap between V* and Vãc. With an extra En the denominator, we can also make the optimality gap of Va precisely
(1- bringing the combined value gap (between V and V up to 11-))(1(1--2aa11)
EA7"
25
2. Pp(. st, +t, and T(. St, ग (st)) differs by only en (again in TV distance as required by the strongly open-loop consistency assumption) where precisely en probability mass is moved from reaching state G reaching state Z. This causes the Vat to precisely
overestimates the value of Vat by (I-9)[-(1-Ea>"")
617"
We use special action 2 where upon taking the action the agent immediately transitions to Z and receives a reward of 0 (except in G). As we will see soon, this action is useful for constructing a data distribution with an easily 'controllable' probability of reaching Z for the top branch and an easily 'controllable' probability of reaching G for the bottom branch. Before we start constructing D, we first check the condition that 2* is strongly En en-open-loop consistent.
Step 2. Strong Eい open--oo consistency of D*: It is clear that one possible x* that achieves 1/(1 (7) value is
T (X,)) 0 x =C()== =1
(134)
T" (x²)= 0
We can easily check that D* collected by x is strongly é ex-open-loop consistent by observing that
the only path that 7* outputs (0, 1, 0, 0, ...) has en probability, which causes the state distribution of
$141 to differ by at most E under the TV distance (subject to a = (0, 1,0,0, ..... or a = (0, 10,0.00 0, conditioning). This concludes that D* generated by '²' above is strongly EY oo-open-loop consistent.
Now, depending on the first action ag, the MDP can be decomposed into two parts: the top (a = 0)
and the bottom (a = 1). We construct the data distribution for each branch and analyze the actual and nominal optimality gap for each branch in the following steps.
Step 3. Data distribution Drop for the top branch: we use mixture of the following two policies to construct strongly Eд -n-open-loop consistent Drop-
Policy :
(Xpp(X)) (C) = (Z) 0, {
(135)
Topp (X1) = T.oo (X1) = 2.
47

---

Published as a conference paper at ICLR 2026
Toop always take = 2 unless it is in state Xo. C or Z where it always takes 0. It is clear that this policy only produces two possible action chunks: deat+ (0, 0.... ,0) aca+h dorth (0, 00..0, We note that the aza+hh policy always leads to state Z:
PD
(8tth = Z St: Otith 0) = 1.
(136)
Top
Policy mipp'
top (Xo) = 0,
Ttop (X1) To (C) = 1,
(137)
Top (a - 0 X1) = 1 òG 2
"top (a 1 X1) = ÔG. with some 6G 1). "oop can also only produce two possible action chunks: ama+h (0,0,...,00 or Otzt+h = 2.t4h == (0, 1.0.....,0) ,0.
The distribution of Se+A conditioned on agrth 0s
PD (s1th = Z St, ac:t+h +h 0) 0,
PD Rop
(81th =G 81, at:++h 0) O,
(138)
PD. (SLth = Xo St, at:++ = 0) 1.
stop
Mixing mlop and Tip'' Let Drop be a mixture of D Dx10, and DRiup'
PD =1 - P) + <PD
(139)
where
(140)
2(1 - -G)
It is clear that 0 < < 1 (because 8G E (0, 1)), making it valid mixing ratio.
We now compute the marginal state distribution of the mixture by first analyzing the action probability:
PD. (aitth (t)) Eh.
(141)
Ppz sop
(aitth (s:) =(1- (1-2Es) (1 - 8G).
The state marginals are then
Proo (80th L, orenn stt
Pr (stth = 2 St, a((4h)
Poop (a 8e)
(1 P (are4h (4))
5) P Di top
(8)) + <Ppp (aiithh 1 se)
(142)
Eh 5)
E1 -5) + (1 2E (1 -bG)s
2Eh
Therefore,
PD (81+h = 7 sc, k(.th) 2Eh:
PD (St-h Xo St, (rett)) 2
(143)
Paw. (814h G St, (((th) 0.
Step 4. Strong En ch--open-loop consistency of Drop: Now, we check for strong open- loop consistency for the three possible action chunks on the top branch:
dirth 0,, O.....
ai
10,1,0,...
(144)
Gurth (0, -....
48

---

Published as a conference paper at ICLR 2026
For Botth 0. we can compute open-loop marginal state distribution follows:
T(s+hh = Z st, (xe+h) = Eh
T(s++h = Xo St, ai:+th) = 2
(145)
T(8Lth 18, (beth) = Eh-
Combining this with the data distribution calculated in Equation (143), it is clear that
Drv (T(t St, atath 0) || PDiop (5cth St, Oztthh 0)) En.
(146)
We can repeat the same procedure show that
Drv (Tr((T())t) 8t, OEL4N = 0) || PD (SE4R) | St, Or:th = 0]) (00=))=3 = Ens Vh' E {1,-.. , 1}
(147)
because the only difference in these distributions is that they occupy St-M = Xx with 25, probability instead of St+h Xo with 2 probability.
For aa44h
(0, 00.2,0...00 -....,... ), it is clear that
Drv (T(*++)) 88: Crath = a(ath) || (S.,,N | 88, Oz:-h = (bt+h)) = Eh
(148)
holds for any h E {1,2, ... h) since the only difference between these two distributions is the e
probability path (i.e., Xo + C + Z where the probability is under T(- st, (.(4)))
For L:4th = (0,1,0,...)) we first compute the marginal state distributions:
(1 -- )8G
Pe ($Lth = Z St, ((.tth)
25 + +1--2nn00c
2
PDop (scth X $t, a(tth))
Z + +(1 11-22nnd: 2
Priop (8tth = G $t, ((tth))
= 0.
(1 - 1-2En)0
(149)
PDee (81+1 X1 81, ((tth))
25h + (1 22
Pr (8e+1 = X1 St, ((.th)
E7
25A + (1 25n)0c
Eh
Priop ($t+1 = C St, ((tth)) = 25 + (1 - 2c )OG
We can also compute the open-loop marginal state distribution as follows:
T(stth Z St, ai.tth) 1 2
T(stth Xo
8t, (r:+thh))))) = 2
T(stth = G
8t, ((tth) = O.
(150)
T(8141 = -X1 St, a1 = 2
T(s141 = X1 St, af = E/.
T(s++1 = C 8t, (h.tth) = Eb.
Let e3 be any value that satisfies cg (0,5n/2) we can set
5g = En 25h 20a)
(151)
(EA, + c3) 25h)
such that
Pri (8tth %
St, 0( = 1 - 2 203,
PDio (8tth = Ng
St, O n)) = 25h + 203,
Pr (51th G
St, ( 0,
(152)
Priop (st+1 = X1
St, (r
-EE 203:
Priop (9t+1 = X1
St, (( = =th + C3
Po (8++1 = C
St, Gi. = Eh + C3-
49

---

Published as a conference paper at ICLR 2026
It is easy to check that 0 < óg (a valid probability) because in Equation (151), each term in the numerator has a larger term in the denominator (i.e., En < En + C3 and 1 2E,, 20g < 1 2E,)).
Now, for all h/ {1,2,-.. , .h.
using the values calculated in Equations (150) and (152), have
Drv (T($4+h St, aLa+N d((+w) || PD ($L+h/ St, ata+h d(=+1)) 20g. (153)
Since Cg < 2, the strong open-loop consistency assumption holds for ci as well.
Step 5. The optimality gap and value estimation error for the top branch:
Now we can compute the optimality gap for the estimated value for dz:thh
(1 - 25,, - 20a)7
V (*oo) Vi (1- 2(nh + )a)))1)²
(154)
where the h-step reward sub-optimality gap is due the agent reaching Z with (1 22, 20a)
probability, and the lt-tep distribution gap is reflected in the (1 - 2(ex + ca)? term at bottom because the probability of reaching Xo after h steps is 2(EA + oz).
Similarly, we can compute the optimality gap for Va and Vi=:
7 7mh
ENT
V*(X)) - (Xo) = E/
+
1-9 + +1 (1= -Vo)
1 y
(155)
ENT
(1 -1-18 (1=))(1- (111--1-- )at)
2n - -gh1)
V*(X)) Voe(Xx)
7-7 + 1-7
7011 22x) -Voc)
(156)
2E
- (1->) - (11-2s))"
Now, we observe that
28, - 20g 38, > 28hr
(157)
where the first inequality is due to c3 E0,Eh/2 and the second inequality due to En (0, 1/5) in our assumption. This allows us to oower-bound the estimated optimality gap for oi.44. as follows:
(1 2 2c3))
(*(X)) Vo (Xo)
(1 11-717 (1-r)(1= - -2(En + c2))²)
2ENY
(158)
(1 7)(1 - (1 (/
=Vo(Xo) Vit
where the inequality is obtained by triggering Lemma (e.g., by setting E = 28,, E2 = (1 2 2cx), 7 )²). The bound above rules out the possibility of airt4 being picked by Age because has a lower estimated value compared to ai
Finally, for a since it is correlated with st4h Z and receives no reward except the first step in Drop: the estimated value is just 1, being trivially smaller than V (Xo) and would never get picked by n Up now, we have finished our data distribution construction and analysis for the top branch. We summarize the key intermediate results as the remark below:
Remark 1 (Intermediate results from Step 1-4) The optimal action chunk is aisth and the estimated values for the two other possible action chunks dheth Coath are smaller than that aiathi
2547
(Xn) < Ve (Xo) Vi (Xo) VXx) - (1-7)(1 22a)7²)
(159)
In addition, both Drop and D are strongly En-open-loop consistent.
50

---

Published as a conference paper at ICLR 2026
Next, move to the bottom branch.
Step 6. Data distribution Dbottom for the bottom branch: For the bottom, we again two policies.
Policy "hottomm
bottom (Xo) = π bottom (G) bottom (2) = 1,
(160)
Woottom (Y1) Toottom (Y1) =2
Tottom takes 1 Xo and and Z, and takes a otherwise (at 11, Yi). Is clear that this policy only produces two possible action chunks: A = (1,1,1,-..) or 2z = (1,2,1,...)
Policy nottom"
2 bottom (Xo) 1,
hottom (a Y1) 8z
bottom
Yi) 8%,
(161)
botton (Y)) = 0,
bottom (Yi) = 1, Vi (2, .. ,,, 1},
where 8z (0, 1) and we shall specify the exact value of 8z shortly.
Tbottom takes a when it is at Y; and takes either = 0 (with 8z probability) or (with 1 - 6z probability) when it at Y,. It is clear that this policy only produces two possible action chunks: La+ (1, ...) Lhsth 11,0,1,....
Now, we observe that the marginal state distributions for both policies conditioned on dur+A are independent of and 8z because the action chunk only appears when hottom reaches G and when bottom reaches Xo. More specifically,
22
Phhattom ($1+1 G St: 0rrth)) P'Diartam (Se+h G
St, am = 1,
(162)
bottom
A PDę (51+1 X₁ Se, (1thh) PDa (s1th bottom
Xo
St, (1+tt) ) 1, vi {1, . 1-11-
bottom
(163)
We can now mix D hottom and D with an appropriate ratio to control the state marginals for S1:4hh G and stithh Xo arbitrarily (St.thh Z stays at 0 because none of the policies take/have A
taken dtt+ when they reach 2).
Mixing म bottom and 2 bottom Let Doottom be mixture of D bottom and D D
Prratsam = P bastom + OPP harton
(164)
where we set the mixing ratio to be
C1 1 (1 (En ci)
(165)
This mixing ratio helps the calculations to be simpler later on.
We compute the marginal state distribution of the mixture. We start by analyzing the action probability:
P (ae (8)) = Ccs
(166)
Ppa
bottom (a (ss) = =(1 --)1- oz).
51

---

Published as a conference paper at ICLR 2026
The state marginal is then
Piosiom (#eth Xo, Corth 8t)
Pestiom (st4h = Xo sc, ((ett)
Prootoo (Coth 186)
Pp bostum (0 se)
(1 1) P Dbottom (rr+h (1) + PDP Dootom (Q st)
(1 8q (1)(1- 82)
c1 (1 0) + (1 Eh - -c1) -8z)
1 &p C1.
(167)
We can use it to deduce the rest of the marginals as follows:
A Prinvom (Be+h = 8t, (1t+)) = Eh + CI, Ah' E (1,... ,0.,,,,11
PDootoo (81th Xo
St, a -1--5 -h C1s
Prosuso (82+h = Z
8t, btth) = 0,
(168)
PDnottum (81th' Yk
St, (1+h 1 -Eh C1, wh {1, ., 2},
A
PDoottom (91+1 Yi
St, (1t4)) = 0.
Up to now, we have established Dottom and we are ready to check the strong open-loop consistency.
Step 7. Strong En open--ooo consistency of Dottom :
A For aurth = (I,1,,..), we can compute the open-loop marginals as follows:
T(8++N = 8t: (zthh) ) = 61, wh' E (1,....h 1)
T(s+th Xo St: r+th)) 1 Eh C1:
T(8tth = 2 8t, 1zith) = Eh.
(169)
T(s+tk YN St. oreth) =1 C1. Vk' e [I, .h. 2}
T(s+11 = P1 St, ( Eh.
Combining it with the marginals calculated in Equation (168), is clear that for all / (1,
1}.
DTY (T(sethe 8t, autt o Prootim (Se+h) satisfying the open-loop consistency.
8t, aet+h = atstn)) Eh. (170)
For axth (1,2,1, the data and open-loop state marginals
PDoottom (seth = Z St, ax1th) = 1,
Panatoom (8t+1 Yi 8t, ax1th) =
x
Eh
Ci
Paostoom (8t41 Yi
St, ax aat+t)
C1
T(s = Z
St, a(th)
= 1 -
(171)
T(8t-h G
8t, (etth) C₁
T(5+11 = Y1
St, a()4)))))))))))))))))))))))))) = EN - C1s
T( Ý
x
St, a1 Ehi
T( G 81, (xtth) = C1.
This allows us to bound the TV distance for all R'' E {l...., ........ ...,h - 1] as
Drv (T(s+4 I 85, amtn ax)
P Pa (se+h St, aeeth = (x 5 -
C1
(172)
52

---

Published as a conference paper at ICLR 2026
Since a <tn/2 < 1/10,
61
10
5En/9 < Eh. g
(173)
1-c
satisfying the strong open-loop consistency assumption.
For Gatth (1.0,1,...)) ), first compute the state marginals in Doottom follows:
(1 Eb c1)8z
Promsoo (0eth 8t, diith)
Eh (1 - Eh 01 (0
Poootoom (81th Xo 8t, (i:+h))
Eh (1 Eh 01)0z (1 Eh - c1)6z
(174)
PDootom (8141 = Y1 St, (iath)
Eh (1 Eh 01 )0%
P Prootoo (81+1 = Vi St, (.14h) = Ek (1 En c1 012
Eh
We can also compute the open-loop marginal state distribution as follows:
T(s++h 2 St, (b(44)) = - Eh C1.
T(stth = No
8t, (g.th)) = Ehi
T(sthh G
86; (g(tt)) = 61,
(175)
T(s++1 Y1
St. (rtth)
Eh C1.
T(s+1 = Vi
8t, 0 (gith)
= =Eh7
T(8+11 G
8t; (g(th)) = 61.
Let c4 (n), and we set
En - En c1)
8z
(176)
(Eh + c4)(1 Eh - C1)
Then, we have
Proatoom (51th = 86; rz(hh) = 1 - 8/ Ca:
o
PDrostoo (Atth Xg
8t, Ozt = Eh + Ca,
PDocttom ($1th = St: + = 0,
(177)
PDoottom ($t41 Y1
St. 0 ) 1 Eh CA:
PDhostom
($111 = Vi
St, o O.r = Eh + Ca-
The TV distance is then
Dry (T(sethr St, Ge:tt a(u44)) Pa (SLth) St, Octth = d(r = ca. (178)
Since 64 Ex. the strong open-loop consistency also satisfied for of
Up to now, we have checked that all three possible action chunks in the bottom branch satisfy the strong open-loop consistency assumption. Since Drop and Doottom have non-overlapping supports for arath and they are both strongly éx-open-loop consistent their own, construct D as
Pp(- 8t) = (1 P St) + oPPantsan (- 8t):
(179)
for any o (0, and conclude that
Remark 2 (Intermediate result from Step 5-7) D is strongly E -n-open-oo consistent.
Up to now, we have constructed and checked both D and D are strongly en-oppen-loop consistent.
As the final step, calculate the optimality gap and value estimation error for these action chunks.
Step 8. The optimality gap and value estimation error for the bottom branch:
53

---

Published as a conference paper at ICLR 2026
We first note that similar to a "br+h is correlated with 8ith Z and always receives 0 reward except the first step in D. Thus, the estimated value fx is just 1, being trivially smaller than Vn and would never get picked by fge- The only top contenders are outth Gatt and C+t++ (which we already analyzed in Step 5 above).
We start with Lats where we can compute optimality gap as follows:
(X.) (Xo) (1 Eh c4)7 + 5(1 7) + (en + (a)6(77-7)
(180)
- (1->) (EN + +,)gx))
Now, observe that
Eh + C4 < 2ch 25h
(181)
where again the last inequality comes from the fact that En 1/4.
We can now lower-bound the optimality gap as follows:
V*(xo) - 1ro (Xo) > 2 + 8(1 - 7) (En + c4)6(7 )n)
71 -11-2tn)?"
25A7
(182)
(1 7)(1 (1 -11-2hh)7))
= (Xo) - (Xo).
where the first inequality is obtained by triggering Lemma 5 (e.g., by setting 1 2 S2 (1 - Sh. - c4,, = =²))
With this lower-bound, we can conclude that Lhe4h would not be picked by t as well because Ve < (Xo).
Up to now, we have climinated both 42.44 and aieth (for the possibility of being picked by m and the only remaining contender left is aur+4
A
We can also compute the estimated and the actual values for a+sthh aotth = 1 in terms of their optimality gaps:
8(1 c1)7
V*(X)) (X,) (1 (1-7)(11 (1 En -c1)/4)
(183)
15(1 En ()) + Enl (1 - 7) - (1 11-6 Eb - c1 )))
V*(Xo) V.a
(184)
Let
2EAT C2 1- (1 c1
8
(185)
(1 Eh -1 1 (1 2
We first check 1 - 8 is a valid reward value (within [0, 1)):
N 1 (1 5/1 )
8
Eh
Ci
(1 28h)
2Eh
(1 -
2
(1 - ),
(186)
25h
1- 25h
$1,
where the first inequality is because C2 0, the second inequality is due to c₁ < Ens and the final inequality is due to eい < 1/4.
It is also clear that 8 because all terms are positive in the fraction (Equation (185)).
54

---

Published as a conference paper at ICLR 2026
Next, we substitute 8 in to obtain
('(o)) V/c (Xo) (1 (1-7)(1 (1 2
2817 - 62
(187)
('(X)) (Xo) = (1 7)
2 - 02
2xh) +
Eh7
(188)
(1 7)
En c)77))
where intuitively the second term in (*())) - V(e
is due to the fact that from Pb(- 8t, (1:+h))
T(- 8t, dpr4n)) there is shift in Ey probability mass from Stitth (Xo, G..... to Sta+t (Xo. Y1. Z. ... incurring an additional ENT
I-7 sub-optimality in terms of the h-step reward, and then
amplified by the value recursion by an additional factor of 1-(1-a--0177" (where 1 Eh 01 the probability that (ruth reaches Xo for the value recursion to occur).
A Since 02 0, now show that buth achieves the highest estimated value among six possible action chunks:
NENY
V.A <
V
(189)
(1 (1-7)(1- (1 22a)7²)²
which means that te (Xo) Finally, putting everything together, we have
a tat+
A
). equivalently Vi- Va
(*(o)) (Xo)
2 Y C2
+ (1 - (1 (1 Eh 1)
EnY
(190)
(1 - 7) (1 - -2
as desired.
55

---

Published as a conference paper at ICLR 2026
H.9 PROOF OF PROPOSITION 1
Proposition 1 (Optimality Closed-loop Execution of Action Chunking Policy) Let V be the value of the one-step policy, π result of the closed-loop execution of the action chunking policy Tae learned from D. That is, for each E supp (D(())),
(st) a where death = (sp).
(17)
If D and D both strongly En on-open-loop consistent and supp(Po(te) (us+h)) supp Pa= (Se, aee+h)), then for all 9t E supp(Po. (8))),
(*()) - (8)) $ 11 - 71-
ENY
1
$ 32, (18)
1 11 - -
1 - (1 En)r
Proof. We observe that
(st) ( tett
(191)
(s,,
Combining this with Theorem 2, we get
Q" at 2 (*())) A,
(192)
where A
Eu. 1-7 -1-1111 2yt + 1-(1-
Now, we can bound V as follows:
(*(s:) - (8r) S (*(,,,
(" at) + A
S 7lBT((oraatt
[V*(s++)) V (s1+1)] +
(193)
1 ENT
2 - )
+
(1 7)2
1- 1- 1-(1- =n)
56

---

Published as a conference paper at ICLR 2026
H.10 PROOF OF PROPOSITION
Proposition 3 (Comparing action chunking backup and n-step return backup) Let DD be strongly Eμ Ex-open-loop consistent and o,-sub-optimal, and supp(Pp(ss)) 2 supp (Pr- (s+)). Let पे 5t + arg max Q (st, az) be the policy learned from D, via n-step return backup:
(st, 0²) = E Beatn + 7 "," ( (8e+n, o. (st+n))
(38)
Then, for all 8, supp ( (sc)) (and with Hn 1/(1 (1")),
Vat (st) Vーt (st) 2 8n ENY
2
7th 1-7
1 (1 25k)7
1- Eb)
(39)
8n Hn - 3.
To prove Proposition 3, we first prove the following helper Lemma to quantify sub-optimality for n-step return policy.
Lemma
Let O be the solution the uncorrected n-step return backup equation:
Q7 (5,, ax) (l) |Rutt + + max Q (sc+n;
(194)
actn
The following inequality holds as long as D 5n sub-oottimal::
8n
Q" a.) 2 a.)
ss, S, E
(195)
ym
where Q" is the Q-function of the optimal policy in M. For the n-step return policy
Ty 8t + arg max O (8,, at),
(196)
ae
its corresponding value admits a similar bound:
6n V*(+)) (x)) + -m
Vor
(197)
Proof.
Using the definition of sub-optimal data (Definition 4), we have
(i at) = Epo(-lenay (e) Rectn + max Q" (St+n, attn)
actn
(198)
s Q ax) - 6n + + "E"pppo(lona
max (714,, a(+n) (*(++h)
Rearranging the incquality above yields
(7, ax) Q" (st, ae) < -, + 7 (p(()))) (V7 V (M))m))) S, E A (199)
By recursively applying the inequality above, we have
6n 1-m Vss SS EA
Q" (st, a.) 2 (st, +
(200)
By choosing at = (st), we see that
V*(s)) 2 (st,
8n
> (sr, az) +
7
(201)
6,
= (s)) +
70
57

---

Published as a conference paper at ICLR 2026
Now are ready to prove the main Proposition 3.
Proof of Proposition 3. From Lemma 6 and Theorem have
6n 1- 2 (*(s)
(s) + Eby
2
(s)
+
(202)
(1 20n)7"
1 (1 en
Rearranging the terms give
Vat (s) (s) 2 6n
EN
2
1
(203)
7"
(1
2
1 - -m
H.11 PROOF OF THEOREMM
Theorem 7 (Closed-loop Execution in the Absence of Stochastic Shortcuts) D is a-open-loop mixed and M is free of On-Stochastic shortcut, the value (V*) of the one-step policy (n as a result of the closed-loop execution of the action chunking policy mhe learned from DD admits the following bound for all S, E supp(Pp> (st)):
DAT (*(ss) (s:) < (1 7) xh a)) + (1 7) -1)
(47)
Before we start proving the main theorem, we first prove the following Lemma, which shows that the overestimation of V is bounded when the advantage of the stochastic shortcut (as defined in Definition is bounded.
Lemma (Lack of stochastic shortcut bounds overestimation) If M free of On- advantageous stochastic shortcuts for horizon h, then
9n 1-
V (8+) (s.)
(204)
where Vi is the value function of the action chunking policy learned from any data distribution D from M and V* the optimal value function in M.
Proof.
V+ - (st) Epp
Butth + + (o(th)) v
SEp
(((t(()) V (((+ta)) + Un
(205)
< On
nh
Lemma 8 (Monotonicity of optimality) Let 'D be any data distribution that is collected by open-loop policy. Then, for all st, Guithh
(*t) 2 (e [ra + <V**(*+1))
R Ppo (1le))0044)) Reath + (Reat)
(st h]]
(206)
Proof. The first inequality is clear from the definition of Q* ur) and V*(s:):
(s,, ax) (T [ra 7V* (V44))))
(207)
V V*(ss) max Q' af) Q' (8), of
(208)
58

---

Published as a conference paper at ICLR 2026
For the second inequality, observe that
E Po (111)0044411)
[Be44h11 + - ((sttnn1)
Eppo (1e
R + 4iin
h-1 max Q 11
(209)
2 E (1eeaaen))))
Bea+h-1 + n- (0 [[+h-1]]
Epp ((le)
REath-1 + / [r(stth 11
Epp (188,)0444)
Reath + 7 V²*
By induction,
BT() Ior + +V+*(((11)) 2 R P (1eo
Beath + (
(210)
as desired.
With these two lemmata, we are now ready to present our main proof.
Proof Theorem 7. The key idea of our proof is to analyze the action chunk taken by the policy The (i.e., The 8t + arg max, (st, a1;+1)). Due to our construction D, the action chunk learned by The either comes from D or DD. We can D express as the aggregation of two open-loop data distributions, Di and De Dout:
Po- (1)())) == Pp (- (+18)) (1 Pb. ( st)
(211)
where supp(Pat (artth 8t)) Ssupp( Pp. (0ttth 8t)) = 20 and
a33
à S - a1-a) 33)
(212)
Case 1, from D*: If Ttc (:1 st) Ssupp (atsth st)) = 2, then we know that
Lithh E supp( (0reen st)),
(213)
for any aaus E oh Thus, the closed-loop execution policy ा' takes the optimal action at st. This leads to the following equalities:
V*(st) (*()s) = =V*(st) (",
(214)
= -)(Br())eeaa)) [(V - V (8++1))
(++( V Epp. (( V*r)
(215)
= =7"Eppr (ot((s44))) V (8e+h))
Case 2, from DD: If m se) (Popp((P:: 86)) & Ø, then there exists af m st)
such that
Citth (Poop 8x))-
(216)
For any attth E tt( (: se) such that attt & $ supp( (ameth 1x))), know from the previous case that V*(ss) (st, at) = yEE(()) [(V*($++1) (*(++)))] We are left with analyzing
V*- -Q" a)) for the second case.
Since Pp(. 82) ( st) + (1 F) Po. (. 81) + (1 3) a) P (. 82) with supp(Ppa... cus (atie-h st)) n suspppppp (Burth st) Ø, can isolate the contribution of Do in Do by an event that is 1 when aue+h E supp Pp- (aua+h st)) and otherwise. Now, we can remove D when conditioned on
follows:
3
(1 - B)å
Pp(. 8t, I - 1) =
P (. (.1 T= +
Poo (. ( T= (217)
3+ 3)a
3+ (1 B)
= (1 - P (: Sr, = 1) + aaP
St, = 1)
59

---

Published as a conference paper at ICLR 2026
a9 Since á (1-a)[1B)) with some algebraic manipulation, can obtain à S
By Lemma I, there exists Lheth E supp((p-((au)) (x)) such that Pb(: St, ((eth) (1 Pb- (. 8e, (ett)) @P 8t, g(rth)
(218)
with à S S a.
We are now ready to bound Vut as follows:
(st) - (s)) 2 Q (st, d(ath) - (st)
(1 a) p Qrc(st (itth) + aBpps Qto(st, destho) -V*(s))
(219)
2 (1 a)
[ (((Vtt()etn) - (*(++t))) as)
2 (1 a)? Bro.
Vit - - -V*(+++))
a
where we drop the term aE @tt (se: (gath) 2 0 for the second inequality.
By combining Equation (219) (when rt (t))sspp((Pp))) (poupp((><(pp 8x)) * Ø) with Equation (2he))
Tt se) Suupp(Pps Pp- (: 8t)) = Ø), we can now recursively bound Vi (st) - (st) as follows:
V (s:) (5c) 2 - (1-
(220)
Combining this with the result from Lemma have both a lower-bound and upper-bound on Vtt in terms of V*:
(*())) C. S (st) S (*)) Co. where C. [1->)(1~> 11-3)0~>(1-0a) Cy As I-
(221)
Now, we are ready to bound V (st, (s),a{) for the second case (when Tt( (- 8t) n supp( Pn- (:
s()) * 2). For notation convenience, for the following equation, we use PD as abbreviation for
Pos (- St, d.
V (st) Q- (st, a))
V - Epp [rt + (
V Eppe [Rueth + ((( V (+
Vt (se) + Ca -Ero Reath + +1" (Vat (814h) - -Ca) - (VV V (8++1))
(s)( - -Epo Beuct + (Vat + Ca + , + Er( (v(V((0++1) VV (s++1))
= =Ca + + CC + +EET mn (V (st+1) -
(s
(222)
where the first inequality uses Lemma 8, ET(( [ra + y VV* 2 Epp (1s RRethh + (Rett
* (s+++))
(223)
and the second inequality uses the lowerbound for (xt) and the upperbound for Vat (81+)).
Combining the bounds of V - (st,, af) for both cases (Equation (214) and Equation (222)), we have
V* (s.) S max (11s, (8++1) V (+++1))
Co + - + + (B (V*(4)))))) (5(+1))
= Ca Ca+g"Ca + 1 - V (st+1)
(224)
Ca Cr00
1-7
(1 - 21 - (1 a)) +
(1 71 ))
60

---

Published as a conference paper at ICLR 2026
H.12 PROOF THEOREM
Theorem (Closed-loop AC Policy under Bounded OV) Let DD the data distribution collected by optimal policy. Assume D can be decomposed into mixture of data distributions (D", D1, D2, D such that each data distribution component satisfies Assumption and for some 0/, 0h 0, they satisfy the following two conditions:
1. Locally bounded optimality variability condition:
every DD (including D*) exhibits oz- bounded variability in optimality conditioned on St, a2 for all (st, az) E supp( (st, at)), and
2. Globally bounded optimality variability condition: D whole exhibits -/avriiabilitty in optimality conditioned on 8t, det+h for (8t, amt+h) supp(Po(ss, dcc+h)).
Then for all 8, supp (upp(Pr-((p (8t)),
0 - (1 7) - 77) 0% + HH.
+ 0h 4 minmmin(n min(mkinkkkk))) 0%
-
v
(20)
Proof of Theorem 3. Consider any s E supp(Po-(st)) Let aien = (s)) and
Grtth arg max max Ru Esupp(Pp a (anssals)) Ep (praann))) [Reeth + [ (225)
We first observe that
Epp (1lerof44))) [Reeth + ( 2 (s)),
(226)
because
(*t) =
Emga Par (.1aa) Epp (1910111))) [Retth + (s4a)))
(227)
and the maximum value random variable is no less than its expectation.
Let
Quiu arrth)) =
min supp( (
REeth + V* (s++n)),
(228)
Qiaaxe (8e, aeath): =
max
RReeth + (8(+n))
(229)
supp( (Po (P
Since DD exhibits -vrriiibbyyyyy in optimality, we have
Qmin (or, a(eth)) 2 (mt (st, -
(230)
V*(ss) (s,
= V*(a)) (oo(()) + -(t)
(*())) Qtc(,, attte) + Qt (st, a(eth)) Q' (st,
< <V*(s))) (tr, + 04 + 7 BB Po(-benatiin))
V (+++h)))) V (s(+h))
(231)
=V*(*+) (10 + 0% + 7 " R Po
V (st4a)) ("(ttth,
7 "'Epotlonation [V*(844) - (s atttll
61

---

Published as a conference paper at ICLR 2026
We use it lower-bound Vac (st) as follows:
Vot (st) Qic(Se: a(i+h)
2 Qae (Ne: ((xth)
Epol bin
BEith + V (st#N)]
Epol soat Reath + + + Epp(lan (1"(Vat V (
Qmin (h) + (po(lenaienn)
Qmax (rethh 0% Brrolseaaae [ (V(att (*(oh)))
Ep (144))))))))
Reath + (V 0h (Po((snniios)) (Vet()) "
V (s(th)))
(s:) 0% + " Ey"Epo(Rooouun) (Vat( (V ($4+1))
g6
(st) I -n
(232)
Let M+ = (D1, Das } be all data distributions from (D' D1, D2, "DM) where (st, thet))
in the support. Let D+ be any mixture of M. where each mixture component has non-zero weight:
M Wi P.,
P =
(233)
where w/ 0. O. w1
Let
Qm (se, at)
min aupp(Po (1))))))
[Ra4h V (sc+a)],
(234)
Q (5t, ac) =
max
[Re+4 + V (8c+n)),
(235)
supp(Prr (-le.a.))
Sinin (Sin (4)
min (upp(Poi(((s..a))
|Rethh + (8(++)))
(236)
imaa (8t, at) = max supp(pp:((let.a))
[Reeth + V* (8(++a))
(237)
(St, at)
(St,
PRith + V (8e+h))
(238)
supp(po
[Rethh + V (sc+a))
(239)
supp(Pp+ (lantta)
The minimum and the maximum is over the remaining trajectory conditioned on 8t, ag or St, 0t:t+h that is still in the support of the corresponding data distribution.
From the of -bounded variability in optimality and the Assumption of each data mixture, we observe that
Q'(ss, ar) 2 Qminn (St, ar) 2 ( (sc, 11. Vi (1,2,...,,.))
(240)
(s a.) 2 Qunin (St, ax) 2 Qinax (St, 0z) - 11.
(241)
We can then derive that
Qinax (St, at) max max (min ) (st, (M at))
(242)
< <s a1) + 1
62

---

Published as a conference paper at ICLR 2026
With this, now upper-bound Vad (st) as follows:
Vat (st) Qte (Be, (hs+t)
Epo snater)))
Rea-n + / ao (setN)
Vat
Ep (onn))))))
Vat Beath + M ac (scth))
-Erg+ (laaatit)))))))
[BetHh + t (sEtN)) + t Bot (leateea
1 ac
(8et1)) - V (sc+aa)
Qtaax (8t, (1eth) + +1 " Polleennten)) (Vo((e+th) - V (*(+
Quaee (5t, at 7 po w
(
(st, + oh + " Pal Pottle (ot(6)))) - V (8eth))
(243)
Let
(8)) = V*(+)) (st
(244)
((s)) = Vat (8t) (s,,, at
(245)
From the inequalities above, we have
(st)) S o + +1th sup ((seth) (scchh)).
(246)
00
0 < (8)) 5
+ A(se);
(247)
7h
0G 1(t))
(*+) - (sx) < min
(248)
1-
The minimum operator allows us to obtain two upper-hounds on A:
(1 + 11+0000
A (8x) S 0f +
(249)
1 - 7/
0/ + J A < 1 1-gh + Å *st) S
G
(250)
1 7h
Finally, combining these two upper-bounds together and recursively applying the inequality yields our desired results:
0h 7² AuC 06) 1-7 (1- 71 - 7²) +
V*(ss) (" at)
(251)
(1 - 7) - 7")
63

---

Published as a conference paper at ICLR 2026
H.13 PROOF OF THEOREM 6
Theorem 6 (Worst-case Closed-loop AC Policy under BOV) For any h 1,7
(0, 10,0% R 0k (0. | ; 0,
7- 4(1-7% , E 0, (iin( there exists M and 'D
satisfying the assumptions in Theorem 3 such that there exists 8, E Supp(P2> (st)), where
of + + min (1/ 0%
(s)) - (s))
o, (s) (st( 2 (36)
7 (1 21 7h)
To show that our upper bound is achievable, we need carefully design both the MDP and the data distribution. For clarity of the proof, we divide up the construction into two parts. The first part (Lemma 9) focuses on designing part of the MDP and two data distributions DD and Do such that
of any action chunk that has value bigger than V 1- preferred over the action chunks in D and D The second part (Lemma 10) focuses on constructing the remaining MDP and the DA that contains the action chunk that The picks where V overestimates the value of this action chunk by 0 + 3 '=min(0"__ 28)
1-yh Finally, we assemble these two results (combining D Do, PP) to show that
the MDP and the mixture data achieve our upper-bound exactly.
Lemma 9 ("The Castle") For 5 E (0. 1), 0% 7-7" 2(11))> consider a 2-state, 2-action MDP in Figure 11. Let there be two data distributions, D* and D D is collected by the following optimal closed-loop policy from X and Y:
H = 0, (X)
(252)
DD is collected by the following optimal closed-loop policy from X and Y:
T (X 7Y)
(253)
Let D be a mixture of D* and D with
Pp = - Pp + spo.
(254)
There exists ei E (0, 1/2) such that
DD and Do both individually exhibits O-variability in optimality conditioned on 8t, at for all 8t, E (popp(
2. D exhibits varriabbbyy in optimality conditioned 8t, authh for all St, azat+h supp(Po(se. am)+))),
and
(X) (Y) 1 - S0C
(255)
2(1->)(119))
Proof. Set
(1 7000 7 7
C1
(256)
We first check whether e₁ E (0, 10,) For the upper-bound, clear that C₁ 1/2 because 2|1- For the lower-bound, > 0 because all terms in the fraction are positive.
06
We now check the two optimality variability conditions. The first (local) one is trivial because n always receives r = 1/2 ei and nh always receives = 1/2, and the optimal value for X and Y are both (X) V+ 2(17))'
64

---

Published as a conference paper at ICLR 2026
Y
X
0 0
a 1
a
a 0
r 1/2 c1
r 1/2
1 1/2 - C1
r 1/2
1-8
1-5 1-8
Y
X
Figure 11: MDP construction Part 1 for Theorem 6 (" castle"). This diagram describes state X and and how actions a and a transition between them. The main purpose of this construction to make V (X) underestimate V* by exactly 0 /11- - > This allows the action chunk that appears in the second part of the construction to be preferred (by age) over the action chunks that start with a = or
Next, we check the second (global) condition by analyzing all possible states and action chunks in D. We observe that for any dze+h that starts with O, we have
Qmin (X, Gesehh) = 1 - 2c (7 - x
(257)
2(1 - 7)
Qmaxx(X (X, (xett)))
1
(258)
2(1 - 7)
which gives
@ X, acsth)) - Quinn (X, (r:tt))) 06.
(259)
By symmetry, we also have
Qmax ((att))))) - Qmin an, 06
(260)
for all aesth that starts with 0r 1.
Now, for any aatth that starts with a₃ = 1, we have
Quin (X, amtth) 7 - 201 (7 - -1
(261)
2(1 - 7
Quaxe (X, amsth)
7
(262)
2(1 7(1
which admits the same gap the case when a, 0. The same also holds for Y with a Thus, D exhibits of -vrriabbbyy optimality conditioned on St. Outth for all St, antth E Supp(Po(ss (ra+t)).
65

---

Published as a conference paper at ICLR 2026
Finally, check for the value,
+ 201) 7
ac (X)
2(1 7)
1
1/2 + (1-2cc)(7 - -/0
1 7/
2(1 - 77)
(263)
+ - 7hth 2(1 7)009
2(1 74)
--9 ?
7 + 5(7 - 7/
5/
2(1 11) 7)
1 1 1-7h²
as desired.
Lemma 10 ("The Flower") Assume 0h
0h (0.
(o, ( YE (0. 1), and
Consider a 5-state, 3-action MDP in Figure 12 building on top of the transitions that already in Figure 11. Let D be data distribution induced by cycling, time-dependent (with time cycle length of h) policy T (we use the subscript to indicate the time step from 0 to
To
(st (st
X) = To (st X) = 2,
(264)
To
A st Y) 3
(265)
* (81+8 = =0 = # (st+k = D) = 2, & (1,2, -2}
(266)
A "h (814h-1 =Õ) --= k (11th-1 D) 0,
(267)
F (81+ = X) = 0, Ak E (1,2, .h....,1111 1},
(268)
" (8t+== Y) 11 Ak .1,
(269)
Let Vi be the nominal value of the action chunking policy म. learned from DA and let
JC
+ 7 mimmmn (0 ..)
A 0
(270)
7b
1-9h
For any E (o, - ). there exists some 0 < 51/2,0 11/2, E (0, 1). such
20 that for every 0 < Â min A (-)
1. DA exhibits O-varibbilityy in optimality conditioned on 8t, aurth for all St, aut+h E supp( (Pao (st, ama+))),
2. DA exhibits -f-variability in optimality conditioned St, a for all St, at supp(Ppa (8t, a)),
and
H
V+ (X)=
+ A (271)
2(1 7)
1 1 h
V*() (X() A Â
(272)
V*() (X) >
(273)
1
C
V*(X) (X) >
(274)
1-
Proof. Without the loss of generality, we assume we always start from state X. Due to symmetry, the same analysis applies to state Y (with the first action being a, 3 rather than d; = 2).
66

---

Published as a conference paper at ICLR 2026
62
0 2
146
a=0
r= 1/2
r = 1/2
a
114
52
X
(1-62)(1-8)
1-02
(1-4a36
3
b 0
a 2
Figure 12: MDP construction Part for Theorem ("the flower"", This diagram describes the remaining states Č, Đ and X, and what actions 2 and do in state X and Y. The main purpose of this construction is make Vit (X) overestimate the optimal value of the action chunks that Face Q*x (X, of ), by exactly 0f + 7 min(0f 0f) >²))
67

---

Published as a conference paper at ICLR 2026
Due to cycling nature of the data collection policy, we observe that all action chunks starting from X are in the form of a1:+h = (2,
or at:++ = (2.2. ... ,2,00 0). These two possibilities
0's and 1
correspond to two different paths that the data collection policy takes:
Borth (2, ): Stay in either X or Y. The agent going on this path receives
O's
constant reward of 1/2 except the first step where it receives reward of (1 c2)
crtt = 22, 2, 0): Visit Ò and then stays there for h - until it goes out with = 0 to visit X. The agent going on this path receives a constant reward of (1 cs) except the first step where it receives a reward of (1 c2)/2.
A
Similarly, all action chunks starting from X are in the form of Ott-h (2,
ac:++
0's and I's (2, 2, 0). These two possibilities correspond to two different paths that the data collection policy takes:
Barth (2, -: ): Stay in either X or Y. The agent going on this path receives
0's constant reward of 1/2.
Certh 2,, 2, 0): Visit C' and then stays there for h until it goes out with 0 to visit X. The agent going on this path receives a constant reward (1 + ca)/2 except the first step where it receives a reward of 1/2.
A
Now, we divide up the problem into two cases depending on the relative values of 0f and OG.
Case 0 2 0h:
Set
(1 + 2"000""
02 0 +
2Á 0,
(275)
7
2(1 7)04
> 0, 2(1 7)00
(276)
7 7/h
> O. C4 7 7
(277)
Next, we check that C22 Cg. C4 $ 1/2.
We first observe that
(1 (1-))
=
(1 7) 2 0.
(278)
Dividing both sides by 8(1 - 7) yields
7 hb
2 0h 2 00.
(279)
4(1 - 77)
Now, using the inequality above, we have
(1 + 1/00
C2 = 2
14
1 - n/h
(1 + 74 00
2
04
1 n
(280)
40f
1-9h
$ 1/2.
68

---

Published as a conference paper at ICLR 2026
Furthermore,
= 2(1 7)0% 7 7-7h - 1/2.
c4 S C3
(281)
Next, we check the data distribution D satisfies both optimality variability conditions. We first note that we only need to check for 9 {x, because all other states are out of the support due to the cycling nature of the data collection policies. The first (global) optimality condition is trivial A because the h-step reward received is deterministic conditioned on de:t+ E { " J. and the optimal value of V ($2+4) is always スエー This leads to 0- O-ariabiility in optimality conditioned on 8t Ourth- For the second (local) optimality condition, we check the difference in optimality for two paths from 8t, ar = for both 8 = X and 8, = X.
For s; X, the optimality gap is
7 - 7h c3 2(1 7/)
0
(282)
For S Xx the optimality gap is
7 x
C4 2(1 - 7²) = 0 S 04-
(283)
This concludes that the second (local) optimality condition is also satisfied.
Next, we first analyze which action chunk TIc prefers by computing Qt''s 's:
[1 c2) + 7 7-7"
Qt (X, (itn)) 2
+ 1 Vut (X),
(284)
1-7
7 74)
@t artth)
(1 02) + (1 03)
+ */"/et(()
(285)
7
1 h
Qne a antth)
=
1-7
+
(x),,
ac
(286)
7 7d
Qie koeth) = 2
1 1 1 + (4)
+
*
v
(x²).
(287)
1 7
ac
We first observe that
C2 (cg ca) 7- 7/
(xt, - Qt af (xeth))
=
- 7
(1 + 1000
= 0 +
01. + 06 A
7b
(288)
200
A
7h
> 0.
Also,
Qite (2.atth)== - Qtte (X, ((att))) = =0>
(289)
Therefore,
(2,, V+ (X) max (X, (p Qte (X, A ((t+t))
< < (X, ,., (Xt, (((+)))
(290)
-itt (X)).
Now, we can compare the values for the action chunks for X and X:
7/t Qt (X, af(e+t) - 2 (X, alccth) = C3 23 - 7)
+ 7 (Vat - Vist (X)) > 0,
(291)
Ot (Xt, (g:44)) = -c 7 h1
+
- vat 0,
(292)
-c12(1 - 7)
69

---

Published as a conference paper at ICLR 2026
since c3, C1 > 0 and h > 1,0 < 1 (and thus
2-1"
1-7 > 20).
This concludes that (X) Toe (X)
are+h
A
22, ... , 0) and thus
(X) = 1 - 7 + (7 - -1 + c1)
(293)
2(1 -4h -7)
and
1
7 2-1
7/1
ght
Vic
(1 - 02) + +1+ C3)
+
Vat (X)
2
1-7
1 1
(294)
1 g4
2(1-7)
1 2 2
We can now compute the remaining values as follows:
1
V'(X) 2(1 -7)
(295)
Q'(X,a=2) = 2) (1- (1-02) - -7) +
(296)
2(1-7)
1 02
Q*(Xaa= 2) 2(1 - 77)
(297)
Substituting the value of 22 yields
0
(1 (1+7"100
Á
+
(298)
1-7 7
(1 (1 7m)
2(1 7)
2. Case 0f < 0
Set
O + 000
A
(299)
1 7h
00 + 00
C2 2
- 0,
(300)
1 h
2(1 0
C3 C4
>0
(301)
7 - nh
where again Á is any value that satisfies 0 < A A.
From the definitions above and the value range of 06 (Ug < --x²m ), is clear that
40 h 7 7h 1/2.
2(1 - 7)
c3 = C4
(302)
Next, we check the data distribution DA satisfies both optimality variability conditions. With the same argument as the previous case, we can quickly conclude that the global optimality condition satisfied. We just need to show the remaining local optimality condition. We repeat the procedure from the previous case.
For 8, X, the local optimality gap is
7h C3 2(1 )/)
-0h.
(303)
For s; X the local optimality gap is the same because C4 c3:
n/
C4 2(1 h))
1/.
(304)
70

---

Published as a conference paper at ICLR 2026
This concludes that the second (local) optimality condition is also satisfied for the second case.
Now, follow the procedure as the previous to show that Qte (X, a(eth)
(X, p(s+s) > 0 and Qic (X,, ((t+)) 2the ((t+)) > 0.
This concludes that Ttte( = (X) a1tt/ = 22 ... 2, 0), and thus
A
;
1 7- (1 +3) 7²)
V+ ac
(X)
(305)
(1 71- - 7%)
and
1 V (X)=
7 - 7b 1
ght
(1 (2)
02) + +1+
+
Dit (X)
2
1 --
(306)
g
2(1 7)
1 - h
Repeating the same procedure the previous case, we obtain
- -xX,= = 0 g0
A
(307)
1 7hh
resulting in an optimality of
0 + 00
= (1 71 7
(308)
1-7
3. Sub-optimality of Vit: Finally, we can use pretty crude upper-bound on the actual value of the action chunking policy nt (reparamcterizing 82 = 1 - (1 - -=))))
8(7 -01 721
82
V+ (X) S (1- (1-d2) (1 c2)/2
(309)
2(1 7)
-
82
82
< 2(1 - 7 (1 7)
[1- 0(7 - - + 1 1-7
(310)
Set 8 = 1/2, we have
82
82
Vi (X) 5
11- - +
(311)
2(1 - xth (1 -
1-7
We set
7-
4c(1 - 21)
1/h
82 =
(312)
2 - 2² + 7
which results in
82 7 vb 4c(1 7M)
(313)
2 - 3r +
clear that < 82 < because e < 7- and 2-3 1. 3(1-
Substituting 82 in the bound of Vit (X) above, we obtain
c V*(X) - -ot)-vtt (X) >
(314)
1 7
n
71

---

Published as a conference paper at ICLR 2026
Proof of Theorem 6. Let
UC 7b
2 min(O 01
A
0
(315)
Consider the 5-state, 3-action MDP constructed in Lemma and Lemma 10 and a data distribution consisting of a mixture of three data distributions D*, D (from Lemma and DA (from Lemma 10):
PD a(1 P) + <PD + (1 PD.
(316)
We set a to be any value between 0 and 1 (non-inclusive) and set any positive value such that
( -1-7" -00 -
$
(317)
(7 -リ - 20 - 7)
where Á = =(11>) < min 0/ 0 < min A 2
(satisfying the condition for in Lemma 10).
The numerator and the denominator are both positive:
- >) 21 - () --) (1-)))) (1
0,
(318)
meaning such always exists.
Substituting the inequality the result of Lemma 9 results in
1-7+507-
C9 2(1 - (11->-([1-) -1 - 7h 2(1 (1-7) 1 74 + A
0
(319)
which shows that te will always prefer agr4h action chunks in D and Do
This means that the value Vi+ and the action chunking policy The we learn from D coincides with these of D allowing to directly use the results of Lemma 10.
Thus, we can conclude that
V*((s)
Vat 2 1 -
(320)
and
g
+ 2 min (H,, 0()
("(X- =
1
(1 (1 /h)
(1 7) -イ²) o,
(321)
as desired.
72

---

Published as a conference paper at ICLR 2026
H. PROOF OF PROPOSITION 5
Proposition (Worst-case analysis of n-step return backup) For any n N+, on (0, y 2²) and (0,3n/(1-7)) there exists MDP M, and 8, opttmmal data distribu- tion DD with supp(Po(s;, a.)) 2 supp Pp- (st, ax)) such that for some E supp( (8:)),
8,
(s) (x) = 1-9 a.
(42)
and for all (s)p(P)-(ss)),
V*(s) = (s).
(43)
Proof. Consider an MDP in Figure 13. Let D be the data collected by the following policy:
T(a = 0 x) = T(a 1 || X) = 1/2. (a 0 Y) = at
(322)
(323)
<(a = 1 =1-a. a.
(324)
== 0 r=1
a=0
r=1-o
11-02
Figure 13: An MDP where the learned action chunking policies optimal and the learned n-step return policies can be arbitrarily sub-optimal.
is clear that the optimal policy is x* *X") = 1, (Y)
Since the dynamics are deterministic, the data distribution D is strongly open-loop consistent and thus by Theorem 2, (a.) = V*()) for all 8, E supp(Pa-) (s:)).
To make sure the data distribution satisfies the 8, optimal condition, we set
8n
a
C2 = 8n - 101-011 (1-7)0.
(325)
clear that at state Y, n (Y)= = We can then calculate the optimality gap for Q+ exactly as follows:
5 1-9 1-7 o,
V*(X) - (x, 0)
C2
(326)
- 7 + 0x 10770177
- 77)
8n
V*(X) - (x,
1-9
(1 - 7) - )
=
(327)
Since 0, म, (X) 0. Now, we can compute (X) follows:
V*(X--Va(K - Vx(X) = 02 8
o,
(328)
as desired.
73

---

Published as a conference paper at ICLR 2026
H. PROOF OF PROPOSITION 4
Proposition 4 (Deterministic Dynamics are Weakly Open-loop Consistent) If a transition dynamies M is e-deterministic, then any data D collected from M is weakly En open-loop consistent with respect to M for any h E N+ as long as EY 2 3(1 (1-
Proof. Since T is e-deterministic, it can be represented as T(- s,a) = (1 - z) =)d + ET(- s, , ,ah) f(. (( an)
for some f 8 A + and T' 8 A As. Let f(s, ".,
Let E } binary indicator variable that is and only if
+ f(*++k; Or+k), vk E 1,,1,2, 1}
(329)
Intuitively / = when the trajectory is generated deterministically until but not including the last state in the trajectory chunk.
From the fact that 7' is s-deterministic, we know that
PD - 1) 2 (1 (1-8)h-1
(330)
We also have
Po(unthh 81) = PD(1 = 1)PP (u | st. In 1) + PD( = Po |8,, I 0)
(331)
Then we have
(rv(Pa(ae4t) st) || Polaceth |t, Ln 1)) S S (1- -))--)))
(332)
we transform each distribution of desth deterministically by f(st, -), by data processing inequality (DPI; Lemma 4), we have
Drv (Eaga (Baai)>PP((()) [5s(snmee1n]]
E Po 1) [5 [() S (1 (1 - --)"
(333)
Similarly, we have
Prv(Po(Orat*a) 8x) || Po(ar:+441 11,1441 S (1- -1- 8)a²)
(334)
which be also deterministically transformed by taking 0c++h+1 (f(se, -)s 1,atth) (again with
DPI, Lemma 4) to obtain
Dry (BacinnPolll (aeth st, (esth) II
(335)
PorcssooP((leeahanee [ (at-h Sty0ra Th+1 1)T 1) < $)²)
Now, if we analyze the distribution of seth subject to the open-loop execution of the action sequence from Po(- 8:) and break it up into the deterministic and the non non-deterministic case, we get
E Ba [Teees ( 8t) = Pr(1 = PeEEantt~PPPllla (5((s))s))))) +
(336)
Pr = 0) B). (-(-)s)) [Tociis (- St, In = 0)
Note that Pr 1) denotes the probability that an open-loop executed trajectory using aut++ ~ Pp(- I st) is deterministic. This is different from PD( 1) because the latter is based on Pp(Seat1+1: aratn) whereas (J 1) based on open-loop trajectory distribution: P sr) I T(s+*k St, 0..4)) They both admit the same lower bound of 2(1 - (1 -1-66 -1)
Therefore.
Drv (Panss))P [Torss ( [s)]] BanterPo(le)) ( (( 1) s (1 - (1 - (1-=)^ h-1)
(337)
Similarly for the state-action case, we can multiply both side by the same conditional distribution (arth St, (ma+k)) which preserves the TV bound. For the left-hand side, we have
( arth st) Bauc Po((ls) (acth
St, ata+hh Y Tarea (scun st)]
(338)
74

---

Published as a conference paper at ICLR 2026
Therefore, get
Drv (PB((esttth a4th (x) Po((ss) aT (acto St. (e.aacth)4 ((s
(339)
S (1 -1--)
We also have
PD(srth st)) = (1 (
Po(se-n -t,1==11 (1 (1 -) Po(sc 1 8t,
0)
(340)
Similarly, we have
Div(Po(suth st) PD(sita |t. = 1))
(341)
Drv (Po(seth) st) || (Pulan-P(((ee) [ (8((saaa)))) (1- =)m-1)
For state-action, we can also get
Po(st-h; aeth 81) = (1 0eth St, Jat1 =
(342)
+ +11 )) Po(Seth, a4th I 8t. In-1 = 0)
which can turned into the TV distance bound:
(ro(Po(s acth st) || Pp(s a0th St, I++1 =1)
Drv Polsith, atth st) |I
(343)
Ea ~ Po(((e). (Tp(a St, aeath I441 1)1l (s)
- (1-(1-s)))
Connecting all three total variation inequality (Equations (333), (337) and (341)) together, we get
Drv (Pw((pst st) Bec [Taneep (: st]]) I 3(1 (1 - -) S Eh
(344)
Connecting all three total variable inequality for state-action (Equations (335), (338) and (343))
together, get
Drv PPv(P5(94A-1 st) PD(s arth st)) s 2(1 -)^-1 (1 k)h-2
$ 3(1 (1 - 2)^-1)
(345)
Eh
Therefore, ' is -o-oon--o consistent as desired.
75


Reviews:
[{"review_id": "YBD8ui7m2k", "reviewer": "Reviewer_WP1s", "summary": "This work improves on the concept of chunked critics in the context of off-policy RL.\nThe chunk length of the critic and policy is decoupled by optimizing the policy against a distilled critic, allowing the policy to output shorter chunks.", "questions": "- Notably, the only experiment where the proposed method is outperformed is by the $NS$ approach in the puzzle-4x6-1B environment. Can you provide an explanation/intuition of why this is the case? Are there certain environmental characteristics where you would expect your approach to perform worse than computationally simpler methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Well-written and detailed theoretical investigation.\n- Strong and robust results.", "weaknesses": "- Some minor grammatical errors (e.g., l. 26, 182)\n- There is no discussion of the computation overhead compared to the baseline methods (e.g., from maintaining the additional distilled critic). Since the performance gap to the baselines is substantial, providing a short intuition should suffice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RlepEPY4D9", "reviewer": "Reviewer_uJSe", "summary": "This paper proposes Decoupled Q-Chunking (DQC), an offline RL algorithm that mitigates bootstrapping bias in long-horizon learning by evaluating long action chunks while predicting only short ones. By decoupling the policy and critic chunk lengths, DQC eases policy learning without sacrificing the benefits of multi-step value propagation. Theoretically, it formalizes the open-loop bias in chunked TD backups and proves when chunked critics outperform standard $n$-step returns. Empirically, DQC consistently outperforms prior state-of-the-art methods—including SHARSA—on the most challenging OGBench tasks.", "questions": "in weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The core idea of decoupling the policy and critic chunk sizes is novel, effectively addressing a known trade-off in multi-step Q-learning to get \"the best of both worlds.\"\n2. The paper provides deep theoretical backing for Q-learning with action chunking, formally identifying and quantifying bias, and proving the conditions under which the approach is superior.\n3. It demonstrates superior, state-of-the-art results on challenging long-horizon tasks, significantly outperforming previous methods on the OGBench benchmark.\n4. The work includes comprehensive experiments, comparisons to relevant baselines, and ablation studies that strongly support the authors' claims.", "weaknesses": "1. The approach still suffers from the inherent bias of open-loop value evaluation in action chunking and lacks a mechanism to actively correct it.\n2. Its theoretical guarantees rely on a strong \"open-loop consistency\" assumption for the offline dataset, which may not hold in many real-world scenarios, limiting the generality of the claims.\n3. The use of a fixed, global chunk size for both the policy and critic is a limitation, as the optimal action horizon might vary depending on the state.\n4. The framework introduces additional components (e.g., two Q-networks, best-of-N sampling) and hyperparameters, increasing computational overhead and the practical difficulty of tuning and deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MOHDhR6lIb", "reviewer": "Reviewer_WFcb", "summary": "This paper introduces DQC, that builds upon Q-chunking but decouples the critic’s chunk length from the policy’s chunk length. It aims to address bootstrapping bias from TD backups and open-loop policy inefficiency in off-policy TD learning. Experiments on the OGBench suite show that DQC outperforms Q-chunking and other baselines.", "questions": "1. The distilled partical critic $Q_\\psi^P$ is not Bellman-consistent with the long-horizon cirtic $Q_\\phi$. $Q_\\psi^P$ is trained to approximate an optimistic projectiion of $Q_\\phi$ rather than its bellman target, thus there's an inherent objective mismatch between the two critics. Will it introduce systematic bias or training instability? Especially, when in non-markovian or sparse-reward settings, the optimistic assumption might not hold. Could the authors clarify why this training remains stable and effective in practice? It would be helpful to see a quantitive analysis showing how closely $Q_\\psi^P$ tracks $Q_\\phi$.\n\n2. How the key hyperparameters were chosen, for example, $h$ and $h_a$? How were these values selected in practice—through tuning, fixed ratios, or heuristics? And how sensitive is the method to their choice?\n\n3. What is the computational overhead of maintaining two critics with different horizons?\n\n4. In Algorithm, it introduces two expectile parameters, $\\tau_d$ and $\\tau_b$ for the two critics. However, I do not find how the two factors are selected or tuned in Table 4. Expectile regression is known to be sensitive to $\\tau$. It would amplify overestimation bias when using a large $\\tau$. Are $\\tau_b$ and $\\tau_d$ the same across envs, or task-specific? Did the authors observe significant sensitivity in performance when varing $\\tau_b$ and $\\tau_d$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper derives explicit bias and near-optimality bounds, offering theoretical insights of when action-chunking methods succeed or fail.\n\n2. Empirical performance of DQC is nice.", "weaknesses": "1. Theorem 4.6 is somewhat idealized. It depends on the open-loop consistency assumption. It is hard to hold in the practice for realistic offline datasets, especially in long-horizon settings, in such cases $\\epsilon_h$ may not be small, and thus the errors scale and then the bound can become vacuous. Could the authors provide an empirical analysis on how big is $\\epsilon_h$ in the OGBench datasets?\n2. Theorem 4.6 does not seem to cover the decoupled critics. Is this bound still valid when the two critics are trained with different horizons and objectives?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xhh1JCfoHE", "reviewer": "Reviewer_c39B", "summary": "The paper addresses the bootstrapping bias problem by proposing a novel method to decouple the chunk length of the critic from that of the policy, allowing for more efficient multi-step return backups. The proposed algorithm optimizes the policy against a distilled critic, retaining the benefits of multi-step value propagation while avoiding the challenges of open-loop sub-optimality and long action chunk learning.", "questions": "Please see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow, with well-structured explanations.\n2. The authors effectively discuss the scenarios in which action chunking Q-learning is preferable over standard n-step return learning.", "weaknesses": "1. My main concern is that, by comparing the performance and curves of DQC and QC-NS, QC-NS—without decoupling the action chunking of the Q-value from that of the policy—achieves nearly the same performance as DQC, except for the cube-quadruple-100M task.\n2. How much does the performance degrade if the expectile update in Eq. (24) is replaced with a quantile update?\n3. Why does setting $N=128$ not lead to better performance?\n4. In most algorithms, the batch size is typically set to 256, but in this paper, it is set to 4096. Could the authors conduct an ablation study to justify this choice?\n5. Could the proposed method be evaluated on standard offline RL benchmarks such as D4RL?\n6. There appear to be some minor errors, such as two consecutive \"the\"s in the first line of page 2, and potential mistakes regarding $\\tau_b$ and $\\tau_d$ in Algorithm 1. The authors should carefully proofread the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)