Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling
Authors: Yuejiang Liu
Jubayer Ibn Hamid
Annie Xie
Yoonho Lee
Max Du
Chelsea Finn
Venue: ICLR 2025 Poster (2025)
Keywords: Robot Learning
Action Chunking
Action Decoding
Test-Time Sampling
Primary area: applications to robotics, autonomy, planning
TL;DR: We present a thorough analysis of action chunking and a decoding algorithm to improve it

Abstract: Predicting and executing a sequence of actions without intermediate replanning, known as action chunking, is increasingly used in robot learning from human demonstrations. Yet, its effects on the learned policy remain inconsistent: some studies find it crucial for achieving strong results, while others observe decreased performance. In this paper, we first dissect how action chunking impacts the divergence between a learner and a demonstrator. We find that action chunking allows the learner to better capture the temporal dependencies in demonstrations but at the cost of reduced reactivity to unexpected states. To address this tradeoff, we propose Bidirectional Decoding (BID), a test-time inference algorithm that bridges action chunking with closed-loop adaptation. At each timestep, BID samples multiple candidate predictions and searches for the optimal one based on two criteria: (i) backward coherence, which favors samples that align with previous decisions; (ii) forward contrast, which seeks samples of high likelihood for future plans. By coupling decisions within and across action chunks, BID promotes both long-term consistency and short-term reactivity. Experimental results show that our method boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks. Code and videos are available at https://bid-robot.github.io.

Published as a conference paper at ICLR 2025
BIDIRECTIONAL DECODING: IMPROVING ACTION CHUNKING VIA GUIDED TEST-TIME SAMPLING
Yuejiang Liu, Jubayer Ibn Hamid, Annie Xie, Yoonho Lee, Maximilian Du, Chelsea Finn Department of Computer Science, Stanford University
ABSTRACT
Predicting and executing a sequence of actions without intermediate replanning, known as action chunking, is increasingly used in robot learning from human demonstrations. Yet, its effects on the learned policy remain inconsistent: some studies find it crucial for achieving strong results, while others observe decreased performance. In this paper, we first dissect how action chunking impacts the di- vergence between learner and a demonstrator. We find that action chunking allows the learner to better capture the temporal dependencies in demonstrations but at the cost of reduced reactivity to unexpected states. To address this tradeoff, we propose Bidirectional Decoding (BID), a test-time inference algorithm that bridges action chunking with closed-loop adaptation. At each timestep. BID sam- ples multiple candidate predictions and searches for the optimal one based on two criteria: (i) backward coherence, which favors samples that align with previous decisions; (ii) forward contrast, which seeks samples of high likelihood for fu- ture plans. By coupling decisions within and across action chunks, BID promotes both long-term consistency and short-term reactivity. Experimental results show that our method boosts the performance of two ttate-of-the-ar generative policies across seven simulation benchmarks and two real-world tasks. Videos and code are available at hhtpps: /ectps://bicc robot github.
INTRODUCTION
The increasing availability of human demonstrations has spurred renewed interest in behavioral cloning (Atkeson & Schaal 1997 Argall et al., 2009). In particular, recent studies have highlighted
the potential of learning from large-scale demonstrations acquire variety of complex skills (Zhao
eet al. 2023b Chi et al. 2023 Fu et al. 2024b Lee et al. 2024 Khazatsky et al. 2024). Yet, existing
methods still struggle with two common properties of human demonstrations: (i) strong temporal
dependencies across multiple steps, such as idle pauses (Chi et al. 2023) and latent strategies (Xic
al., 202T Ma et al. 2024). (ii) large style variability across different demonstrations, such as differences in proficiency (Belkhale et al. (2024) and preference (Kuefler & Kochenderfer] (207)) Often, both properties are prevalent yet unlabeled in collected data, posing significant challenges to the traditional behavioral cloning that maps an input state to an action.
In response to these challenges, recent works have pursued a generative approach equipped with action chunking: (i) predicting sequence of actions over multiple time steps and executing all or part of the sequence (Zhao et al. 2023b: Chi et al. 2023); (ii) modeling the distribution of action chunks and sampling from the learned model in independent (Chi et al. 2023 Prasad al. 2024)
or weakly dependent (Janner et al. 2022 Zhao et al. 2023b) manner for sequential decisions. Some studies find this approach crucial for learning a performant policy in laboratory scenarios (Zhao
let al.) 20236] Chi et al. 2023), while other recent work reports opposite outcomes under practical conditions (Lee al. 2024). The reasons behind these conflicting observations remain unclear.
In this paper, we first dissect the influence of action chunking by examining the divergence between learned policies and human demonstrations. We find that, when a policy is built with limited context length- little or no history is used as input for robustness or efficiency (Mandlekar et al. 2020
Bhharddwajj et al. 2023b [Brohan et al. 2003a bb] (Shi al., 2023 Collaboration, (023) increasing the length of action chunks allows for implicit conditioning more past actions, thereby improving its ability to capture the temporal dependencies inherent demonstrations. However, this advantage
Equul contribution. Correspondence yuejiang. liu @ stanford.edu. Videos hups://hidrrooot.gifhh iithbu.

---

Published as a conference paper at ICLR 2025
action chunking
(b) receding horizon
bidirectional decoding
Figure 1: Illustration of different inference methods applied to robot policy with action chunking. The robot tasked with catching moving trolley. (a) Vanilla action chunking (Zhao al. 2023b) executes actions based on previous predictions, resulting delayed reactions to object motions. (b) Receding horizon (Chi et al.) 2023 enables faster reactions, but leads to a jittery trajectory in the presence of multimodal demonstrations (e.g. both left- and righthandders. (c) Our Bidirectional Decoding explicitly searches for the optimal action from multiple predictions sampled each time step, achieving both long-term consistency and short-term reactivity.
comes at the cost reduced access to recent state observations, which can be crucial for reacting to unexpected dynamics arising from modeling errors or environmental stochasticity. This trade- off raises a crucial question: How can we preserve the strengths of action chunking in long-term consistency without suffering from its limitations in short-tern reactivity?
To this end, we introduce Bidirectional Decoding (BID), an inference algorithm that bridges action chunking with closed-looy adaptation. Our main idea is to sample multiple predictions at each time step and search for the most desirable one. Specifically, BID operates on two decoding criteria: (i) backward coherence, which favors samples that are close the sequence selected at the previous step: (ii) forward contrast, which favors samples that are close to the output of a stronger policy and distant from those of weaker one. As illustrated in Fig. | BID updates the chunk of future actions based on the previous strategy, promoting temporal consistency over extended periods while remaining reactive unexpected changes.
The main contributions of this paper are twofold: (i) a thorough analysis of action chunking ($3)
and (ii) a decoding algorithm to improve it ($ Empirically, we validate our theoretical analysis through one-dimensional diagnostic simulation and evaluate our decoding method on two state-of- the-art generative policies across seven simulations and two real-world tasks ($5). Our experiment results show that the proposed BID boosts the performance of recent policies by more than 32% in relative performance. BID is model-agnostic, computationally efficient, and easy to implement, serving as a plug and- -lay component to enhance generative behavior cloning at test time.
2 RELATED WORK
Behavioral Cloning. Learning from human demonstrations is becoming increasingly popular in robot learning due to recent advances in robotic teleoperation interfaces (Sivakumar et al.) 2022] Zhao et al.] 2023b Wu et al., [023]] Chi et al. 2024). Generative Behavior cloning, which models the distribution of demonstrations, 18 particularly appealing due to its algorithmic simplicity and em- 2022 Florence et al. 2022 Brohan et al., 2022 Shafiullah et al.) [022] Zhao et al.) 2023b Chi et [202 Brohan al. 2023a). However, significant limitation is com-
pirical efficacy (Jang et al.
pounding errors, where deviations from the training distribution accumulate over time (Ross et al., Ke et al. (02T). These errors can be mitigated by gathering expert correction data (Ross al. 2011 Kelly et 2019 Menda et al., 22099] Hoque et al. 202 b) or injecting noise during data collection (Laskey al. 2017 Brandfonbrener al. 2002). ). but such strategies require additional time and effort from human operators. To address this, recent work proposes predicting a sequence of multiple actions into the future, known as action chunking, which reduces the effective control horizon (Lai et al. (2022) ZZho et al.) 2023b George & Farimani] 2023 [Bharadhwaj et al. 2023a). By handling sequences of actions, action chunking is also better at handling temporal dependencies in the data, such as idle pauses (Swamy et al., 2022 Chi et al. 2023) or multiple styles (Li et al.)
2011
2017] Kuefler & Kochenderfer, [017]] Gandhi et al. 2023 Belkhale et al. 2024). However, inde- pendently drawn action sequence samples may not preserve the necessary temporal dependencies for smooth and consistent execution. Our work provides a thorough analysis of action chunking and proposes decoding algorithm to improve it.
2

---

Published as a conference paper at ICLR 2025
Sequential Decoding. Decoding algorithms have been studied in generative sequence modeling for decades, with renewed attention driven by recent advances in large language modeling (LLM). One prominent approach focuses on leveraging internal metrics, e.g.. likelihood scores, improve the quality of generated sequences. Notable examples include beam search (Freitag & Al-Onaizan 2017 Vijayakumar et al. (0018), truncated sampling (Fan al.] 2018 [Hewitt et al.) 2022), minimum
Bayes risk decoding (Kumar & Byrne 2004 Müller & Sennrich 2021), and others (Welleck et al.)
2019 Meister al. 2023] Fu et al., 2024a). Another line of research explores the distinctions be-
tween multiple models to jointly optimize for the desired properties such as quality or efficiency (Li al. 2023 Leviathan et al.) (003).. More recently, several studies have highlighted the potential of guiding the decoding or sampling process through the use of an external model, such clas- sifier (Dhariwal & Nichol 202T) or reward model (Khanov et al.) 2023). In the context of robot learning, recent works have explored guided decoding for long-horizon robotic planning (Huang) (ct al., 2023) and manipulator geometry designs (Xu et al. 2024). Nevertheless, effective decoding strategies for low-level robotic actions remain lacking. Concurrent to our work, Nakamoto et al. (2024) propose to select action samples by querying value function learned from reward-annotated demonstrations HHnnsnn-Estruch et al. 2023). Our work does not rely on separate value function; instead, we propose a decoding strategy that addresses the consistency-reactvve tradeoff inherent in action chunking through sample comparison.
ANALYSIS: TRADEOFFS IN ACTION CHUNKING
3.1 PRELIMINARIES
Consider a dataset of demonstrations D (T, 1N where each demonstration т, consists of a se- quence of state-action pairs π; = ((1,01), (82, .2))... (st,ar)) provided by a human expert. At each time step f, the demonstrated action 0, influenced not only by the observed state 8t, but also by latent variables 2,, such planning strategies (e.g., subgoals) and personal preferences (e.g., handedness). These latent variables can persist across multiple time steps and vary signifi- cantly between different demonstrations. Fig. 2 iillustrates the decision process of human expert, highlighting the inherent temporal dependencies.
To model these temporal dependencies, some recent works (Zhao et al. 2023b Chi et al. 2023 Lee
ett al. 20044) utilize action chunking, i.e., modeling the joint distribution of future actions conditioned on past states (u,, 0,41, az+7 St-es-.. Sr), in short ( (.) Here, c the context length, i.e., number of past steps for state inputs, and / is the prediction length, i.e., number of future steps for action outputs. Training such a policy typically involves minimizing the divergence of action distributions between the model π and the expert x².
π = arg min Σ Σ TED el agal
ATtanatileeell x'" (0estt(8t-ca)
(1)
During deployment, the policy operates with specific action horizon h [1, i.e., executing part or all of the predicted action sequence for time steps without re-planming. This approach essentially takes in states as context and executes h actions, which refer as (c, h)-policy.
The choices of context length e and action horizon h often play a crucial role in the effectiveness short context length c, as extending the context
of the learned policy. Recent policies often can lead to performance degradation in the presence of limited training data (refer to Appendix [A.2 for more details). Conversely, extending the action horizon h has produced mixed results. Some studies report benefits in laboratory settings (Zhao et al.] 2023bb Chi et al. 2023), while others find detrimental in real-world scenarios (Lee et al. 2024 The reasons behind these conflicting outcomes are not well understood yet.
Notably, Zhao Zhao unclear why this would hold when deviations within a chunk cannot be corrected before replanning. Alternatively, Chi et al. (2023) draws parallels to model predictive control (MPC) (Borrelli et al.) 2017 Löfberg 2012), emphasizing its role in improving consistency and planning. However, unlike MPC. which typically uses short action horizons (e.g.. h = 1). recent imitation learning methods employ much longer action horizons, such as substantial portion of the predicted sequence (e.g., h 8 in Chi et al. (2023) or even the entire prediction length (e.g., h 2 50 in Zhao et al. (2023b); Black et al. (2024) The lack of clear understanding of action chunking hinders its effective
al. (2023b) hypothesize that action chunking mitigates compounding errors, but it is
3

---

Published as a conference paper at ICLR 2025
observation
prediction
replan
St-
St
St411
(cchh)
0t-1
at
G+++11
((ch+d)
T(k.1)
t-k
t-h-d
t-h
t+1
Figure 2: Illustration of the expert decision process, where a latent variable introduces temporal dependencies in actions.
Figure 3: Illustration of (k, 1)-expert, (c, -elaeae and (c, + de-learner. Shaded regions represent observed history; darker indicate greater influence current decision.
across tasks and environments. We next explicitly analyze the strengths and weaknesses of action chunking, with particular attention to the choice of action horizon h.
3.2 ANALYSIS
To understand the influence of action chunking, we focus on the last time step of the executed chunk, where the discrepancy between the expert policy and the learned policy is most pronounced. At this time step t, a (k, 1) ee-exeett written as π == π (ax St--:: kt+k:)) predicts a, by conditioning on steps of the past states and the corresponding latent variables. In contrast, (c, h) -laamer written as ((c,h) = ((c.k) (a; St-h-est-ht ht-ha-1) is constrained to observe c steps of the past states and h steps of the past actions within the predicted chunk.
The divergence between a learned policy and an expert policy is attributed to two factors: (i) the importance of unobserved states in predicting the current action, and (ii) the difficulty of inferring unobserved states based on the available information.
To more clearly see the influence of action horizon on these factors, we next compare the per- formance of two policies that have the same context lengths but different action horizons, Th = (oel))oeee----eee
Qt-h-d:t-1). where d > 0 is the extended action horizon. As illustrated in Fig. 3 each policy has access to unique information that is unavailable to the other. Ty observes some recent states, where Th+dd only aware of the executed actions. On the other hand, Th+d has access to some earlier states and actions, which precede all information available to The We characterize the importance of observations as follows (formal definitions in Appendix E.2):
Qt-h:t-1) and Th+d = (e). h
Definition (Expected Observation Advantage). If a policy can observe a state 8t, we say that has observation advantage 0i another policy that cannot observe it. More formally, this is the difference between the expected divergence accumulated by a policy π (aae (8t) that does not condition on the state 8, and that by policy (ay | 8x) that conditions St-
Definition (Maximum Inference Disadvantage). state St, its maximum inference disadvantage e, the largest possible divergence arising from inferring incorrectly. Here, the maximum is taken over all possible incorrect inferences of st-
policy cannot condition its prediction on
Hence, denote the observation advantage that Ty gains from the observed recent states by aj and the maximum inference disadvantage it incurs from the earlier unobserved states by whereas Thad conversely gains as but incurs ef-
Let P(s St, az) denote the true transition probabilities and P( st az) the implicit dynam- ics model learned by the policy. The difficulty of inferring unobserved state hinges on both the relevant observations and environmental stochasticity, which we quantify as follows.
Definition (Forward Inference). Let (t) P(S, ge SS It-1- A a,-1), where de and 9t-1 are the ground truth states at time and t respectively in a deterministic environment. In deterministic environments, (t) 1; whereas in stochastic settings, (t) is smaller. The
prediction error is characterized by (t) = P(S, S. = 0c-1,0(11 P(S, = 9e = It-1, 0.-1).
Definition (Backward Inference). Similarly. let (t) = P(S, = g. = 9++1) where gr and ge+1 the ground truth states in the deterministic environment at time and + 1, respectively.

---

Published as a conference paper at ICLR 2025
Since Po(t) is not conditioned any action, it has higher entropy in general. In stochastic environ- ments, (t()) is small. Furthermore, let (t(t) P(S, gc1S.+1 9c+1) P(S, 9r S1++ I(+1).
sy' under expert policy as the diversity of past strategies up to time t'. Given that the forward inference is generally casier than the backward inference, the performances of Th and Wh+d differ as follows (proofs are deferred to Appendix E):
We refer the variance of the distribution P(a a.i.........
Proposition (Consistency- teeactvt Inequalities), Let L be a non-linear and non-negative conver function measuring the prediction error with respect to demonstrations. Let C {ar-hst-1, If US+ where S+ are the common states that both Th and Th-d observe. For the ease of notation, let t h-c c 1}. Then,
t h} and let ть = {t the difference in the expected loss between the (c, h + d)-policy and the (c. (c.hiypolicy, Aa = millaai a ,E [[[(Ta+aa xt" [=] - minz, E [[[[TT Tt" ]C]] is bounded as:
Ti =(t -t-h h d +++1
ay-bb 1 - II Po(T) +6())) SAuser 1 II P(())/(())+ at. TETY
TETS
Remark 1. Eq. (2) provides a general comparison of the performance of the two policies. Intuitively. the advantage of each policy stems from the additional information it has access to (i.e. oy for n and as for Th+d) while the disadvantage is bounded by the maximum divergence arising from inferring missing information incorrectly (i.e. Eb and Ey scaled by the maximum probability of incorrect inference).
We next examine the implications of Proposition !
Corollary 2 (Consistency). Suppose the truin and test environments are identical and deterministic. Suppose 0g is influenced by at least one state at time steps тr and let dy(t') Rs 0 for all l' E Ty. If the diversity in past strategies up to time t c not and Ej is finite, then
min E [[[mettd [[] < min E [[(T,, CCC Wh+d
(3)
Remark 2. In deterministic environments, while both policies need to infer the same number of unobserved states, Th+dd benefits from conditioning on additional actions, which may significantly aid in inferring the corresponding states through its action chunk. However, this is only true if the implicit dynamics model learned by the policy is approximately accurate and the maximum errors of arising from inferring these states are bounded. Note that Ta+d performs better as we increase the diversity of strategies present in the dataset i.e., as the distribution of actions taken by the expert prior to the context of the short horizon policy becomes higher variance.
Corollary 3 (Reactivity). Suppose (t't') is small or both (t')') and [ty(t') are large for all E TJ. If temporal dependency decreases over time such that eb is small and at is influenced by at least one state at time steps in Ty. then
min E [C(Th+a n []] > min E
(4)
Kh.
Remark 3. Recent states are often more important decision making and, therefore, the disadvan- tages of action chunking become pronounced when inferring these recent states becomes difficult. This can happen if the test environment is stochastic. This can also happen if the test environment deterministic as long as the implicit dynamics model is inaccurate either due to distribution shift or because of learning difficulty.
In summary, depending on the experimental conditions, action chunking may have varying effects on the learned policy. On the one hand, it benefits the modeling of temporal dependencies in the demonstrations, due to extended access to actions executed at past time steps. On the other hand, it hinders reactions to unexpected dynamics, due to reduced access to state observations at recent time steps. As result, there is no universally optimal choice of action horizon across all conditions. When both temporal dependencies and prediction errors are significant, tuning the action horizon entails an inherent trade-off between the opposing factors.
5

---

Published as a conference paper at ICLR 2025
previbus
Algorithm Bidirectional Decoding
Sp-1
Require:
current state 8, batch size N, mode size K, pre- vious decision a, strong policy x, weak policy x'
Lg
positive
1: Generate samples from each policy ~ x (s), a'
π
(s) construct the initial sets A and A
2: Compute the backward loss L3 for each sample
Lpp
3: Select K samples with minimal La from A and A construct A and AT, respectively
negative A
St
4: Compute the forward loss Lr for each sample
5: Select E A that minimizes total loss
Figure 4: Illustration of bidirectional decoding.
Update decision memory à <-
METHOD: BIDIRECTIONAL DECODING
As analyzed above, action chunking improves long-term consistency but sacrifices short-term reac- tivity. In this section, we propose to address this tradeoff by bridging action chunking with closed- loop adaptation. We will first outline the general framework in $4.1 and then describe two specific criteria 4.2
4.1 TEST-TIME SEARCH
Recall that for policy with prediction length I, an action chunk a {ap a. pled at time t is expected to follow a consistent strategy over the subsequent 7 time steps. However, executing the action chunk in an open-loop manner leaves the policy vulnerable to unexpected state action chunk and executing
(t)
sam-
changes. Alternatively, one can maximize reactivity by resampling only the first immediate action at every time step. Yet, this simple closed-loop approach destroys the consistency preserved within each chunk, potentially leading to oscillations between different strategies. How can we combine the benefits of both approaches?
Our key idea is bridge action chunking and closed-loop resampling by making use of additional computes at test time. In particular, we seek to restore temporal consistency in closed-loop opera- tions by scaling up the number of candidate samples. Intuitively, while the probability of any single pair of samples sharing the same latent strategy is low, the likelihood of finding a consistent pair increases with the number of samples. We thus frame the problem closed-loop action chunking as searching for the optimal action among a batch of samples drawn at each time step.
a arg min LB(a) + LF(a).
(5)
aeA
where A is the set of sampled action chunks, LB and LE are two criteria approximating the opti- mality with respect to the backward decision and forward plan, which we will describe next.
4.2 BIDIRECTIONAL CRITERIA
Backward coherence. To preserve temporal dependencies in closed-loop operations, a sequence of actions should (i) commit to consistent latent strategy time, and (ii) react smoothly to unexpected changes. These desired properties motivate us to keep the action chunk selected at the previous time à == (ل,"1") (,41-11. between the new action chunk and the prior across
(t-1) } as prior, and minimize the weighted Euclidean distance overlapping steps:
LB LB=
Σ
p"
(1) aitr a
(t-1)
6
Here, is decay hyperparameter to account for growing uncertainty over time. By default, use the L2 norm as the distance metric between predicted actions, while other metrics, such as LI or cosine distance, ilar latent strategies between consecutive steps while allowing for gradual adaptation to unforeseen transition dynamics.
also be effective (see Appendix (.6)). This backward objective encourages sim-

---

Published as a conference paper at ICLR 2025
ZIZZ Expert Palicy
Horizon=1
Horizon=5
Horizon=10
Probabiiiy
prosable
Actions
(a) long-idle demo, low-noise env
(b) mid-idle demo, mid-noise env
(c) short-idle demo, high-noise env
Figure 5: Effect of action horizon on idle actions in I-dimensional simulations. All policies share the same prediction length I. Long action horizons lead to idle distributions closer to the long-idle expert in low-noise environments, whereas shorter action horizons align more closely with the short-idle expert in high-noise envi- ronments. When both idling and noise non-negligible, moderate action horizon performs the best.
Nevertheless, the backward criterion alone presents a potential caveat: the prior chunk could be suboptimal due to the lack of information at the previous time step (e.g., unexpected object motions). In such cases, selecting the next action chunk based solely on the prior may perpetuate suboptimality. Ideally, the sequential decision-maker makinn process should effectively correct suboptimal plans based on the latest observations. We next address this need through another forward criterion.
Forward contrast. Our design of the forward criterion is motivated by the need to identify the most optimal plan from a set of candidates. Within the same latent strategy, suboptimal samples may arise from (i) low likelihood under the learned model and (ii) divergence between the learned model and expert policy. To address this, we draw inspirations from LLM decoding techniques (Wang et al., 2022 Li et al. (203)) and introduce a forward contrast criterion. Specifically, we compare each candidate sample with two sets of reference samples: one positive set from stronger policy and a negative set from a weaker one. The stronger policy is obtained from a well-trained checkpoint, whereas the weaker policy is taken from early underfitting checkpoint and is expected to be further from the expert policy. Our forward objective is thus framed as minimizing the average distance between candidate plan and the positive samples while maximizing its average distance from the negative ones,
Σ -2 ( E.A+
1 LF= N
Σ
Σ
Σ
(e) detr
aitrl])
R-EA-
where AA = A {a} is the positive set predicted by the strong policy म. A is the negative set predicted by the weaker one x², and N is the sample size.
Fig. iillustrates the combined effects of the backward coherence and forward contrast criteria on sample selection. Since samples in A+ and A not all subject to the same strategy, we trim each set by removing samples that deviate significantly from the previous decision. This is achieved by summing over the K smallest distance values for in the positive and negative sets in Eq. (7). The full process of our decoding method is outlined in Algorithm | Since all steps in BID can be computed in parallel. the overall computational overhead remains modest on modern GPU devices. Please refer to Appendix B for additional discussions about our decoding method.
5 EXPERIMENTS
In this section, we present series of experiments to answer the following questions:
How does our theoretical analysis on action chunking manifest under different conditions?
2. Can our decoding method improve closed-loop operations of policies built with action chunking?
Does our decoding method perform well across different policies, tasks, and environments?
4. Is our decoding method scalable to larger sample sizes and compatible with existing methods?
To this end, we will first validate our theoretical analysis through one- -ne-dimensional diagnostic sim- ulations. We will then evaluate BIID on seven tasks across three simulation benchmarks, including
7

---

Published as a conference paper at ICLR 2025
Vanilla
Warmstart
EMA
BID
1.0
0.8
1.0
0.8
0.5
0.6
0.7
0.9
0.4
0.9
0.7
0.4
0.5
0.4
0.8
0.0
0.8
0.6
0.3
0.4
0.1
push-t
lift
square
transport
tool hang
kitchen
Figure 6: Comparison of different inference methods for closed-loop operations of diffusion policies. Each method is evaluated for 100 episodes on seven manipulation tasks in simulation benchmarks. Results are averaged across three seeds. BID significantly outperforms existing inference methods most tasks.
Relative Perf Gain
Relative Perf Gain
0.5
0.3
0.4
0.2
0.2
0.1
0.1
0.0
0.0
15
Warmstart EMA
BID Warmstart EMA +BID +BID
Number of Samples
Figure 7: BID benefits from large sample sizes (left) and complements existing inference methods (right). Each method is evaluated seven simulation tasks across three seeds. Relative performance gain is measured with respect the vanilla baseline. When combined with EMA, BID results in 46% relative improvements.
RoboMimic (Mandlekar al. 2022), and Franka Kitchen (Gupta al. 2020). We will subsequently examine the generality and scalability of our method under various base policies, sample sizes, and stochastic noise. We will finally assess the effectiveness of BID in two challenging real-world tasks that require interactions with dynamic objects.
Push-T Chi al. 2023
5.1 ONE- SIE-DIMENSIONAL DIAGNOSTIC EXPERIMENTS
Setup. We start with a diagnostic experiment in a one-dimensional state space {so: .1, ,80 where so is the starting state and is the goal state. The demonstrator plans to move forward by one step in each state, except in 85 where it pauses unless the last five states visited were 8s- Each forward move has probability of 1 8, where 8 denotes the level of stochastic noise in the environment (as described in (3.2). Given these demonstrations, we train a collection of policies with different action horizons h 2, 3, 5, 7, 10}. We investigate under what action horizon learner can better imitate the distribution of idle actions taken by the expert over multiple rollouts.
Result. As shown in Fig. when the environment is deterministic (ō 0.0), larger action horizons capture the expert distribution better, consistent with Corollary 2 With an action horizon of 10, the learner achieves zero total variation distance with the expert distribution. Conversely, when the environment is highly stochastic (6 = 0.4), an action horizon of I outperforms all other learners, corroborating with Corollary 3 With moderate noise (8 0.2), there is no monotonic pattern and the optimal policy turns out to be the one with action horizon 5. Refer to Appendix A. for more detailed results. This controlled experiment validates the tradeoff identified Proposition which will further demonstrate in robotic manipulation subjected to stochastic noise in 5.2.3
5.2
SIMULATION EXPERIMENTS WITH STOCHASTIC NOISE
Next, we evaluate our decoding algorithm on seven simulation tasks of robot manipulation. We will first compare BID with existing inference methods in closed-loop operations. We will then assess the efffcttiveness of our method under different conditions, including policy classes, sample sizes, and levels of stochastic noise.
5.2.1 COMPARISON WITH EXISTING INFERENCE METHODS
Setup. each manipulation task, we use Diffusion Policy (Chi et al. (203)) trained on human demonstrations the base policy. We evaluate BID with batch size of N 16 and mode size of
K 3. We consider three existing inference methods as baselines:
Vanilla (Ci et al. 2023 Execute the first action of sampled chunk in closed-loop manner.
8

---

Published as a conference paper at ICLR 2025
Stochastic Noise
0.0
1.0
1.5
Sample Size
Success (%)
Time (ms)
Vanilla Open-I.opp
64.0
26.9
13.0
1
(vanilla)
49.1
13.2
BID Open-Loop
66.1
31.4
16.0
8
(ours)
52.9
25.2
Vanilla Closed-Loop
48.9
38.3
29.5
16
(ours)
54.2
25.9
BID Closed-Loop
54.4
45.3
31.7
32
(ours)
54.4
Table Success rates of VQ-BeT the Push-T task under various noise conditions. Closed-loop BID substantially outperforms other methods in stochas- tic environments, See Table detailed ablations.
Table 2: Success rates and inference times of VQ- BeT across varying sample sizes. BID benefits from a larger sample size at the cost of a doubled compu- tational overhead. measured on an A5000 GPU.
Warmstart (Janner et al. 2022: Similar to Vanilla, but warm-start the initial noise for the diffu- sion process from the previous decision.
Exponential Moving Average (EMA) (Zhao et al. 2023b); Smooth action chunking by averaging a new prediction a with the previous one à for each overlapping step a, = Aa, + A)ā,. This
method is also known temporal ensembling. By default, set 0.5.
We evaluate each method for 100 episodes and average the results across three random seeds. Please refer Appendix C for implementation details.
Result. Our main observation is that while existing inference methods offer some benefits for closed-loop operations, they lack robustness. As shown in Fig. [6] Warmstart yields mild perfor- mance gains on average, but degrades performance on 3 out of tasks. Similarly, EMA leads to competitive results several tasks, yet exhibits performance drops in tasks. We conjecture that this robustness issue stems from independent sampling across chunks; when successive chunks fol- low distinct latent strategies, averaging them may not yield plausible strategy, as further discussed in Appendix A.44] In comparison, BID offers substantial gains across all tasks. Notably, BID pro- vides 32% relative improvements over the vanilla baseline, significantly outperforming EMA on Pust-T, Lift, Square, and Tool Hang, while achieving competitive performance on the other tasks.
5.2.2 SCALABILITY AND COMPATIBILITY OF BID
Setup. We further assess two key properties of BID: scalability with growing batch sizes and compatibility with existing inference methods. For scalability, we experiment with batch sizes of (1,4,8, 8, 12, 16). For compatibility, we apply BID on top of two other baselines, Warmstart and EMA. The results are averaged across seven simulation tasks and three random seeds.
Result. As shown in Fig. 7 our method benefits from the large batch size, with performance gains not yet saturated at the default batch size used in $5.2.1 Morcover, the strength from BID is com- plementary to that of existing inference methods. Notably, combining BID with EMA further boosts the relative performance gain from 32% to 46%. These properties highlight the potential of our method in practice.
5.2.3 GENERALITY AND EFFICIENCY OF BID
Setup. We next extend our experiment VQ-BET (Lee al., 2024), another state-of-the-art robot policy built with autoregressive transformers. We use the public checkpoint on the Push-T task provided by LeRobot (Cadene et al. (202)) as the base policy. We use a checkpoint terminated 100 epochs as the weak policy in forward contrast. To simulate stochastic conditions, we add temporally correlated Gaussian noise to the executed action each step, scaled by the action magnitude. We measure the computational time on desktop equipped with an NVIDIA A5000 GPU.
Result. Table T summarizes the results of the baseline and our method. The vanilla random sampling performs significantly worse than BID in both closed and open-loop operations. Notably, the vanilla open-loop approach exhibits a rapid performance decline as the environment becomes increasingly stochastic. Even in closed-loop operations, the vanilla baseline still experiences a significant perfor- mance drop. In comparison, the closed-loop BID demonstrates much higher robustness to stochastic noise. Table 2 details the computational overhead associated with BID varying batch sizes. The result shows that the performance gains of our method come with doubled computational overhead. We expect that this overhead will be less of constraint with higher-end GPUs.

---

Published as conference paper at ICLR 2025
Vanilla
BID
Success Rate (%6
Success Rate (%)
80
60
20
Static
Dynamic
Vanilla Open
Vanilla Closed
EMA
BID
Figure Success object delivery. Each method-sctting evaluated across 20 episodes. BID achieves much higher success rate than the vanilla baseline, effectively handling the diverse demonstrations and dynamic target.
Figure 9: Success of cup replacement in the dynamic setting. Each method evaluated across 20 episodes. Existing methods degrade substan- tially under slow cup movements, whereas BID retains strong performance.
5.3 REAL-WORLD EXPERIMENTS WITH DYNAMIC OBJECTS
We finally evaluate BID through two real-world experiments that require rapid reactions to dynami- cally moving objects.
Setup. We consider two pick-and-place tasks, where the target object undergoes unexpected move- ment during evaluation. the dynamic placing task, Franka Panda robot is required to deliver an object into moving cup held by human subject. In the dynamic picking task, URS robot required to grasp a moving cup pulled by a string and place onto a nearby static saucer. In both tasks, evaluate the performance of BID applied pre- pre-trained diffusion policies. For further experimental details, please refer to Appendix C.2
Result. Figs. aagg../aanddooooo compares the results of different inference methods on the two real- weal-world tasks. Vanilla random sampling struggles to handle the diverse demonstrations and dynamic movements, resulting in significantly lower success rates. In contrast, BID achieves high success rates in both static and dynamic conditions. Notably, in the dynamic picking task, BID achieves a 2x higher success rate than all other baselines, highlighting its potential for dynamic object interactions.
Other experiments.
Please refer to Appendix A for additional analyses and ablations.
CONCLUSION
Summary. We have analyzed the strengths and limitations of action chunking for robot learning from human demonstrations. Based on our analysis, we proposed Bidirectional Decoding (BID), an inference algorithm that takes into account both past decisions and future plans for sample selection. Our experimental results show that BID can consistently improve closed-loop operations, scale well with computational resources, and complement existing methods. We hope these findings provide new perspective on addressing
challenges of generative behavioral cloning at test time.
Limitations. One major limitation of BID lies in its computational complexity. While the decoding computation can be fully parallelized on modern GPUs, may remain expensive for high-frequency operations low-cost robots. Designing algorithms that can generate quality yet diverse action chunks under batch size constraints can be an interesting avenue for future research. Additionally, analysis and method have been limited to policies with short context lengths, driven by their em- pirical effectiveness with limited human demonstrations. Developing techniques capable of learning robust long-context policies be another compelling direction for future research.
REPRODUCIBILITY STATEMENT
com/ iungLiiagLU http://iidddfffffii di diffusion and our code for the experiments with VQ-BET is available at http:/tu.ooooJouaaaaaaaaaa Jwwp//uubbybbbrre Haw.eeerrr-aaa///iieeee lerobot. Additionally, detailed descrip- tions of our experimental setups are available in Appendix and complete proofs of our theoretical claims can be found in Appendix
Our code for the experiments with Diffusion Policy is available at httpp::/
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like thank Anikait Singh, Eric Mitchell, Kyle Hsu, Moo Jin Kim, Annie Chen, Sasha Khazatsky, Rhea Malhotra and other members of the IRIS lab for valuable comments on this project. We would like to thank Yifan Hou, Huy Ha, Chuer Pan, Austin Patel, and the REAL lab for helpful support in setting up real-worke experiments. We would like to thank Asher Spector and Danny Tse for their insightful feedback on the theoretical analysis. This work is supported by the Al Institute, NSF CAREER award, and an SNSF Postdoc fellowship.
ONR grant 1000014-21-12000
REFERENCES
Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning, A survey of robot learning from demonstration. Robotics and Autonomous Systems, (7(5):469-485 May 2009.
Christopher G. Atkeson and Stefan Schaal. Robot Learning From Demonstration. In Proceedings of the Fourteenth International Conference on Machine Learning, ICML '97, pp. 12-20, July 1997.
Akhil Bagaria and George Dimitri Konidaris. Option discovery using deep skill chaining.
In International Conference on Learning Representations, 2020. URL hhtpss//appi.
semant wwcccccchlccccoo.cccccccc wwschollar.org/orrpptteeooo wwwp/oor//.eeooooo ID: 209450129922
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in Neural Information Processing Systems, 36, 2024.
Homanga Bharadhwaj. Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmen- tations and action chunking. 2024 IEEE International Conference on Robotics and Automation, 2023a.
Homanga Bharadhwaj. Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Aug- mentations and Action Chunking. September 2023b.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi.0: A lanoonnlnnuagg-atttto flow model for general robot control. arXiv preprint wrXiv:2410.24166 2024.
Francesco Borrelli, Alberto Bemporad, and Manfred Morari. Predictive control for linear and hybrid systems. Cambridge University Press, 2017.
David Brandfonbrener, Stephen Tu, Avi Singh, Stefan Welker, Chad Boodoo, Nikolai Matni, and Jake Varley. Visual backtracking teleoperation: A data collection protocol for offline image-based reinforcement learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11336-1134 11322. IEEE, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wes Edward Lee, Sergey Levine, Yao Lu, Hen- ryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao. Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Lanion-Language-Ace Models Transfer Web Knowledge to Robotic Control, July 2023a.

---

Published as a conference paper at ICLR 2025
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Roboties Transformer for weal-World Control Scale. In Robotics: Science and Systems XIX. Robotics: Science and Systems Foundation, July 2023b. ISBN 978-0- 9923747-9-2.
Remi Cadene, Simon Alibert, Alexander Soare, Quentin Gallouedec, Adil Zouitine, and Thomas Wolf. Lerobot: State-of of-te-of-thh-aeee the- art machine learning for real-world robotics in pytorch. https:
wwa.thub.commuuggingggg.o aate/aacceeeeebbrr lerobot 2024.
Cheng Chi, Siyuan Feng. Yilun Du Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Robotics: Science and Systems XIX. Robotics: Science and Systems Foundation, July 2023. ISBN 978 778-0-9923747- 9-2.
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching with- out in-the-wild robots. arXiv preprint arXiv:2402. 10329, 2024.
Open X--Embodiment Collaboration. Open X-Embodiment: Robotic Learning Datasets and RT-X Models, December 2023.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal Confusion in Imitation Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee. Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, In Conference on Robot Learning. pp. 168.168. PMLR, 2022.
Markus Freitag and Yaser Al-Onaizan. Beam Search Strategies for Neural Machine Translation. In Thang Luong, Alexandra Birch, Graham Neubig, and Andrew Finch (eds.). Proceedings of the First Workshop on Neural Machine Translation, pp. 56-60, Vancouver, August 2017. Association for Computational Linguistics.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding, February 2024a.
Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile ALOHA: Learning Bimanual Mobile Manipu- lation with Low-Cost Whole-Body Teleoperation, January 2024b.
Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, and Dorsa Sadigh. Eliciting compatible demonstrations for multi-human imitation learning. In Conference on Robot Learning, pp. 1981 1991. PMLR, 2023.
Abraham George and Amir Barati Farimani. One act play: Single demonstration behavior cloning with action chunking transformers. arXiv preprint arXiv:2309, 10175, 2023.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, pp. 1025-1037. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2025
Philippe
Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies, May 2023.
John Hewitt, Christopher Manning, and Percy Liang- Truncation Sampling Language Model Desmoothing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3414-3427, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural Computation, 9:1735- 1780, 1997. URL
www.seewtwwncccccccccccoooooo. org
1 915014
Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown, and Ken Goldberg, Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. arXiv preprint UrXiv:2109.08277 2021a.
Ryan Hoque, Ashwin Balakrishna, Carl Putterman, Michael Luo, Daniel S Brown, Daniel Seita, Brijen Thananjeyan, Ellen Novoseller, and Ken Goldberg. Lazydagger: Reducing context switch- ing in interactive imitation learning. In 2021 IEEE 17th international conference on automation science and engineering (case), pp. 51205022255 509. IEEE, 2021b.
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, and Brian Ichter. Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents. Thirty-Seventh Conference on Neu- ral Information Processing Systems, November 2023.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Confer- ence on Robot Learning, pp. 991-1002. PMLR, R, 2022.
Michael Janner, Yilun Du Joshua B. Tenenbaum, and Sergey Levine. Planning with Diffusion for Flexible Behavior Synthesis. In International Conference on Machine Learning (ICML), May 2022.
Yiding Jiang, Evan Zheran Liu, Benjamin Eysenbach, Zico Kolter, and Chelsea Finn. Learning options via compression, 2022. URL arxiv. org/ /bs 2212.2222222 04590
Yushan Jiang, Wenchao Yu, Dongjin Song, Lu Wang, Wei Cheng, and Haifeng Chen. Fedskill: Privacy preserved interpretable skill learning via imitation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1010-1019, 2023.
Liyiming Ke, Jingqiang Wang, Tapomayukh Bhattacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp- 6195- IEEE, 2021.
Michael Kelly, Chelsea Sidrane, Katherine Campbela and Mykel J Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8077-8082 IEEE, 2019.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. ARGS: Alignment as Reward-Guided Search. In The Twelfth International Conference on Learning Representations, October 2023.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungiae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pan- nag R. Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Bai- jal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul
13

---

Published as a conference paper at ICLR 2025
Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jack- son, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mir- chandani, Daniel Morton, Tony Nguyen, Abigail O' O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang. Patrick Yin, Yunchu Zhang, Oshert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J. Lim, Jitendra Malik, Roberto Martin-Martin, Sub- ramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A Large-Scale In- In-The-Wiid Robot Manipulation Dataset, March 2024.
Alex Kuefler and Mykel Kochenderfer, Burn-in demonstrations for multi-modal imitation learning. arXiv preprint 1:X0::7710000000 2017.
Shankar Kumar and William Byrne. Minimum Bayes-Risk Decoding for Statistical Machine Trans- lation. In Proceedings of the Human Language Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 169-176, Boston, Massachusetts, USA, May 2004. Association for Computational Linguistics.
Lucy Lai, Ann Zixiang Huang, and Samuel Gershman. Action chunking as policy compression. 2022.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference robot learning, pp. 143-156.10 PMLR, 2017.
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior Generation with Latent Actions, March 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast Inference from Transformers via Specu- lative Decoding. In Proceedings of the 40th International Conference Machine Learning, pp. 19274-19286. PMLR, July 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive Decoding: Open-ende ended Text Generation as Optimiza- tion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp- 12286-12318 Toronto, Canada, July 2023. Association for Computational Linguistics.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in neural information processing systems, 30, 2017.
Johan Löfberg. Oops! cannot do it again: Testing for recursive feasibility in mpe. Automatica, 48 (3):550-555, 2012.
Xiao Ma, Sumit Patidar, lain Haughton, and Stephen James. Hierarchical Diffusion Policy for Kinematics-Aware Multi- Task Robotic Manipulation, March 2024.
Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4420, May 2020.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. In Proceedings of the 5th Conference on Robot Learning, pp. 1678-1690. PMLR. January 2022.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally Typical Sampling. Trans- actions of the Association for Computational Linguistics, 11:102-121, 2023.
Kunal Menda, Katherine Driggs-Campber and Mykel J Kochenderfer. Ensembledagger: A bayesian approach to safe imitation learning. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5041-5048. IEEE, 2019.
14

---

Published as a conference paper at ICLR 2025
Mathias Müller and Rico Sennrich. Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 259-272, Online, August 2021, Association for Computational Linguistics.
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance. In 8th Annual Conference on Robot Learning, September 2024.
Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency Policy: Ac- celerated Visuomotor Policies via Consisteney Distillation, May 2024.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing sys- tems, 35:22955-22968, 2022.
Lucy Xiaoyang Shi, Archit Sharma, Tony Z. Zhao, and Chelsea Finn. Waypoint- Imitation Learning for Robotic Manipulation. In Proceedings of The 7th Conference Robot Learning, pp. 2195-2209. PMLR, December 2023.
Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: Learning robotic hand imitator by watching humans on youtube. Robotics: Science and Systems (RSS), 2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Gokul Swamy, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Causal imitation learning under temporally correlated noise. In International Conference on Machine Learning, pp. 20877-20890. PMLR, 2022.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy, arXiv preprint arXiv: 12213, 2024.
Ashwin Vijayakumar Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse Beam Search for Improved Description of Complex Scenes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. Coff-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203. //1 [cs], April 2022.
Sean Welleck, Ili Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural Text Generation With Unlikclihood Training. In International Conference on Learning Representations, September 2019.
Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting Copycat Agents in Behavioral Cloning from Observation Histories. In Advances in Neural Information Processing Systems, volume 33, pp. 2564-2575. Curran Associates, Inc., 2020.
Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint arXiv:2309.10 13037, 2023.
Annie Xie. Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning Latent Repre- sentations to Influence Multi-Agent Interaction. In Proceedings of the 2020 Conference on Robot Learning, pp. 575-588. PMLR, October 2021.
15

---

Published as a conference paper at ICLR 2025
Xiaomeng Xu, Huy Ha, and Shuran Song. Dynamics-Guider Guided Diffusion Model for Robot Manipu- lator Design, February 2024.
Tianxiang Zhao, Wenchao Yu, Suhang Wang, Lu Wang, Xiang Zhang, Yuncong Chen, Yanchi Liu, Wei Cheng, and Haifeng Chen. Skill disentanglement for imitation learning from sub- optimal demonstrations. In Proceedings of the 29th ACM SIGKDD Conference on Knowl- ACM, August 2023a. doi: /ttp://ddx doi org/10 1145/3580305 3599506
edge Discovery and Data Mining, KDD '23, pp. 10.1 10.1445/58833033999566
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Robotics: Science and Systems (RSS) 2023, April 2023b.
Hankz Hankui Zhuo, Yantian Zha, Subbarao Kambhampati, and Xin Tian. Discovering underlying plans based on shallow models. ACM Transactions on Intelligent Systems and Technology (TIST), 1132):130,, 2020.
16

---

Published as a conference paper at ICLR 2025
demo distribution
previous decision
negative samples
positive samples
Figure 10: Distributional interpretation BID. The backward criterion (Equation 6 favors samples close to the past decision; the forward criterion (Equation 7) promotes samples with a high likelihood under the target distribution.
Action Horizon
Noise Level
7
10
0.0
4.03
2.07
1.54
1.06
0.00
0.2
1.43
0.94
0.39
0.71
1.32
0.4
0.36
0.42
0.59
0.835
1.11
Table 3: Total variation distance between the action distributions of each model and the expert in environments with varying noise levels. Lower values indicate better performance.
A ADDITIONAL EXPERIMENTS
A.1 ONE-DIMENSIONAL SIMULATIONS
In addition to Fig. 5 summarize the total variation distance between each learned policy and the demonstration in the one-dimensional simulation. Our results indicate that shorter action horizon more effective in noisier environments, whereas a longer action horizon yields better performance in static environments.
A.2 ACTION HORIZON vs. CONTEXT LENGTII
Setup. Our work builds the premise that the action horizon is longer than the context length, as commonly designed for recent policies. While BID mitigates the inherent limitations this de- sign choice through test-time decoding, important question remains: could extending the context length yield stronger policies? To understand this, we trained diffusion policies with varying com- binations of action horizons and context lengths on the Push-T task. Specifically, we use short context length (c 2) and short action horizon (h = 2) as our baseline, set the prediction length equal to the action horizon I h, and incrementally increase these parameters to larger values 6, 10, 14 to assess their impact.
Result. Fig. IT compares the performance the policy learned with different Ah = h c. As ex- pected, the policy with both short action horizon and short context length does not perform well, due to its limited capability to model long-range temporal dependencies. Interestingly, extending the context length initially boosts performance (Áh -4), but this trend reverses the context length becomes too long (Ah < -8), likely due to overfitting to an increased number of spurious fea- tures. In contrast, expanding the action horizon results in more robust performance improvements, validating its pivotal role imitation learning from human demonstrations.
A.3 ABLATION STUDY FORWARD CONTRAST
Setup. To understand the effect of forward contrast (Equation 7), we evaluate the full version of our method against three reduced variants in open-loop operations: Vanilla (without forward contrast), Positive (without negative samples), and Negative (without positive samples). Similar to $52..2 our ablation study is conducted in seven simulation tasks, each with three random seeds.
Result. Fig. 12 summarizes the result of this ablation study. Notably, both positive and nega- tive samples are essential for effective sample selection, and omitting either leads to significant performance declines. Without negative samples, maximum a posteriori estimation, which can result in suboptimal decisions due to modeling errors. Conversely, without positive samples, the sampling process may be biased towards rare instances.
decoding method reduces to approximate
17

---

Published as a conference paper at ICLR 2025
0.9
Test Score
0.8
0.7
-12
A 4hh
Figure 11: Effect action horizon (h) and context length (e) on diffusion policies in the Push-T The baseline is set at / = 2 and e = 2, with Ah = h c = 0. Extending the action horizon (h > 2) consistently improves performance, whereas extending the context length (c > 2) can cause substantial performance de- clines. Each model is trained for 5k epochs. Results averaged over the last five checkpoints.
pusht
toolhang
Relative Test Score
1.00
Success Rate
0.75
0.79
0.50
0.78
0.25
0.77
0.00
Vanilla
Positive
Negative
Contrast
Method
Decay Rate
Figure 12: Effect of positive and negative samples on forward contrast. Performance ablated variants forward contrast is evaluated across seven simu- lation tasks. The absence of either positive or nega- tive samples prevents achieving the full performance gains observed with the contrast objective.
Figure 13: Effect of the decay rate for the exponen- moving average. each task, measure relative performance among different decay rates. The optimal decay rate varies by task, leading to a practical challenge of identifying universal tempo- ral ensembling strategy (Zhao et al. 2023b).
This result highlights the importance of both components and suggests the potential for extending this paradigm in future work.
A.4 CHALLENGES FOR TEMPORAL ENSEBMLING
EMA exhibits competitive performance in Fig. However, tuning its decay rate can be difficult in practice. Fig. 13 shows the sensitivity of EMA the decay rate across three different tasks, where the optimal choices differ significantly. We conjecture that this high sensitivity stems from the variability in the latent strategies between consecutive predictions. When consecutive predictions follow similar strategies, lower decay rate (i.e., stronger moving average) can enhance smoothness and improve performance. Conversely, when consecutive predictions diverge in their underlying strategies, averaging them introduce adverse effects. Our method promotes coherence in latent strategies and thus effectively complements temporal ensembling, as evidenced Fig. 7
A.5 ABLATION STUDIES ON VQ-BET
Table presents the performance of various sampling methods with VQ-BeT (Lee et al. 2024) on the Push-T task under different levels of environment stochasticity. We compare BID against vanilla sampling and EMA, for both open-loop and closed-loop executions. BID consistently outperforms the baselines, particularly in highly stochastic settings, with close-loop BID attaining the best per- formance on average. Moreover, the performance of EMA degrades significantly as the environment gets more noisy, likely due the divergence in strategies between consecutive action chunks. Ad- ditionally, we conduct an ablation study on the forward contrast objective, isolating the roles of positive samples (from strong model) and negative samples (from weaker model). We observe
18

---

Published as a conference paper at ICLR 2025
Stochastic Noise
Average
Stochustic Noise
1.5
Average
Vanilla
64.0 1
26,9 1:288.2..
13.0 1 0.4
34.6 1.7
Vanill
48.9 1: 2.7
38.3
3,4
29.5
09
38,9
EMA
64.1 1
2.36.:3.3
1299
1.1
34,9 1.3
EMA
52.6 1: 2.9
35.7
1
1844
2.3
35.6
1.4
Backward (ours)
64.0
2.
35.1 14
Backward (ours)
52.7
4.2
29
41.4
Positive (ouns)
29.7 0.
15.7
37.0
Positive (ours)
53.8
30.4
42.8
Negative (ours)
65.1
2..6
14.8
24
35.5
Negative (ours)
53.0
14.3222
30.8
42.7
BID (ours)
66.1 +: 3,5
31.4 +:
16.0
+ 1.2
37.8
1.6
BID (ours)
54.4 +:
1.8
45.3 3.8
31.7
0.3
43.8
(a) Open Operation
(b) Closed Loop Operation
Table 4: Success rates of VQ-BeTT on the Push-T task under various conditions and sampling methods. The left table shows open-loop success rates, while the right table shows closed-loop success rates. BID consistently outperforms vanilla counterparts in both settings.
Vanilla
L2 (Ours)
L1 (Ours)
Cosine (Ours)
Square
0.68 + 0.06
0.76 1 0.03
0.76 0.04
0.73 0.04
Lift
0.12 + 0.02
0.58 + 0.06
0.68 0.05
0.70 0.02
Kitchen
0.22 + 0.04
0.64 + 0.05
0.70 + 0.04
0.61 + 0.03
Table 5: Effect of distance metrie. We evaluate BID with different distance metrics on three robot manipulation tasks. Regardless of the chosen metric, BID substantially outperforms the vanilla baseline. Notably, the LI distance yields an even stronger performance than the default L2 distance used our other experiments.
that both types of samples contribute to the strong performance of BID across varying levels of environment stochasticity.
A.6 CHOICE OF DISTANCE METRICS
As described in 4.2 our method uses the L2 distance the default distance measure for quantifying similarity between action chunks. Yet, L.2 distance is just one several viable options. To assess the sensitivity of our approach the choice of distance metrie, extend the experiments in $5.21 to further evaluate BID with alternative choices, including L1 and cosine distance. Results from three robot manipulation tasks, summarized in Table 5 demonstrate that BID achieves substantial performance gains regardless of the metric used. In particular, BID with L. distance surpasses the default I.2 distance in performance, highlighting the untapped potential of our method.
B ADDITIONAL DISCUSSIONS
Interpretation of our method. Our method makes changes to the learned policy; instead, it intervenes in the model distribution through sample selection. As illustrated Fig. 10 randomly sampled sequences may be misaligned with both the previous decisions and the target demonstra- tions. Given a set of candidates, the backward step first identifies the behavioral mode from the past decision stored in memory; the forward step then removes the samples with low likelihood under the target distribution using prior knowledge of positive and negative samples. By comparing sam- ples across time steps and model checkpoints, our method bridges the gap between the proposal and target distributions at test time.
Relation recent methods. Our method builds upon the receding horizon (Chi al. 2023 and temporal ensembling (Zhao et al.) (2023)) used in previous works, but with crucial distinctions. Receding horizon seeks a compromise between long-term consistency and short-term reactivity by using a moderate action horizon (e.g., half of the prediction length), which is inevitably sup-optimal when both factors are prominent. Temporal ensembling strengthens dependency across chunks by averaging multiple decisions over time; however, weighted-averaging operations can be detrimental when consecutive decisions fall into distinct modes. Our method more effectively addresses cross- chunk dependency through dedicated behavioral search and is not mutually exclusive to previous methods. We will demonstrate in the next section that combining our method with the moving average further improve closed-loop action chunking.
19

---

Published as a conference paper at ICLR 2025
name
value
batch size
16
mode size K
3
prediction length
16
temporal coherence decay P
0.5
moving average decay À
0.5
Table 6: Default hyper-parametters in experiments.
C ADDITIONAL DETAILS
C.I SIMULATION EXPERIMENT DETAILS
C.1.1 ENVIRONMENT DETAILS
Our simulation experiments are conducted on three robot manipulation benchmarks. We use the training data collected from human demonstrations in each benchmark.
Push-T: We adopt the Push-T environment introduced (Chi et al.) (023), where the goal is to push T-shaped block on table to target position. The action space is vro-dimensional end-effector velocity control. The training dataset contains 206 demonstrations collected by humans.
Robomimic:: We five tasks in the Robomimic suite (Mandlekar et al. 2022), namely Lift, Can, Square, Transport, and Tool Hang. The training dataset for each task contains 300 episodes collected from multi-human (MH) demonstrations.
Franka Kitchen: We use the Franka Kitchen environment from (Gupta et al. 2020). featuring a Franka Panda arm with seven-dimensional action space and 566 human-collected demonstrations. The learned policy is evaluated on test cases involving four or more objects (p4), a challenging yet practical task for robotic manipulation in household contexts.
C.1.2 IMPLEMENTATION DETAILS.
Our implementation of BID for Diffusion Policy is built upon the official code of Cf et al. (2023), with modifications made solely the inference process. The policy takes in state inputs and predicts chunk of 16 actions as outputs. For each simulation task, we train the model for 100-1000 epochs reach near-optimal performance. We evaluate it in closed-loop operations, i.e., action horizon is set to 1. For forward contrast, we train the weak policy for 10-100 epochs, resulting in suboptimal policy for each task. The core hyperparameters are summarized Table
Our implementation of BID for VQ-BeT (Lee et al. 2024) is built upon the code of LeRobot (Cadene al. 2024).. We use the best public checkpoint as the strong policy and a checkpoint trained for 100k iterations as the weak policy. Since BID requires sample diversity, set the temperature to be 0.5 for all methods in our experiments.
C.2 REAL-WORLD EXPERIMENT DETAILS
C.2.1 DYNAMIC PLACING
Task. We consider task where the robot is to deliver an object held in its gripper into cup held by a human. As shown in Fig. [4] this task comprises four main stages and presents two core challenges. First, due to the similar size of the object and the cup, the robot must achieve high precision to place the object accurately into the cup. Second, the position of the cup is not fixed, requiring the robot adjust its plans based the latest position continuously. This task mirross real-world scenarios where robots interact with a dynamic environment, accommodating moving objects and agents.
Demonstration. In light of temporal dependencies and style variations in human behaviors, we intentionally collect diverse set of demonstration data, differing in factors such as average speed, idling pause, and overall trajectory. We gather a total 150 demonstration episodes: 50 clean and consistent demonstrations, and 100 noisy and diverse demonstrations. All demonstrations success- fully accomplish the task. Additional, the location of the cup is fixed and static within each episode.
Robot. Following previous works (Chi et al.) 2023 Prasad et al. 2024), we use a Franka Panda the robot hardware and the vision-based diffusion policy for its operation. The robot is equipped
20

---

Published as a conference paper at ICLR 2025
(a) initialize
(b) approach
(c) decelerate
(d) place
Figure 14: Human demonstrations Franka Panda robot for real-world object delivery task. The robot is tasked with delivering an object held in its gripper into cup held human. Each demonstration consists of four main stages: (a) initialize the robot position randomly, (h) approach target cup, (c) slow down near target cup, and (d) release the object. The position of the target cup may change during an episode.
(a) initialize
(b) approach
(c) grasp
(d) pick
(c) place
nearby. The four main stages are (a) initializing the robot, (b) approaching the larget cup, (c) grasping target cup. (d) picking up the cup,
Figure 15: The robot is tasked with picking up cup and placing
and (e) placing the cup on the target saucer. The position of the target cup may change during an episode.
with two cameras: one ego-centric camera mounted at the wrist of the robot, one third-person person camera mounted at static bracket. Both cameras provide visual observations at resolution of 256 256 pixels. The robot operates at frequency of 10 Hz, with a prediction length of 16 time steps.
Evaluation. We evaluate our method comparison to vanilla random sampling under two con- ditions: static target, where the target cup remains fixed throughout the evaluation, and dynamic target, where the target cup gradually moved. In the dynamic setting, the location of the cup stays within the range of training locations, but the movement is not encountered during training. This evaluation protocol designed to explicitly assess the ability of the policy to react to unexpected dynamics in the environment. Each method-setting pair is tested over 20 episodes, with both the initial and target locations randomized across different episodes.
Result. We summarize the result of the real-world experiments in Fig. The success rate of vanilla random sampling is generally limited due to oscillations between different latent strategies, which quickly diverge from the distribution of demonstrations. This issue is particularly pronounced in the dynamic setting, where the vanilla baseline struggles to account for the target movements within an action chunk lasting for 1.6 seconds. In contrast, the proposed BID method significantly improves performance in both static and dynamic settings. Notably, BID maintains similar success rate in the dynamic setting in the static setting, suggesting its potential extend action chunking into uncertain environments.
C.2.2 DYNAMIC PICKING
Task. Next, we consider task where the robot is required to pick up cup and place it onto nearby saucer. The cup was pulled with string until the robot's gripper successfully grasped it. The task consists of five main stages, which are illustrated in Fig. [5] This setup also tests the robot's capability to interact with a dynamic environment, a critical challenge real-world applications.
Policy. We utilized the publicly available diffusion policy checkpoint from UMI (Chi et al.) (024)
without any additional fine-tuning. Notably, the policy was originally trained using demonstrations in static setting, where the cup's position remained constant throughout the task. Our experimental setup mirrored the one described by UMI, using the same UR5 robot hardware. This allowed us to directly evaluate the policy's transferability changes during the task. Due to the absence of an early checkpoint, we omitted negative samples in forward contrast, focusing solely positive consistency discussed in Appendix A.3
dynamic environment, where the cup's position
21

---

Published as a conference paper at ICLR 2025
Evaluation. We evaluated BID against three baselines: vanilla random sampling in both open- loop and closed-loop configurations, and EMA (closed-loop). These methods were tested under two conditions: static targer, where the cup remained in fixed position, and dynamic targer, where the cup was moved using the string. Each method-setting combination was tested across 20 episodes, with the initial positions of the cup and saucer kept consistent to ensure controlled comparisons.
The results, summarized in Fig. | highlight the challenges of the dynamic setting. Open- loop vanilla sampling performed poorly due to its inability to adapt to the cup's movements, often failing to approach the cup it was pulled. While closed-loop vanilla sampling showed improved reactivity, it suffered from inconsistent trajectories, resulting in jittery behavior when attempting to grasp and place the cup. Similarly, closed-loop EMA sampling demonstrated higher adaptability to environmental changes but often failed to firmly grasp the cup, likely due to the limitations of naive averaging, which compromises commitment to specific strategy. In contrast, BID achieved at least a 2x improvement in success rate compared to all other methods in the dynamic setting, while maintaining its performance the static setting, demonstrating both adaptability and precision in dynamic environments.
Results.
D ADDITIONAL DISCUSSIONS
Relation to Option Discovery. Action chunking and option discovery share similarities in mod- cling temporally extended actions. However, their designs and outcomes are often different. Option discovery typically aims to learn hierarchical policies, explicitly discovering high-level skills from low-level action sequences (Bagaria Konidaris 2020 Jiang et al. 2022, 2023; Zhao et al., 2023a). In contrast, action chunking operates directly on low-level action sequences with fixed horizons, by- passing the need for abstraction into bigh-level skills. This simplicity has proven effective in large- scale robotic foundation models (Team et al.) 2024 Black et al. 2024), addressing the scalability challenge that option discovery has yet to overcome. Future work could explore variable-horizz option discovery compelling alternative to action chunking, potentially combining the benefits of temporal abstraction with the scalability demonstrated by action chunking.
Relation to Long Context. Our analysis in highlights the benefits of extending the action hori- zon to better capture temporal dependencies across actions. Another natural approach to capturing these dependencies is to extend the context length. However, long-context policies often suffer from robustness issues, due to spurious correlations between past and future actions, as evidenced in Ap- pendix A2 and studied in de Haan et al. (2019); /Wen et al.] (2020). Nevertheless, the increasing availability of large-scale robotic datasets for pre-training and fine-tuning may help mitigate these challenges. Advances in leveraging long context may open up new opportunities in policy design. such as long-context transformers (Su et al. 2024) and recurrent neural networks (Hochreiter &
Schmidhuber 1997 Zhuo et al. 2020).
E PROOFS
In this section, will clearly write out the transition dynamics in the training environment to be Pusin($141 | St, at), the transition dynamics in the test environment to be Ptest(St+1 St, a;) and the implicit dynamics model of the training environment to be Perain implicit dynamies model attempts learn the dynamies of the training environment and not the test environment since our policy is assumed to have been trained only on data from the training environment.
St, at). Note that the
First, we establish the following lemma which will help us compare different function classes based on the information they have access to:
Lemma 4. Let L be convex function and let X and Y be two random variables. Let G be the class of functions g(X) that accept X an input. Then
min Exy x(X)G
Ex
min Ey ().((x,))c))xx)
eER
22

---

Published as a conference paper at ICLR 2025
Proof.
The left hand side is less than equal to the right hand side by the following logic:
Bx
min
By [4(
= Exx [Ry I
FER
2 min Exy ().((X a(XYEG
where we used (x)) arg min: Ex[ x)]. We get the inequality by recognizing that & G. For the reverse inequality, consider any g(X) E G:
E]A(((X))))( = Ex EY 9X)xxXI
2 Ex min Ev [C(](X) X.
= Ex
min By [C (), (),
With these two inequalities, we conclude.
Next, we prove the following lemma. This straightforward, and almost trivial, result provided as separate lemma because we simplify terms in this manner quite often throughout our proofs.
Lemma 5. Let L. be convex function and let X, Y be two random variables. Then,
Exy [P(X' xX) Ex,
((x)))((((x))xxxx
X
<min(Ex,y
+
5
where (XXXX and f argmin/ ((X,
Proof.
Ein - =)x +xx. Σ ((x))(((xx))xxxx)
X'XX
Σ P(X))(((((3)5((x1))
X'
<min[Bx,y [
Σ P(X)
X'XX
< [ii, (X(U()), Ex,
Σ P(X')e
X'XX
s minnExxy ((X, Y))] +
E. ASSUMPTIONS
Considering that recent policies often use short context length c, we assume the range of temporal dependency modeled by a (c, -oolccy is limited:
Assumption 1. The sum of context length and action horizon is less than the length of temporal dependency in expert demonstrations, c + h k.
23

---

Published as a conference paper at ICLR 2025
This assumption allows to focus the problem that is relevant us and allows us ignore edge cases. However, our analysis can be extended to the case where this assumption does not hold. the case where + h 2 k, the larger action chunk model will not get any advantage (making ag 0) since the additional states has observed in time steps т are irrelevant to at- Then, the longer horizon policy either attains the same expected loss the shorter horizon policy suffer greater loss from not having observed the recent past states.
Assumption 2. optimal Te.h must infer the unobserved states based on the observed states and actions by modeling the transition dynamics Pirain sp-11, ay-1) accurately for all time step t'. We consider this implicit model be the same for both ((c,b) and T(a.hdd)-
In other words, write using the law of total probability:
(cc,k) (telst- -ot-- ae- 11)
E Sy 1886 aa Parsin (aels) ( hh hy ag a---ut-1
and
((o,h=a) St-h- ai-h d----1) --
= E st
Perain ((1:1)) (aez)(aeele) (Re(8e-kit) ht-h-d--eh -h Ot
E.2 DEFINITIONS
We, first, analyze the effect of reducing context horizon. We show that, provided action horizon constant, decreasing context horizon causes performance of the optimal policy to decrease.
Consider (c, h)-policy (i.e., the policy has context length c and action horizon h) whose probability of taking action a, at time l in chunk generated the other hand, consider (c-1, pollic whose probability of taking action a, in a chunk generated at time 1 is referred to as ((++1,k) = ((c+1,)) (.))).))..) Lastly, consider a (k:, )-eepeert whose probability of taking action a at time it n²"
referred as ((e.h) : "T(,,A) (cr(st- On
Proposition 6. Let C be a non-linear, convex function. Let < k. Let G = {ar, 8t -t-kit-o-1 ktt and let C == (st-ett). Then,
Eg [[(TICLMM S
min Eg [[(TiCc)) ((c.))
min ((41.8)
In particular, this is an equality if and only if st-c-11 ~ Prest C) and 8 ~ Perain C) take only one and the same value almost surely.
Proof. We refer to the class of functions that accept a, and St-o-1i as inputs as X2. Similarly, the class of functions that do not accept as inputs but accept 8:-c-1:t as inputs is X1. The function class that accepts only st-est and not st-c-1 a₂ inputs are elements of Xo. Lastly, the function class that accepts St-att and az inputs, but not 8t-c-1, are elements of X-1-
min M(cti,a)EX;
EG [C(T(C+)))))
min Ba--- Be [ "") st--11, ac. c c|
Em, [(c
(Lemma
Esg ki 2:21 de ((T(a,A) /en (a d
Exg
E,
min Flons EXo Exo
(Lemma
Es, e-1 En, ki c 2i ((Tl(a)) "l cl
< Ens
min
"ccn) eXp
(Jensen's inequality)
24

---

Published as a conference paper at ICLR 2025
min Ttc.h)
Bag
[
[ -- ((F(c.))) x¹) al or, |
(Lemma
Use the law of total expectation to conclude. The equality conditions for Jensen's inequality provides us the equality condition for this relationship too.
Now, we formalize the definitions of Expected Observation Advantage and Maximum Inference Disadvantage.
Recall that, $3.2 we have two policies: *(a,)) and d() the former sees more recent states while the latter remembers more past states. First, we define an agent that gets access to all the information that both learners, combined, have: (c + hi-policy whose probability of taking action in a chunk generated at time h is
((c-d,)) ((o+d,N) (a: (alllss--ee h-d----tt - Ot-bott1).
Observe that (c+d,)) has access to more context than T(c,h), particularly the knowledge of states 8t-h St-h-o-me dit h-
Definition (Expected Observation Advantage (og)). We know, using Proposition 6 (e++d, has lower divergence with respect to π than T(c,))- We say that the advantage T(c+d,)) gets from the extra information is og. More formally, we say that
Obp min E [A(T(CA)) >0|c] T(c,))
[((FFctaM)) x" 1|
min Teel
where C is defined as Proposition [ $t-h-d-ot h-e-1 can be deterministically inferred by T(c,h) when the expert policy is inde- pendent of them.
Clearly, as 2 particular, as 0 when
Definition (Maximum Inference Disadvantage (ey)). Consider the maximum divergence that can be accumulated by the (c, h d)-policy from not knowing the recent states at time steps --h-dd-:-hs and let that be Cy. More formally, we say that, for fixed C' from Proposition I any states in S = [st-ket St, any 24-k4, and any á bt-h-d41t- & st-n-dtia-h- the following holds:
d- # 8t 1--h-d411-h at-htt1);; S Sey.
(9)
Here, ((cdd)) = arg min min ((()))- TT [C] is the optimal (c+ d, h))policy.
To define a and es. prove second version of Proposition 6 Consider (c, h))policy whose probability of taking action a, at time a chunk generated at referred to as T(aA)) == the other hand, consider (c 1, h + 1)-policy whose probability of taking action a, in chunk generated time is referred as (e-e1, == (oo-aa-n((((((--ett--1 -et--11). Lastly, consider (k: 1)-expert whose probability of taking action at at time / is n².
(e.,)) (#r(8t-er))
Proposition 7. Let L be non-linear, convex function. (ar, St-kit St, 2e-ki and C (st-ait-1, at-1 Then,
Let < k. Let G
min min Eg [((T(e))) x >>|c a(o-1,A-1) (go,h)
Eg (
In particular, this is an equality if and only if the state st PLes( C) and Sn Pieain (- C) takes on only one and the same value almost surely.
Proof. The proof is similar to that of Proposition 6
We refer to the class of functions that accept and St-ezt as inputs as X2. Similarly, the class of functions that do not accept as inputs but accept St-est as inputs is X1. The function class that accepts only st-ct-1 and not 8, a, inputs
25

---

Published as a conference paper at ICLR 2025
elements of Xg. Lastly, the function class that accepts St-ett1 and ds inputs, but not 8t, elements of X-1.
min T(2.h)
Eg ([(T(..):
Bu, -e (T x *') of aa col@ (
Eug
min
(Lemma
" EXx
Eee Pec-te---- C [(T( of ar, c c
Eai
min
(Lemmal
1,411 ENo
< Fat
min
Eet [B ( [(
as c
C
11) EXO
(Jensen's inequality)
min
Eut
[Bn.
an
|c]
(Lemma
EX
Use the law of total expectation to conclude. The equality condition can be seen from the equality condition of Jensen's 's inequality.
Using this, we can define < and af in a similar manner:
Definition (Expected Observation Advantage (of)), Recall that we have two models: ((c,)) and ((a,++d) and a hypothetical (c d, -pooicy that has access to the information both our learners have (as in Eq. (8) and Eq. (9)). Observe that (co-d. has access to ticularly the knowledge of states st-b-d41shh- Therefore, we know, using Proposition 7 (,
context than b-d)) par-
has lower divergence with respect to π² than (c.. We say that the advantage ((++-d,)) gets from the extra information is ay. More formally, we say that
0
EE [[(TTc.)+a)) (eckk
min
TE [E((T(++M):=T m)|c m(rldd))
min
(10)
where C' is defined in Proposition 1. Clearly, ay 2 O. In particular, ay when ((c infer $--h-d+1:-h perfectly. This makes sense-in the static environment, observing these states does provide any advantage since the optimal ((c.h+d) actions taken at those time steps (assuming that the implicit dynamics model is accurate).
infer these states anyway using the
Definition (Maximum Inference Disadvantage (c.)). Consider the maximum divergence that can be accumulated by the (c, mmmolel from not knowing the past states 8t-h A-d---t------- --s--tt------ and let that be es. More formally, we say that, for fixed from Proposition any states S-, any 2t- kit and
any 8--h-d-ctth---11 * hi-h-d-ct-h-c--
C( (a
4 St h---- 8t --th-ett--- -o at at-h:t-))) σ²) S Eb. (11)
Here, ((o+d,)) arg min (e
d,1) (I((T((-aa ((-)))) T"" [C] is the optimal (c d, h)- -policy.
As a warmup, we see that the intuitive relationship between ay and (and the same for og and (6) holds:
Proposition 8. as S e, and ab S Eb.
Proof. We prove the first inequality; the second can be proven in the same manner. We use As- sumption 2 write ((c.ktd) Bec-n-atia--a (T(+d,))) sr-hhd-otthh-d at-hh he-h-dit-1 P is the environment's 's transition dynamics. Let
where
P Pownect inference P(8g-h-d42a-- St -
and
P. Pincorrett inference
P($--b-441 -
26

---

Published as a conference paper at ICLR 2025
Then,
a
min ((a+t))
E
x)
min E [(T((td))) π
((tdAA)
min m(oddb))
[( inference (c))d,))) + Pincoreect inferceeeeeeeee (c+ (
E ((T(ctd))) क |cc]
min ((c4d,))
min M(+dd)
{E (faaercc on incorrect inference),
)
+ E Paurret inference ( ( on correct inference). [
min E ((((T(4dd))) ||] ((odd,h)
(Convexity)
P Pincorraa inference (f( ( (conditioning on incorrect inference), x x)//c
E [(miotd,h) π ||] E [(o(ooa))) T c
(Bounding probabilities by 1 and Lemma 5
E [P Pincorrect Ef C7
$ Es
Here, ((o+d,)) arg (ei E [[(m(ctam) x") [l]]
We will provide tighter bound after the proof of our main theoretical results.
Definition (Forward and Backward Inference). For a fixed time step t and C' (as in 3.2) consider the time steps Ty {t h d + 1 t h} and Define
{t h d - c:t h
(t Piesa (Sv ge S -r- Av -1
for any t' Ty with gt', 11 - being the ground truth states and action in the deterministic test environment. As such, (t') = in a deterministic environment. a stochastic environment, Py(t') < 1 for all t' and the stochasticity increases, these values decrease and approach 0. Let
(t()') = Perain (ge -e av - (t(
where 01 ap-1 are still the ground truth in deterministic test environment. Define Py(t') = Pt(t') + (t)) Similarly, define
Po((') = (Se = ge [S++1 = 9r-
for any t' To where -1 are the ground truth states in the deterministic test environment. Then let
(t()') Perai (gr (g (tt)
for any t' E Tb- Define Pa(t") Pt(t') + 8t),).
Intuitively, (t') and (t') characterize the error that the implicit dynamics model has in capturing the test environment's dynamics.
E.3 CONSISTENCY-REACTIIIT INEQUALITIES
Now we prove the Consistency-Reacciiity Inequalities.
Proposition 1 Deasiistnny-Reacttvii Inequalities) Let be a non-linear and nog-negative convex function measuring the prediction error with respect demonstrations. Let S {s be the states both the (c, h) and the (c, h + d) policies observe and let S == {st-ku) St. Let C (a--ha11) S+. G = {as, = U {at (a--h-da-h- US- For notational ease, let ty
27

---

Published as a conference paper at ICLR 2025
h h) and let 7, {t - h d c - h c 1}. Then, bound the expected loss of the (c, + d)-policy and the (c, h) pollic as:
as a6-1 II (c(E)((())) + do(t))) &
min Be [C( T"] maid
min Eg [C(TTs, x []C
PEE
and
min Eg [[(T=+d xt" ][]] min Eg [L(T,, a s II + -t.- VET
Proof. We first prove upper bound, For ease of notation, we will write =b to mean Zazs- Additionally, for greater clarity, we will explicitly include the context length of each model, so Ta4d. We start by writing, using Assumption 2,
((c,k) Ta and ((c.+dd)
(
d: 1
To hh-a)
1-1 a:) Pt---d-et R---dd
t-h-d
Using this, we expand the left hand side of our inequality:
min Ee [C(M(c.n44) C] *(o,h+d)
Ee [[Bs, d11a [Totem]/] . Ic]]]
min "(c+d.))
min Ba [[ Coinaalliihaaaa 1 1o *(otd,))
Σ
Preain d-h hh de - metaaylaal ( Ic]]
Addh-thte au melll
min 7(eld.))
C( II Py (7(0) aft Perr
Σ
Prain(6f-h-atil de - aizh=ld motasslall.... a [|]]
Rt- -- 1at all
where we computed the expectation Eg. ((c+d,)) c7 by grouping into two terms : one where every &t &r--ddt:thh ft-h -t-h-d+1=hh h and one where there is at least one term á; that is not gi. This grouping was done using the definition of noise in our environment. We introduce the following notation here
P P#g. =
Σ
( (a d+1. (8t-
of a//
8 h
90
Similarly,
P#ge = Σ
1-h-1 Pues P d-
ft-h- de not all ger
With this notation, continue expansion:
28

---

Published as a conference paper at ICLR 2025
min (te,hdd)
Ba ((()))a))))
min ((ctd,)
Be
L( II Py am annd NETT
Σ 4--A-dtii-h
Puriia
hi2d) mesdarlasl... , [Ic]
ne!
< min *(c4d,))
Ba
Ll II ( -ith--ss Pooy motaarlaal ,81 - d. ]ic]
eery
min *(+dd.))
Ba
II Py (tTcaaaa) (o ot an
Bo [ooo (l a
where got the inequality using the fact that L is convex function and, thus, convex in each argument. Next, we take the expectation over 8t-h-d+1:-A by grouping the terms into two: one where every st-h-d-1:thh = 9t-h-d+1c-h and one where there at least one term 8 & 9s. Then, with some suppression of notation in the expression of (c+ddb) and G' = G {ar, - d+ #
min EG [L(TicA+t)) *)]]] ((,h+d)
min Ea, ((4dd.))
II (f) By [mmcctass st-h si atr)), R²) ..., s hh -i-h-dt1;
1-h
METy
Peay
II Py [L 02-h-d+1): .t 8 # 9 -
(t') (t') Bey S1 * gi hi 8 = h-rh-
CETT
+ P4ay Pooy Ba [lTTcttanss.. 8t # ot - -i # f ai
Now, we group all the terms into two one representing where the learner's simulation matches the reality and one where it does not. Continuing from where we left off, first, define s=x to be the probability that the inferred states at timesteps in ry are the states that were visited in reality. Then,
Ec [C(F #"][[ F(c.bid)
min
Eag (c+d,h)
II ()) (E)) (i = sich-aen = hh -in = of shaa Perr
+ Pooy
P/_.B [C st = oi # aich tt & si
II (1r [ 81 am # d- am = and) al)):=")[...ff . ann of s1l) Meey
PPaao Py(t') (( e'ery
ami a andd"")))) a am sh
+ Psay Peram ( # of h h ath of
B (ettea th # h h
4t1: # of th
|C, [110] o "/. than # sh
29

---

Published as a conference paper at ICLR 2025
For the match terms, the fact that Ier Py(t') $ 1 and
P/ Pirain (s = =
- h+1 st-h-d, - h-1 S 1. For the mismatch terms, the
h-
definition of and Lemma 5
Then, we continue:
s min EG [[TT(-d.M) [|cc ((otd,))
(Simulation matches reality)
II (t) (1-P)(t))))
Py(t))ess} II (1- (() (P)())+ Poo c1-
PEET
METy
(Simulation does match reality)
Recall that we write (t') as simply Pt(t') We simplify the mismatch terms further:
II Pr(t) Pyy(t)) II P(()) (Pr(t) + Prov.ur CETT
< II Pi(() (1-1)(0))))
II (1 P)(()) -1000 e'eT
+
e'er
II P(0))11(1)
= =
II P(()))(((())))))))))) e'ETy
=
Next, simplify the match terms by using the definition of oz:
min
Eg
[[((etaM): [|]
min Eg [A(T(e)); [ as. *te.k)
Substituting these two terms back in, we conclude.
Now, we prove the lower bound. We proceed in a manner similar to the proof of the upper bound. For ease of notation,
will write zb to mean Fasb. Additionally, for greater clarity, we will explicitly include the context length of each model, so (cch) = Th and ((,,hdd) = Wh+d. We start by writing, using Assumption 1,
min Ee [[(T((,M))'' 7') |c] ((o,k)
min Eg C(Prranlggith Co + Σ T(cdd.))
Puran (at--sst--aa (s--h- - x
min
Rg
II Po(t') (((t)
Σ
Prrain (8) - 1 (( +
META
not all
We introduce the following notation here
Σ ait-
Persin ( 1at-h-c-1
(1- (t-hhho))
not all grr
Similarly,
P4ge
Σ
Prest (s) |s sp-h-a).
9t-h-d-aa
St
gr
30

---

Published as a conference paper at ICLR 2025
With this notation, we continue expansion:
min Eg (T(T),,)) []]]]
((ck))
min EG
( C +
((tdd.))
L'E
Eg
Po C(( F(e-daj((a)--e 8 - |
st-h-e-1
where we got the inequality using the fact that L is convex function. Next, we take the expectation over st-h-d-ctthh--1 by grouping the terms into two: one where every st-h-d-ct-h-e- ft-h-d-ct-hh- and one where there is at least one term 8; + gi. Then, again suppressing some terms inside the expression (()+d.));
min Be [A(T(.,):='')IC] ((och)
min Ea. T(old.)) (d.))
II Po(E))((t) E (1 81 = Ii-h-d-o h(h--d)) TT)
1-h-e-1 1-h-e-1
t-h-c-1
s-h-a-1 = Pi-h-d--
P*ay II F (o = Di-h-d-- -
t-h-c-1
# Di hi-h-d-
MEn
II Poot') LlTcctassl 8h-h-o-1 + Pe%
9i-h-d-a s(-n-d-)); xt)
1-h-e-1
h
9 Pi-h-der
dt-h-e-1 * ginh-da) x²) S gich C,,a |c] -Paa P#ay E [lTietans 8
s-h-c-1
Now, we group the terms into two one representing where the learner's simulation matches the reality and one where does not. Continuing from where we left off and defining po
á== =
h--h---1 hh-h- scb
min EG [E(F [C] *(o.))
min Ze, ((td.A)
II Polt') PO(E [C st 1-h-e-1 at)) LeE
1-h-e-1
Paay P
-
--h-c-i # ---h-d-o
+ II Pa(l))aap E [[lm der
1-h-o-1
(hh---1
hi-h--eeh
Ch-d-e
Peny
II A.()) (l af -t-k--11 = - - ) fethh
(-h---1 # 10221111,,0 1I0, [c]
Proye (8 di-a-deo Si-h-d-m
d-h-e-1
(t-h-c)
B [( aytaelin st= of dichess) st")
d-b-o-1 - a1 [c]
For the match terms, we use the fact that Iren Po(t') < and
Prrmin (8, h- - 1. For the mismatch terms, the definition of e
8t-h-d-c
31

---

Published as a conference paper at ICLR 2025
and Lemma 5 Then, we continue:
Eg (((T(cdd)): o |c]
min
(Simulation matches reality)
((otd,)
II Pa(t)(1 P6( II (1-))((t))PPC))) P4ay.ey ccb
MET
PET
(Simulation does not match reality)
We simplify the mismatch terms further:
II Pa(t)( Po(t)) + II (1 - Po(t))(P((tt) + Poov
N'E
S II (a())(1-PPA))))) Pc(())s II (1- P(LL) (1 Po(())))
den
E'n
II P((()
= 6
II ((0)(B()++ + PEN
tb
Next, we simplify the match terms by using the definition of a which follows from Proposition in manner similar the definition of ag:
min Ee [(M(ctd.M) T T(o+d,n)
min Bo [A(T(c,Mta) n []C az. F(c.h+d)
(Proposition
We substitute these terms back in to get the desired bound.
Now, we prove Corollary 2 as a direct consequence of the Consistency-Reaction Incqualities. Re- call that in near-deterministic environment, Py is close as the transitions purely determined.
Corollary 2 (Consistency). Suppose, the train and test environments are the same and it is deter- ministic. Suppose a, is influenced by at least one state at time steps т, and 6y()') Rs for all t' E Ty. If the diversity in past strategies is not O, and ef is finite, then
min EG [A(TT++a n [C] < min Eg [L(TA, π C]]
Proof. This follows from the upper bound of Proposition 1 Since the test environment is determin- istic, we take (t') z 1 and, by hypothesis, (t') ss 0 for all E Ty, menaing our learned policy's implicit dynamics model is accurate. Then, we get
min Eo [C(Tn4d; x") [CC-Cmm min Ba [C(TA, m" | C < -an 8h42
II P(())((((())))(())) -06 FETT
Ki.
since ey is finite. All that remains to show is that as is positive. Note that is temporally dependent on least one state in (St-n-e-dtthhee Furthermore, since diversity in past strategies is not O, the states in timesteps cannot be predicted with probability 1. Then, by Proposition 6 we have
that og 0. Therefore,
min Eg [L(Thtd; x²) CJ min Eg [L(T,, 1²") 11) -as <. Th+d
Xh.
32

---

Published as a conference paper at ICLR 2025
Next, prove Corollary 3
Corollary 3 Suppose Py is small or both (t(t') and joy [t') are large for all t' E Ty. If temporal dependency decreases over time such that e, is small and a is influenced by least one state time steps in ry, then
min BG (a(Th+a 7 ["] > min Ba (TT,,,)))(C)
Proof. Notice that since e Re O, the lower bound in Proposition is a Next, we show that that o is positive. If (t')) is small, then we know by Proposition 7 that a is positive. On the other hand, if Py((') ~ but ((') is large, then since probabilities are bounded, we require ôy((") 2 -1. In other words, the inference of the correct unobserved state is very unlikely causing af to be positive again. Therefore,
min Ba [C(TT x "'][C] < min Ee [A(Tntd x x]][C]
To samity check the Consistency-Reann Reactivvty Inequalities, we prove the next proposition. This shows that the right-hand side of the inequalities is greater than or equal to the left-hand side.
Proposition 9. In all environments,
-6
II (P)))))()+))))))))) -ab MEn,
and
af S e - II +100(1((0++
E'Ery
Proof. We prove the first inequality; the proof of the second proceeds similarly. Using the definition of og and suppressing G and C (as Proposition T for clarity, we have:
a6
min ([[(T + min (E x"]]
Whtd
min E[[(matadi)) **)] ah+d
+ min E [ P
inference)
Ri+d
Pincorret Ty - inference). nT')
< - <-min E[[((Th+d.
+ min E Peorrett inferenes (Th inference), π
sn+d
P Pincorrett inference L (T (Tt inn.ooooeettttnneennneneeeeeee inference), o x²)
< min [[[((T+dd: Whad
+ min [ [ (coorcctt inferencee, T Antd
E Proconeet inforence C ((TTe inference), TT)]
< min [[[(mn+dd "
+ min (E[[((T inference,r m))]
Fhid
+6
II (3('))((3(())66(())))
M'm.
33

---

Published as a conference paper at ICLR 2025
cb II +00))))()++)0))))))))))))))))))) MEE
E.4 CLOSED-LOOP VERSUS OPEN-LOOP HIGHLY STOCHASTIC AND ENVIRONMENTS
The Drasistency-Reactivee Inequalities allow us to make an even stronger statement when we com- pare strietly closed-loop policies with open-loop ones. Consider the same set-up as before with h 0. Thus, K(c.0) represents closed-loop policy whereas ((c,d) represents open-loop one. We can compare these policies' divergences with the expert across the entire trajectory in the limiting cases of the environment stochasticity.
Corollary 10. Suppose (t))') is small or both Py(t') and [8y(t') are large for all t' E Ty. If temporal dependency decreases over time such that en is small and at is influenced by least one state at time sleps in Tf, then the expected divergence between the closed-loop policy over the full trujectory is lower than that between the open-loop policy and the expert.
Suppose, the train and test environments are the same and it is deterministic. Suppose of is influ- enced by at least one state at time steps тr and let ,t''') =e 0 for all t' E Ty. If the diversity in past strategies is not O, and E, is finite, then the divergence between the closed-loop policy over the full trajectory is greater than that between the open-loop policy and the expert.
Proof. At any arbitrary time step I, the chunks of the two policies can be aligned in one of two ways: Case I: (cc.0) is executing as the first action in its action chunk and ((c.d) is also executing as the first action in action chunk.
Case 2: (c,00) is executing a₁ the first action in its action chunk and ((c,d) is executing as the k-th action, where & (1,14 + d] in its action chunk.
Using the Prasissency-Reacttvve ever, in case 2, using Corollary 3, we know that the closed-loop policy will outperform the open-loop one in the first setting of the statement and open-loop will outperform in the second. From this we can conclude the divergence across the full trajectory.
Inequalities, in Case 1, both policies have equal divergence. How-
34


Reviews:
[{"review_id": "VftGx3OHCo", "reviewer": "Reviewer_ayXv", "summary": "This paper examines the impact of action chunking in robot learning from demonstrations, where actions are executed in sequences without intermediate replanning. While action chunking helps robots learn temporal dependencies, it can hinder adaptability in dynamic environments. To mitigate this tradeoff, the authors propose Bidirectional Decoding (BID), a test-time inference method that enhances reactivity while preserving the benefits of chunking. BID samples multiple actions per step, selecting the best based on backward coherence (consistency with prior actions) and forward contrast (alignment with stronger policy predictions and deviation from weaker ones). This approach improves temporal consistency and adaptability, yielding better results across multiple simulation benchmarks and real-world tasks. On the down-side, the approach requires significantly more compute as multiple model outputs are sampled. Authors deal with that by parallelizing the computation.", "questions": "- Can you clarify the definition of horizon, and relationship to MPC.\n- Can you clarify experiments  in section 3.2.\n- Can you explain in more detail dirrente policies for analysis in 3.2. E.g. can you show in Figure 3 with aligned current time. For time $t_k$ you predict the sequence with some length and take context with some length.\n- In Corollary 2 you claim that $\\pi_{h+d}$ is more disadvantageous but you show in eq (2) that the expected loss is smaller. This seems to be contradictory.\n- Why is using negative set important?\n- I would suggest using bar plot for fig 6 right.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Proposed method seems reasonable and well-thought of, with clear description and theoretical backing.\n- Paper is well written, clear and well-structured. The motivation and the idea of the paper is clear. The approach is clearly stated and all relevant background is introduced well. The method explanation is accompanied with appropriate visualisation figures that help understanding.\n- The proposed method achieves good performance on target experiments.\n- The approach is tested in several simulation and real experiments.", "weaknesses": "- The method requires significantly more compute than the base methods.\n- The definition of a horizon is sometimes unclear. In Moving Horizon fields (e.g. Model Predictive Control - MPC) horizon is what is presented in the paper as $l$. Perhaps this should be also labeled on the Figure 3. The horizon $h$ in this paper is used to describe what is replanning time in MPC.\n- Generally, it would be useful to back this more to the MPC field. As focus of the paper is not on the inference itself but more on using any planning/prediction model in the closed-loop. Borelli, Bemporad, Morari, Predictive control book.\n- The analysis on the replanning times and prediction horizon seems to be a bit limited and not fully consistent among different ablation studies.\n- A single experimental problem is used for analysing the main point of the paper (demonstration with iddle action).  Figure 5 is not clear, why the results for 3 are better than for 5.\n- Backward coherence as introduced here is a standard for improving temporal consistency when replanning in many MPC approaches.\n- Description of policies in 3.2. is a bit confusing. It is not exactly clear are they aligned in time and why the importance is not same for the context for the first frame before the predicitons.\n- In line 316 and 318, it is stated “equally optimal” and “the most optimal” that is in contradiction with the definition of the optimality.\n- Using negative sets is not motivated well. This could be also investigated with ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GUoAo87Yb4", "reviewer": "Reviewer_PZsG", "summary": "The paper introduces Bidirectional Decoding (BID), a test-time inference algorithm designed to improve action chunking in robotic policies by balancing long-term consistency with reactivity to unexpected changes in the environment. The authors address the tradeoff observed in action chunking, where increased chunk sizes help capture temporal dependencies but reduce the robot’s responsiveness to dynamic elements. BID operates by sampling multiple actions at each time step and selecting the optimal one based on two criteria: backward coherence (alignment with prior actions) and forward contrast (similarity to a stronger, more optimal policy and difference from a weaker one). The approach enables more flexible, closed-loop decision-making that adapts to real-time conditions. Experimental evaluations across multiple simulation benchmarks and real-world tasks show that BID consistently enhances the performance of robotic policies, demonstrating its effectiveness as a plug-and-play solution to improve the robustness and accuracy of action-chunked behavior cloning models in robotics.", "questions": "Why is it necessary to set the 𝑐 parameter, which represents the window size of incorporated historical states? When employing an LSTM-based policy, the incorporation of historical states becomes more flexible. This flexibility is akin to the context window size used in word2vec for planning. The advantages of using LSTMs over a fixed context window 𝑐 have been discussed in the following paper: [1].\n\nReference:\n\n[1] Zhuo, Hankz Hankui, et al. \"Discovering underlying plans based on shallow models.\" ACM Transactions on Intelligent Systems and Technology (TIST) 11.2 (2020): 1-30.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The concept of \"Action Chunking\" is relatively new, and further research could reveal additional advantages of this technique by deepening our understanding of its potential benefits. \n\n2. Experimental results demonstrate the promise of the proposed method, as it outperforms several state-of-the-art (SOTA) diffusion-policy-based behavior cloning techniques, suggesting it could be a viable approach to enhancing robotic policy learning.", "weaknesses": "1. From my understanding, Action Chunking primarily focuses on capturing implicit hierarchical relationships in action sequences without needing to acquire explicit representation of reusable skills or options (where \"options\" refers to temporally extended actions in hierarchical reinforcement or imitation learning). The field of option/skill discovery is active an active research direction, with an aim to explore methods to learn options/skills from either environmental interactions or recorded demonstrations. \n\n2. Because Action Chunking requires pre-defined window sizes for past state observations and future actions, option/skill discovery methods (e.g. [1][2]) are potentially more adaptive and flexible than Action Chunking. It would be valuable for the paper to discuss and compare Action Chunking theoretically and experimentally with option/skill discovery approaches. Notably, in hierarchical RL, a common approach involves an initial option discovery phase, followed by a reinforcement learning phase utilizing these options. In this work, imitation learning could replace the second phase. Two recent, notable works on option/skill discovery [1][2] could serve as relevant comparisons.\n\n3. Another limitation is that the reported performance improvements (e.g., Table 1) are modest. For example, the difference between the results of EMA (Zhao et al., 2023) in the 3rd row and the proposed method in the 4th row is relatively small. This pattern holds in Tables 2 and 3 as well: when compared only against the weakest baseline (Vanilla), the performance gains remain minor. \n\n4. Furthermore, Tables 2 and 3 omit results for other leading baselines, such as Warmstart (Janner et al., 2022) and EMA (Zhao et al., 2023), which would strengthen the evaluation of the proposed method.\n\nReferences:\n\n[1] Bagaria, A., & Konidaris, G. (2019, September). Option discovery using deep skill chaining. In International Conference on Learning Representations.\n\n[2] Zhao, Tianxiang, et al. \"Skill disentanglement for imitation learning from suboptimal demonstrations.\" Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3eCuaQSNv", "reviewer": "Reviewer_FR7d", "summary": "This paper studies the usage of action chunking when learning policies through behavior cloning of human demonstration data on robots. Action chunking is the prediction and execution of a sequence of actions over a horizon $h$ without closed loop control. Long sequences of predicted actions lead to increased temporal consistency in the policy but lack the ability to react to short-term noise in a given system; on the other hand, short sequences of action chunks allow for this reactivity but at the expense of temporal stability. This is a well-known phenomena that affects the design decisions of machine learning and robotics practitioners. This paper provides theoretical analysis of the problem and introduce bidirectional decoding, an inference time policy change that naturally balances the need for long-term consistency with short term stability. The authors test their method on various toy and scaled up domains and show their method superior.", "questions": "In addition to addressing the weaknesses, I would like the authors to answer the following questions:\n1. In equation (6), why use MSE style loss as opposed to a KL divergence on the action distributions? Is MSE more robust in this context?\n2. Can the authors describe more technically why this methodology has a focus on human demonstrations? The argument of latent intentions is not clear to me.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper is written clearly and well-motivated. To my knowledge, this is the first formalism of the optimality gaps incurred when trading off context length and prediction horizon. They demonstrate some exciting results that attempt to dissolve the issue of deciding an action-horizon on multiple domains like FrankaKitchen and PushT. They also describe some real-world experiments on robots but they are not the main focus of this paper.", "weaknesses": "There are several weak points in the paper that detract from the paper's quality.\n\n1. **Human latents**: There is some motivation surrounding the latent intentions of humans that are not directly observed. However, talk of latent intentions is not mentioned in any technical sense so it is not clear how different latents $z_t$ affect the methodology and analysis presented in the paper\n2. Lack of model-based/model-free discussion: In section 3.2, the authors outline their approach to theoretical analysis. It is not clear how model-free or model-based policies fit into this framework or the analysis. In my understanding of the analysis, the authors expect the policies to learn implicit dynamics models and do not ever explicitly learn them.\n3. Ambiguous definitions in 3.2: I understand that the appendix provides better definitions of the quantities $\\alpha$ and $\\epsilon$, but given that they are crucial to understanding the analysis, the definitions in the main paper are not sufficient for understanding. \n4. Difference between Corollary 2 and Corollary 3: The inequalities in (3) and (4) are exact opposites of each other, except for their qualification in the corollary text. It is not clear mathematically why these two inequalities are opposite.\n5. Minimizing loss in (5): Algorithm 1 suggests that the action selection is not used during training time and only at inference time. Would not the policy benefit from doing BC training with this action selection method?\n6. Weaker policy used for contrastive $\\mathcal{L}_F$: using an older, weaker policy for a contrastive loss may lead to replacing some design decisions with equally harder decisions that must be made by the practitioner. \n7. No error bars: Figure 6 display main results of the superiority of the authors' method but they do not include any statistical significance on the results they report. This is a major weakness in my opinion and indicates that the experimentation was not sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uOOi4ZvJiu", "reviewer": "Reviewer_wSU2", "summary": "The study presents a theoretical exploration of action chunking. It offers valuable insights, demonstrating the feature of action chunking varies across different scenarios, particularly depending on the degree of determinism or stochasticity in the environment. The findings suggest that in deterministic environments, action chunking improves long-term consistency, whereas in highly uncertain environments, it struggles to adapt effectively to changes, as reactivity becomes more critical.\n\nThe paper introduces BID—an inference-time, model-agnostic, computationally efficient, and easy-to-use plug-and-play algorithm designed to enhance action chunking, emphasizing better closed-loop control. BID selects action chunks from samples based on two key criteria: backward coherence and forward contrast.\n\nThe experiments begin with a simple yet effective discrete one-dimensional toy policy, validating the theoretical insights on the relationship between action chunking and varying degrees of environmental determinism or stochasticity. The effectiveness of BID is further evaluated using Diffusion Policy as the base policy and compared against three other baseline action chunking strategies (vanilla, warm start, and EMA) across three simulators: Push T (single task), RoboMimic (four tasks: Lift, Can, Square, and Transport), and Kitchen.\n\nAdditional experiments explore BID’s scalability by varying batch sizes and its compatibility with different sampling strategies. The results also demonstrate BID’s generality and efficacy in real-world applications, reinforcing its practical utility.", "questions": "I have a detailed question regarding the design of backward coherence \n$\\mathcal{L}_B$ and forward contrast $\\mathcal{L}_F$. Specifically, you use Euclidean distance to evaluate the consistency (similarity) of actions at individual time steps. However, there are many alternative metrics that might be more suitable depending on the nature of the values. For example, in typical 7-DoF robotic arm actions, three values represent orientation, and Euclidean distance may not be the most appropriate metric. In some cases, L1 distance or angle-based metrics (e.g., cosine similarity) might be more suitable or even more direct alternatives. Could you provide additional experiments to explore the impact of different metrics on your two criteria and their effect on the experimental outcomes?\n\nI am also curious about the design of forward contrast. I did not clearly see its connection to your theoretical analysis, particularly in terms of its relationship with reactivity. Could you provide more insights into why you designed forward contrast this way and how it specifically enhances reactivity? Alternatively, if there are other possible designs that improve reactivity, I suggest revisiting and reworking Sec.4.2 to clarify these aspects more thoroughly.\n\nAdditionally, I am wondering whether the weighted sampling strategy used in ACT (introduced with [ALOHA](https://arxiv.org/abs/2304.13705)) could serve as a reasonable baseline for comparison. It would be helpful to see comparison results between BID and action sampling strategy used in ACT. Could you conduct further experiments to evaluate this comparison and enhance the analysis of BID’s effectiveness?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The work employs well-founded models and thoughtful improvements to make a commendable attempt at uncovering the essential latent factors of action chunking, such as the environment’s degree of determinism. It offers insightful explanations of the theoretical analysis, clearly highlighting the benefits (consistency) and drawbacks (reduced reactivity) of action chunking.\n\nThe experiments are comprehensive, demonstrating BID’s effectiveness across a variety of scenarios.", "weaknesses": "I found some of the figures difficult to interpret. For example, in Figure 1, I assume the goal is to illustrate how BID combines the benefits of (a) and (b). However, the concepts introduced in the paper, such as backward coherence and forward contrast, are not intuitively reflected in the figure. Similarly, the meaning of Figure 3 is unclear.\n\nA more significant concern is the lack of a clear connection between the BID criteria in Section 4.2 and the intended trade-off between consistency and reactivity. Specifically, the discussion on reactivity is vague, and it is not evident how the BID design enhances reactivity. I believe additional explanations are necessary to clarify how the proposed BID method addresses reactivity, providing more detailed reasoning behind the design choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)