Real-Time Execution of Action Chunking Flow Policies
Authors: Kevin Black
Manuel Y Galliker
Sergey Levine
Venue: NeurIPS 2025 poster ()
Keywords: diffusion policy
flow matching
realtime
real-time
action chunking
VLA
vision-language-action
robotics
control
manipulation
Primary area: reinforcement_learning

Abstract: Modern AI systems, especially those interacting with the physical world, increasingly require real-time performance. However, the high latency of state-of-the-art generalist models, including recent vision-language-action models (VLAs), poses a significant challenge. While action chunking has enabled temporal consistency in high-frequency control tasks, it does not fully address the latency problem, leading to pauses or out-of-distribution jerky movements at chunk boundaries. This paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. Our method, real-time chunking (RTC), is applicable to any diffusion- or flow-based VLA out of the box with no retraining. It generates the next action chunk while executing the current one, "freezing" actions guaranteed to execute and "inpainting" the rest. To test RTC, we introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate that RTC is fast, performant, and uniquely robust to inference delay, significantly improving task throughput and enabling success in precise tasks --- such as lighting a match --- even in the presence of extreme latency.

Real-Time Execution of Action Chunking Flow Policies
Kevin Black! 1.2
Manuel Y. Galliker'
Sergey Levine
Physical Intelligence PUC Berkeley
drevin,manued htt.://www...soooooooooooooooaon company
Abstract
Modern AI systems, especially those interacting with the physical world, increas- ingly require real-time performance. However, the high latency state-of-the-art generalist models, including recent iision-languge--ce a significant challenge. While action chunking has enabled temporal consistency in high-frequency control tasks, it does not fully address the latency problem, leading to pauses or out-of-distribution jerky movements at chunk boundaries. This paper presents novel inference-time algorithm that enables smooth asyn- chronous execution action chunking policies. Our method, real-time chunking (RTC), is applicable any diffusion- or flow-based VLA out of the box with no re-training. It generates the next action chunk while executing the current one, "freezing" actions guaranteed execute and "inpainting" the To test RTC, we introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate challenging real-world bimanual manipulation tasks. Results demonstrate that RTC is fast, performant, and uniquely robust to inference delay, significantly improving task throughput and enabling high success rates precise tasks such as lighting match even in the presence of significant latency. See https://ppt http:::/pp..eebsstt/reeeeee website/research reaceehrrreraaee. time_ chunking for videos.
models (VLAs), poses
Position
Velocity
Acceleration
deg
deg/ss is
deg/ 's*
Real-time chunking (ours)
Syuchroneus
Time (s)
Temporal ensembling (Zhao et, al.)
Figure 1: Top: Real-time chunking (RTC) enables the robot to perform highly dexterous and dynamic tasks, such as lighting a match even in the presence of inference delays in excess of 300 milliseconds, corresponding to more than 30% of the model's prediction horizon. Bottom: RTC performs the same robot motion 20% faster than synchronous inference [5 30 24 31 59). and smoother than all competing methods, including temporal ensembling (68). The shown positions, velocities, and accelerations correspond to the shoulder joint of arm, are taken from the 10 seconds a real autonomous match-lighting rollout.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
As AI systems have become more capable, they have also interacted more and more directly with their environment. Whether they're re executing terminal commands [45], playing Pokémon on livestream 120. or browsing the web on your behalf [65], recent advances driven primarily by large-scale scale deep learning haaeinngg-aveaaee enabled these systems to increasingly control, rather than merely process, the vast heterogeneity of the outside world. Embodied agents, where machine learning models directly control real, physical constructs, are perhaps the quintessential example. The same advances fueling agentic language and vision models are also making great strides in physical intelligence on platforms ranging from humanoid robots 4 to autonomous cars [60].
Cyber-physical systems, unlike chatbots and image generators, always operate in real time. While a robot is "thinking", the world around it evolves according to physical laws. Thus, delays between inputs and outputs have tangible impact on performance. For a language model, the difference between fast and slow generation is a satisfied or annoyed user; for robot action model, on the other hand, it could be the difference between robot handing you a hot coffee or spilling it in your lap.
Unfortunately, the effectiveness of modern large-scale machine learning comes with high latency as unavoidable side effect. Large language models (LLMs), vision-language models (VLMs), and vision- laniinn-anngaaeaacctoon models (VI.As)-the the last referring to a class of models designed for
visuomotor control- billions of parameters 8 30 5 58 These models are not only slow to run, but also require heavy-duty hardware that is difficult to attach to edge devices such as mobile robots, adding even more overhead for remote inference. Edge hardware will improve over time, but as robot datasets grow in size, so will the best VLAs 28]
Thus, applying large models real-time control problems effectively will require some form of asynchronicity: that is, a model must think about its future actions while executing a previous one. Action chunking [68] 33 [I], where model outputs and executes sequence of multiple actions for each inference call, presents a partial solution. A Although action chunking has already achieved many state-of-the-arr results in dexterous manipulation ( 4 (8). it still suffers from the latency problem.
Chunking sacrifices the reactivity of system to external stimuli and also introduces discontinuities in the transition points between chunks, as adjacent chunks may jump between different modes (or "strategies") from the learned action distribution. Such anomalies are especially harmful to learning-bast based systems, as they produce a distribution shift in dynamics that the model is likely not equipped to handle. Naive smoothing strategies, such as averaging multiple predictions together (68] are not guaranteed to produce valid actions and may only make matters worse (e.g., see Figure 2)
A good real-time system must produce a consistent and continuous control signal, incorporating the latest observations without perturbing the environment's natural dynamics the model's ability to produce correct actions. In this work, we present real-time chunking (RTC), which poses asynchronous action chunking as an inpainting problem. Our algorithm generates the next action chunk while executing the previous one, freezing the actions that are guaranteed to be executed (due inference delay) and "inpainting" the rest. It applicable any diffusion- [22] or flow-based (36 VLA, and operates purely at inference time, requiring no changes to existing training recipes.
Our contributions are follows. First, we present novel system for asynchronous, real-time inference of action chunking diffusion- or flow-based policies for continuous control. Since standard simulation benchmarks are quasi-static- and have mostly been saturated with pseudo open-loop inference strategies 11] [iw
-we devise a new benchmark based on the Kinetix simulator [43] consisting of 12 highly dynamic manipulation and locomotion tasks. In the real world, evaluate RTC on 6 challenging bimanual manipulation tasks using the To.5 VLA (24) as the base policy. Across both simulation and the real world, we demonstrate that RTC is fast and performant; to inference latency, even in highly precise tasks such as lighting a match (Figure I]. and it achieves greatly improved task throughput on all real tasks.
uniquely robust
2 Preliminaries and Motivation
We begin with an action chunking policy denoted by TT ((A,lo;), where A, = [ar, a,+1 az- H a chunk of future actions, O, is an observation, and t indicates a controller timestep. We call H the prediction horizon. When action chunking policies are rolled out, only the first < H actions from each chunk are executed. We call s the execution horizon; often it is shorter than the

---

prediction horizon, but still much greater than I (e.g.. $ H/2 ensures temporal consistency at the expense of reactivity. long execution horizon reduces a policy's responsiveness to new information, while a short one increases the likelihood of mode-jumping, jerky behavior resulting from discontinuities between chunks.
I 5
24
Chunked execution
In this paper, we consider policies trained with conditional flow matching [36], though our method can also be used with diffusion policies by converting them flow policies at inference time [8) To generate an action chunk from a flow policy, random noise A is first sampled from a standard Gaussian, and then the flow's velocity field, Vz (a learned neural network) is integrated from to using the update rule
T+ A,
A
V= o
n
where 7 E [0, 1) denotes a flow matching timestep, and n determines the number denoising steps. Now, let At be sampling period of the controller, i... the duration of controller timestep. and let ó be the time it takes for the policy to generate an action chunk. We say that a system real-time if it is guaranteed to produce a response (in our case: a₂) to an event (receiving o,) within a fixed time constraint (At). If 8 < Åt, then meeting the real-time constraint trivial since an entire chunk can be generated between two controller timesteps. However, this is near impossible to achieve with modern VLAs. For example, with an RTX 4090 GPU, the billion parameter To
Inference starts
Ao
naive asyme
temporal ensemble
Deradd
46
Inference finisbes
VLA spends 46ms the KV cache prefill alone, before any denoising steps [5]. and targets a 50Hz control frequency (Ar 20ms). Run in remote inference for mobile manipulation, To lists 13ms of network latency, in perfect conditions with a wired connection. In more realistic setting, the network overhead alone could easily exceed 20ms. Kim et al. [31], who optimize the 7B VLe model [30 specifically for inference speed, achieve no better than 321ms of latency on a server-grade A100 GPU.
An illustration of typical bifurcation be- tween consecutive chunks. Inference is started between timesteps 3 and 4. The original chunk that execut- ing, {ar} (black), had planned go above the obstacle while the newly generated chunk {ai} (red) goes be- low the obstacle. However, {at} is not available until d = 7 steps later. A naive asynchronous algorithm might jump from @1a to ai1. inducing very high, of-distribution acceleration. Temporal ensembling (68), i.c., interpolating between chunks, reduces the acceler- ation but produces poor actions.
Figure 2:
31 59, simply starts inference at the end of the execution horizon and waits while the policy generates the next chunk. When > At, this introduces visible pauses between chunks that not only slow down execution but also change the dynamics of the robot, introducing distribution shift between training and evaluation. To develop a real-time strategy, we must first introduce asynchronous inference, where inference is started early and happens concurrently with execution.
Naive synchronous inference, the default in many prior works 5 30 g
We define d := /8/ At] and call this quantity the inference delay, corresponding to number of controller timesteps between when o, is received and when A, is available Let agris denote the (t' - t) action of chunk A,, generated from observing o,, If Ao is currently executing, and we desire an execution horizon s, then an asynchronous algorithm must start inference at s d. So long as d < H s, then this strategy will satisfy the real-time constraint and guarantee that action is always available when is needed. However, since the policy cannot know what will happen between steps d and s while generating A.-d, the transition point between a,- and agjs-d may be arbitrarily discontinuous and distrr-iisttiuution. Similar to a too-short exccution horizon, this strategy leads jerky behavior that is worsened dramatically with higher delays; see Figure 2
For simplicity. we do not consider delays synchronization issues at the sub- timestep level; we assume that the environment lower-level controller provides o, at the same instant that a--1 is consumed.
3

---

guidence wright
last chunk houndary
0.5
0_
4.3
0.3
d.1
0p
61
0g
0g
0g
05
0g
01
0gg
01
010
011
412
013
horizon, steps inference started
frazen, delay,
region, octions changesi
horizon, empty, freshhy ge
inference starts
next inference starts
Figure 3: A diagram illustrating how action generation attends to the previous action chunk in real-time chunking. If inference starts after the execution of a-1 and the inference delay is d 4, then the newly generated chunk will not be available until after as consumed. Therefore, do:3 are "frozen" and attended with full guidance weight of 1. In intermediate region, 44:10, actions from previous chunk are available but may be updated, since inference will have finished before a4 is needed. This region is attended to with an exponentially decreasing guidance weight. Finally, the last = 5 actions are beyond the end of the previous chunk, and need be freshly generated. The exccution horizon, s, is hyperparameter constrained by d S S H
d.
Real-Time Chunking via Inpainting
The key challenge in real-time execution is to maintain continuity between chunks. By the time a new chunk is available, the previous one has already been executed partway, and therefore the new chunk must be "compatible" with the previous one. At the same time, the new chunk should still incorporate new observations, so that the policy does not lose the ability to react and make corrections.
Our key insight is to pose mil-time chunking as an inpainting problem. To make the new chunk "compatible" we must use the overlapping timesteps where have access the remaining actions of the previous chunk. The first d actions from the new chunk cannot be used, since those timesteps will have already passed by the time the new chunk becomes available. Thus, makes sense to "freeze" those actions to the values that we know will be executed; our goal is then to fill in the remainder of the new chunk way that consistent with this frozen prefix (see Figure 3), much like inpainting section of an image that has been removed. We describe this basic inpainting principle in Sec. In Sec. 3.2 we introduce a soft masking extension that is critical for full cross-chunk continuity; finally, we describe our full real-time chunking system in Sec. 3.3
3.1 Inference-Time Inpainting with Flow Matching
Inpainting is known strength of iterative denoising frameworks such diffusion and flow matching. We build on the training-free image inpainting algorithm from Pokle et al. [48), which is itself based on pseudoinverse guidance (IIGDM: (55)). The algorithm operates by adding a gradient-bart based guidance term to the learned velocity field v at each denoising step (Equation [ that encourages the final generation to match some target value, Y, which corrupted version of the desired result. In the case of image inpainting, the corruption operator is masking, Y is the masked image, and the desired result full image consistent with Y the non-masked areas. The IIGDM gradient correction, specialized to setting, is given by
[.7-7 VHIGDM(A, o2, T) = v(A, o,, + min B, 1-1
(x-A,) diag( (W)
aA
(2)
QAT
where A = A + - T),
(3)
(1 T1-
2 72 + (1
(4)
A is estimate of the final, fully denoised action chunk and W is the mask. We are abusing notation by treating Y, A,, and W as vectors of dimension HM where M is the dimension of each action. Thus, the guidance term is vector-Jacobian product and can be computed using

---

backpropagation. The guidance weight clipping, 6, is our addition; we found that without it, the algorithm became unstable with the small number of denoising steps commonly used in control problems (see A.2 for an ablation).
3.2 Soft Masking for Improved Cross-Chunk Continuity
In practice, naively inpainting using only the first d timesteps of the previous action chunk is often insufficient to ensure that the new chunk takes consistent strategy, particularly when d is small (e.g., see Figure 4. The IIGDM correction is not perfect, and a small d leads a weak guidance signal, which can allow for the new chunk to still switch strategies and cause discontinuities. Our solution, illustrated Figure is to give our policy more cross- chunk continuity by considering not just the first d overlapping actions, but all masking, setting W to real-valued weights rather than Is and Os. The first d actions get a weight I; the last & actions of the new chunk do not overlap with the previous chunk, they get weight of O; the actions in between get weights that exponentially decay from I to O, accounting for the fact that actions further in the future should be treated with more uncertainty, The resulting expression for W is given
Hard masking masking
Prev. chunk
overlapping actions.
do this via soft
d
41
Figure 4: A comparison of naive in- painting (hard masking) and our pro- posed soft masking method: note that hard masking does not match the frozen region very well and produces faster changes direction.
if d i i H -
--s-1 where c = H - - d+1' ie HO.
W; = Ci
H - (5)
Intuitively, W modulates the "attention" paid to each corresponding action from the previous chunk. See Appendix A.4 for a comparison between different decay schedules.
3.3 Real-Time Chunking
We present our full real-time chunking system in Algorithm (complemented by Figure 3 The controller interfaces with our algorithm via GETACTION, which is called every Åt to consume an action a and provide the next observation o,, The INFERENCELOOP runs in a background thread so that an action is always available. forecasts the next delay, by keeping buffer of past delays. The execution horizon, 8, can change from chunk to chunk; the user provides a minimum desired horizon, Smins and the actual horizon for a given chunk is max 8min) where d is the delay encountered when computing the next chunk. Finally, the algorithm describes the inpainting with soft masking procedure GUIDEDINFERENCE which explicitly defines a denoising function (Eq. 3 and computes Jector-Jacobian product, which can be done with reverse-mode aulodifferentiation
Algorithm Real-Time Chunking
Require:
flow policy æ with prediction horizon H. minimum execution horizon 8min. mutex M, condition variable C associated with M, initial chunk A initial delay estimate dmit delay buffer size b, number of denoising steps maximum guidance weight 3
1: procedure INTTAA.IESSHREESTAAT
Initialize mutex-protected shared variables
2: 0; Acur Ainis Ocur
null
3: function GETACTION(O
Called an interval of At by controller
4: with M acquired do
5: t = ++1
6: Our = Onext
7: notify C
8: return Acuelt - 1]
5

---

procedure INFERENCELOOP
Run inference in a looping background thread
10:
acquire M
Q = new Queue([dmil maxlen=b)
Holds a limited buffer of past inference delays
12:
loop
13:
wait on until 8min
14:
=
is the number of actions executed since last inference started Remove the s actions that have already been
15:
prev Acur(s (8, I,.....f. H,
executed
16:
o Ocur
17:
d max Q)
Estimate the next inference delay conservatively
18:
with M released do
19:
new GUIDEDINFFRENCEEN o, prev d,
20:
Acur new t= t=t-s
> Swap to the new chunk as soon as it is available > Reset so that it indexes into Anew b Record the observed delay
21:
22:
enqueue onto Q
23:
function GUIDEDINFERENCETT o, Aprev d, 8)
24:
compute W using Eq. 5 right-pad A to length H; initialize A ~ NO,
25:
for to 1 with step size n do
26:
A A' e
(1--))x(AA,,,
V
Define denoising function (Eq.
27:
(Arrv
f (A7))
diag(W)
b Weighted error term from Eq
85
28:
g
A
Compute vector-Jacobian product from Eq. 2 via autodiff
A'=A-
29:
A
A T
Vn (AT, min
(3, - g)
> Integration step (Eq. I
return A
4
Experiments
In our experiments, we aim to answer the following questions. First, how does RTC compare to existing methods in highly dynamic and stochastic environments, and under increasing inference delays? Second, how important is soft masking (Sec. to RTC? Third, how does RTC affect the performance and speed of real-worr world dexterous robots?
We first evaluate RTC using a benchmark of 12 highly dynamic and stochastic environments in the Kinetix [3] simulator. We use this benchmark to compare the performance of RTC to other methods under simulated inference delays, as well as investigate the effect of soft masking. Then, using the To.5 VLA 24 the base model, we evaluate the performance and speed of RTC on 6 challenging bimanual dexterous manipulation tasks, including 2 mobile manipulation tasks (see Figure ??).
4.1 Simulated Benchmark
Most simulated imitation learning benchmarks are quasi-static, and standard chunked execution with long enough execution horizon can achieve sear-perfect success rates [IT] We instead create benchmark of 12 dynamic tasks in Kinetix 43 which uses force-based control, so inference delay necessitates asynchronous execution (there no concept of "holding position").. We select 10 existing environments and create 2 new ones such that all environments involve dynamic motions like throwing, catching, and balancing. To simulate imperfect actuation, we add Gaussian noise to the actions, making closed-loop corrections crucial for success.
Setup. To generate data for imitation learning, first train expert policies using RPO [50] and binary success reward. For each environment. we train 6 expert policies with different seeds and then generate 1M transition dataset with different policy selected each episode. We then train action chunking flow policies with prediction horizon of H = 8 and a 4-layer MLP-Mixer [6]] architecture for 32 epochs. We report binary success rates with 2048 rollouts per data point, and simulate delays between 0 (fully closed-loop) and 4 (the maximum supported when H = 8).

---

Jaunch
cartpole tbrust
catapult
eatcher
chaio Jander
grasp easy
har unicycle
hard Lunar Jander
Average Over Environments (Fixed d
0.875
Averuge
miji hall cheetah
mjjeswimmer
0,850
Rotre
Sobre Btste
0.825
Solve Bate
RTC (ours)
mie walker
trampoline
RTC (hard maskang)
0.775
BID (Liu et al)
Naive soyne
0.790
Exccution Horizon,
Inference Delay,
Figure 5: Top left: Kinetix enviromments; cach involves getting green object on the left to touch blue one on the right. Bottom left: Execution horizon solve rate with fixed inference delay of 1. Only RTC and BID take full advantage of faster updates, showing strictly increasing performance with decreasing exccution horizon. Right: Inference delay vs. solve rate baselines. Furthermore, soft masking (Sec. (3.2 improves performance at lower inference delays and execution horizons. Each data point represents 2048 trials, and 95% Wilson score intervals are shaded in.
fixed execution horizon of = max(d, 1). RTC outperforms all
Baselines. We compare against the following baselines:
Naive asyne. This strategy does not pay attention to the previous action chunk all when generating new one, naively switching chunks as soon as the new one is ready.
Bidirectional decoding (BID; (39). This strategy uses rejection sampling to keep continuity across chunks. We use batch size of N 32, mode size of K = 3, and a checkpoint trained for 8 epochs as the weak policy.
Temporal ensembling (TE; (68]). This strategy involves keeping a buffer of predicted action chunks and executing an average of actions predicted for a particular timestep.
Results. Figure shows the simulated results. In the delay plots (right): TE performs poorly across the board, even with an inference delay of d 0, illustrating the multi-modality of our benchmark- averages of valid actions are not necessarily valid. RTC shows the most robustness to inference delays, outperforming BIDD, and the gap widens with increasing delay; note that BID uses significantly more compute than RTC by sampling batches of 64 action chunks, 32 from a strong model and 32 from a weak model. Additionally, we find that hard masking somewhat underperforms soft masking, particularly when d is smaller, supporting our claims in Sec. 3.2 Finally, in the execution horizon plot (left), we find that thanks to its continuity across chunks, RTC is better able to take advantage of closed-loop corrections, always performing better with decreasing execution horizon.
4.2 Real-World Results
Next, we deploy our full real-time chunking system to the real world. We use the No.5 VLA base policy, and evaluate RTC on bimanual system with two 6-DoF arms and parallel jaw grippers. Unlike our simulated benchmark, the robots use position control, and so synchronous inference- stopping between chunks is reasonable default strategy, used in many prior works [S] 24 31 47]. Our goal to improve upon synchronous inference combination of both performance and speed.
24 as our
Setup. We use To.5 (H = 50, A/- = 20ms) with n 5 denoising steps, giving a model latency of 76ms for the baselines and 97ms for RTC. We remote inference over LAN, which adds 10-20ms of latency, giving starting inference delay around d z for RTC. However, we would like to understand how the system behaves with higher inference latencies. simulating, e.g., scaling up the model size or rumning inference on distant cloud server. Thus, we also evaluate all methods with 11 and d 16, respectively.
+ +100ms and +200ms of injected latency, corresponding
Tasks and scoring. Each episode gets an integer score corresponding to how many substeps of the task it completed successfully, We evaluate the following tasks:
7

---

Light candle (S steps, 40s cutoff). Pick up a match and matchbox, strike the match, use it to light bowl.
a candle, and drop
Plug ethernet (6 steps, 120s cutoff). Pick up the end of an ethernet cable, reorient it, plug it into server rack, and repeat the process for the other end.
Make bed, mobile (3 steps, 200s cutoff.. Move the corner of a blanket and 2 pillows from the foot bed.
to the head
Shirt folding (1 step, 300s cutoff). Fold a shirt from flattened position.
Batch folding (4 steps, 300s cutoff). Take a varied, crumpled clothing item out a bin, flatten it, pile.
fold it, then place neatly
Dishes in sink, mobile (8 steps, 300s cutoff). Move 4 varied items from a counter into sink.
See the accompanying blog post for videos of each task. We evaluate each task and method for 10 trials for total of 480 episodes, adding up to hours of pure robot execution time. We also post-hoc annotate the score for each episode and the timestamp at which each step is achieved.
Baselines. We compare to the following baselines:
Synchronous. This corresponds to the default inference strategy in prior work S
24
which executes 8 = 25 actions and then pauses while the new chunk is generated. TE, sparse. This is similar to naive async in our simulated results; it executes a time while computing the next chunk parallel. We found it significantly reduced jerkiness to also apply TE, even though only the first H overlapping actions ensemble.
25 actions at
2d executed steps of each chunk have
TE, dense. This strategy is the closest the original TE in Zhao et al. (68). We run inference as often as possible, resulting in s = d for every chunk. This results in there always being at least overlapping action chunks to ensemble, and often more.
We do not compare to BID 39 in the real world, as we found in simulation that underperforms RTC while using significantly more compute-wii -when applied to oo.5 with batch size of 16, BID has 2.3 times the latency of our method (see A.3 for latency measurements).
lutch Folding
making
drebe
light
plug
shirt fokiing
Comdlhhepren proge
0.75
0.50
0.23
0.00
5k
10k
153.0
5k
10%
Sk
10k
0
24
2h
64
5k.
10k
15k
Controller steps
Method
0.8
RTC (ours)
Awcrage threrggopat (tasks/min))
Syachronous
Cumulative progress
0.6
(2ha sparse
TE (Zhan all) dense
0.6
9.4
Delay
02
+Ums
1000ms
+200ms
0.0
0.0
10
102
-100ms Injected Deaay
-200ms
Time (8)
Figure 6: Top: Controller steps (equivalent to elapsed time with inference pauses removed multiplied by 50Hz) vs. cumulative progress for each task, aggregated across all delays. Progress is measured in discrete steps corresponding to the subsections of task. Left:: Time (including inference pauses) vs. cumulative progress aggregated across all tasks. The x-axis is log scale to better show progress during both short and horghorizon tasks. Right: Inference delay vs. average throughput, defined as the proportion of task completed divided by duration of episode averaged over episodes. Error bars +1 SEM. Average throughput gives balanced view of both speed and performance for each method. Neither TE variant latency. causing such high oscillations that robot's protective stop is triggered.
+100 +200ms of injected
In average task throughput, measurement of both speed and performance, RTC achieves the best score at all inference delays with a statistically significant result at +100 and +200ms. RTC is completely robust to injected delay, showing no degradation,
Results. We present the results in Figure 6

---

whereas synchronous degrades linearly and both TE variants do not run at all due to causing such high oscillations that the robot's 's protective stop triggered (see videos). Inspecting the per-task results (Figure 5 top), we can conclude that RTC helps with more than just execution speed: it completes tasks faster than synchronous inference even when inference pauses are removed. All tasks, except for light candle, allow for retrying until the time limit (and To.5 does, in general, exhibit robust retrying behavior). Even though synchronous inference often reaches a similar final score, RTC often completes more of the task earlier in the episode, reflecting fewer mistakes and less retrying. In light candle, the most precision-sensitvv
task-and also the only one without retrying- RTC shows a large advantage in final score, reflecting a higher overall success rate. Interestingly, the same is true in bed making, even though that task does elicit retrying. The policy particularly struggles to manipulate the pillows, and bed making is the hardest task overall, which may be why RTC has strong effect.
5 Related Work
3 action Action chunking, VLAs, and cascade control. Inspired in part by human motor control
chunking has recently emerged as the de facto standard in imitation learning for visuomotor control [1. Learning to generate action chunks from human data requires expressive statistical models, such as variational inference
[68 19 diffusion [T 12
69
68
46
59], flow matching S vector
6
quantization [4| 3 44], or byte-pair encoding 47 Recently, some of these methods have been
37 class of large models built on pre- tre-trained vision- language model backbones. With the capacity to fit ever-
scaled to billions of parameters, giving rise VLAS
S
7 10
g
70
24
47
41 27], as well as Internet knowledge from vision-language pre-training, VLAs have achieved impressive results in generalizable robot manipulation. When applied to real-world robots, action chunking policies are often used in conjunction with lower-level, tigher-frequency control soch-such as a PID controller-whie translates the outputs of the policy joint (c.g.. joint positions) to hardware-specific control signals (c.g.. joint torques). In these cases, action chunking policies can be viewed as form of cascade control [14], with the learned policy acting the outermost control loop. However, this is not always the case: for example, our simulated experiments use learned policies that output torques and forces directly. As such, we defer any exploration of the intersection between cascade control theory and learned action chunking policies to future work.
growing robot datasets [33 29
62
[5]
Reducing inference latency. A natural approach to improve the real-time capabilities of a model is to simply speed it up. For instance, consistency policy [49] distills diffusion policies to elide expensive iterative denoising. Streaming diffusion policy (3 proposes an alternative training recipe that allows for very few denoising steps per controller timestep. Kim et al. [3 augment Open VLA 30 with parallel decoding to clide expensive autoregressive decoding. More broadly, there is rich literature on optimizing inference speed, both for diffusion models [52 38 56 [7] and large transformers in general 33 25 35). Unfortunately, these directions cannot reduce inference cost below one forward pass. So long as this forward pass takes longer than the controller's s sampling period, other methods will be needed for real-time execution.
Inpainting and guidance. There is rich literature on image inpainting with pre-trained diffusion and flow models 48
In our work, we incorporate one such method [48] into our novel real-time execution framework with modifications (namely, soft masking and guidance weight clipping) that we find necessary for our setting. For sequential decision-makine pioneered diffusion- inpainting for following state and action constraints in long-term planning. though their inpainting method is not saidance-bbsed. (See Appendix A.4 for comparison to the inpainting method from Diffuser applied to our setting.) Diffuser and other work [64] [1) have also guided diffusion models with value functions to solve reinforcement learning (RL) problems. Our work is distinct in that it is the first to apply either inpainting or guidance to real-time control.
55
40
Diffuser [26)
Real-time execution. Real-time -ime control has been studied long before the advent of VLAs. Similar to action chunking, model predictive control (MPC; (5T) generates plans over a receding time horizon; like our method, it parallelizes execution and computation, and uses the prior chunk to warm-start planning for the next. Though recent works combining learning methods with MPC have demonstrated real-time control capabilities in narrow domains [53, 21]. they rely on explicit, hand-crafted dynamics models and cost functions. These methods are not applicable to our setting, which considers model-free imitation learning policies and tests them on unstructured, open-world manipulation tasks. Separately, in reinforcement learning, a variety of prior works have developed
time-delayed decision-make making methods [57 16
54
63
66 67]. However, these approaches not
9

---

always applicable to imitation learning, and none of them leverage action chunking. Most recently, hierarchical VLA designs [58] 4 have emerged where the model split into System 2 (high-level planning) and System (low-level action generation) component. The System 2 component contains the bulk of the VLA's capacity and runs at low frequency, while the System component is lightweight and fast. This approach is orthogonal to ours, and comes with its own tradeoffs (e.g.. limiting the size of the System component and requiring its own training recipe).
Bidirectional Decoding. The most closely related prior work is Bidirectional Decoding (BID; (9)) which enables fully closed-loop control with pre-trained action chunking policies via rejection sampling. While Liu et al. [39] do not consider inference delay, the BID algorithm can be used to accomplish the same effect as our guidance- based inpainting. We compare to BID our simulated benchmark, finding that it underperforms RTC while using significantly more compute.
6 Discussion and Future Work
Real-time chunking is an inference-time algorithm for asynchronous execution of action chunking policies that demonstrates speed and performance across simulation and real-world experiments, including under sigmificant inference delays. However, this work is not without limitations: it adds significant computational overhead compared methods that sample directly from the base policy, and it is applicable only to diffusion- and flow-based policies. Additionally, while our real-world experiments cover variety of challenging manipulation tasks, there are more dynamic settings that could benefit even more from real-time execution. One example is legged locomotion, which is represented in our simulated benchmark but not our real-world results.
7
Acknowledgements
We thank Charles Xu and Kay Ke for designing the Ethernet plug-in task. We thank Brian Ichter for suggesting the cumulative progress plots and for later feedback on figures. We thank Dibya Ghosh for suggesting the throughput metric to measure combination of speed and performance. We thank Ury Zhilinsky, Karan Dhabalia, Haohuan Wang, and Dibya Ghosh for help with training infrastructure; Noah Brown, Szymon Jakubezak, Adnan Esmail, Tim Jones, Mohith Mothukuri, James Darpinian, and James Tanner for robot infrastructure; Adrian Li-Bell for evaluation infrastructure; Anna Walling, Chelsea Finn, and Karol Hausman for robot, data and evaluation operations; and Michael Equi, Quan Vuong, and Jost Tobias Springenberg for training some of the No.5 policies used in the real-world experiments. We also thank Claudio Guglieri and Alex Krasikov for their help with visualizations for the blog post, and Jessica Dai for helpful copy editing of the paper manuscript. Finally, we are grateful to the whole team of robot operators at Physical Intelligence for their enormous contributions to running data collection and policy evaluations.
References
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211. 15657, 2022.
[2] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey, Journal of machine learning research, 18(153):1 2018
[3] Suneel Belkhale and Dorsa Sadigh. Minivla: A better vla with a smaller footprint, 2024. URL ttt.////ttttuttcceo com wttp:/tttwwbcoottttoooooooodd--e wttp:///ffooatAA///ppeeee//////e ILADD/opeee/laammii
[4] Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GrOOt nl: An open foundation model for generalist humanoid robots. arXiv preprint N.VV:25503.17334 2025.
[5] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A lasion-language-cttinn flow model for general robot control. arXiv preprint arX1v:2410.24164, 2024.
10

---

[6] Max Braun, Noémie Jaquier, Leonel Rozo, and Tamim Asfour. Riemannian flow matching policy for robot motion learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5151- IEFE, 2024.
[7]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding. Danny Driess, Avinava Dubey. Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Aremas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang- Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In arXiv preprint wrXiv:2307.15818 2023.
[8] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge robotic control. arXiv preprint arXiv:2307.15818 2023.
[9] Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, and Minzhao Zhu. Gr-2: A generative video-language-actte action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06118, 2024.
[10] An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Xueyan Zou, Jan Kautz, Erdem Biyik, Hongxu Yin, Sifei Liu, and Xiaolong Wang. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. arXiv preprint
2tn::2412.44455. 2024.
[11] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 027836492412736688 2023.
[12) Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng. Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10332 2024.
[13] OX-Embodiment Collaboration, A Padalkar, A Pooley, A Jain, A Bewley, A Herzog, A Irpan, Khazatsky, A Rai, A Singh, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.0880 0.88644, 1(2), 2023.
[14] Donald R. Coughanowr and Steven E. LeBlane. Process Systems Analysis and Control, chap- ter 18. McGraw-Hile New York, 3rd edition, 2009. ISBN 978 978-0073397894.
[15] Hao-Shu Fang, Hongjie Fang, Zhenyu Tang. Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20c: A comprehensive robotic dataset for learning diverse skills in one-shot. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 653-660. IEEE, 2024.
[16] Vlad Firoiu, Tina Ju, and Josh Tenenbaum. At human speed: Deep reinforcement learning with action delay. arXiv preprint arXiv:1810.072266 2018
[17] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shorteut models, arXiv preprint arXiv:2410.1255) 2024.
[18] Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli. Kevin P. Murphy, and Tim Salimans. Diffusson meets flow matching: Two sides of the same coin. 2024. URL
httts:///aiffusionn// www.wllwww...t........... github.io 10/
[19] Abraham George and Amir Barati Farimani. One act play: Single demonstration behavior cloning with action chunking transformers. arXiv preprint arXiv:2309.10177 2023.
11

---

[20] Anthony Ha.
Google's
gemini
has
beaten
pokémon
blue
(with a little
help).
www.aachhhccooooooooooooooooceoe https://techcruuch.. www.coomoo/o/o homm/025/05/0333
googles- gemini hwseaaas-aaae
May
2025.
Accessed May 8, 2025.
[21] Nicklas Hansen. Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control, 2022. URL htttt:////axiiv org/ wwwp///..... 04955
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[23] Sigmund H Høeg, Yilun Du and Olav Egeland. Streaming diffusion policy: Fast policy synthesis with variable noise diffusion models. arXiv preprint arXiv:2406.04806 2024.
[24] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. To.s: A vision- language-action model with open-world generalization. arXiv preprint arXiv:2504. 16054, 2025.
[25] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-onn inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
[26] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0m 09991, 2022.
[27] Zhenyu Jiang. Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. arXiv preprint 2rXiv:2410.24188. 2024.
[28] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836 2020.
[29] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee. Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi. Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe. Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin. Zehan Ma. Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang. Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Marttn-Marttnn Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song. Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the- wild robot manipulation dataset. In Proceedings of Robotics: Science and Systems, 2024.
[30] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-languagc-action model. arXiv preprint arXiv:2406. 0924446009996..... 2024.
[31] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision- sssion-anguagg-acttinn models: Optimizing speed and success. arXiv preprint arXiv:2502. 19645, 2025.
12

---

[32] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023.
[33]
Lucy Lai, Ann Zixiang Huang, and Samuel Gershman. Action chunking as policy compression. 2022.
[34] Seungjae Lee, Yibin Wang. Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint S4MV::2003.0318 2024.
[35] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awar weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[36] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 2022.
[37] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang. Ke Xu, Hang Su, and Jun Zhu. Rdt- a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.0786/ 2024.
[38] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXhv:2209.03000 2022.
[39] Yuejiang Liu, Jubayer Ibn Hamid, Annie Xie, Yoonho Lee, Maximilian Du, and Chelsea Finn. Bidirectional decoding: Improving action chunking via closed-loop resampling. arXiv preprint arXiv:2408. 77355, 2024.
[40] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-1147,, 2022.
[41] Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879-893. PMLR, 2018.
[42] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. arXiv preprint arXiv:2305.04331, 2023.
[43] Michael Matthews, Michael Beukman, Chris Lu, and Jakob Foerster. Kinetix: Investigating the training of general agents through open-ended physics-based control tasks. arXiv preprint arXiv:2410.23205. 2024.
[44] Atharva Mete, Haotian Xue, Albert Wilcox Yongxin Chen, and Animesh Garg. Quest: Self- supervised skill abstractions for learning continuous control, 2024. URL htttss://axii org/abs/2444 15840
[45] OpenAl. Introducing openai codex, August 2021. URL https://openai.com cttpaaiicoomiiiii....eee introducing-codee Accessed on May 27, 2025.
[46] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301. 10677, 2023.
[47] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- lasio-llgguage-acttion models. arXiv preprint 07X7 2025.
13

---

[48] Ashwini Pokle, Matthew Muckkly, Ricky TQ Chen, and Brian Karrer. Training-free free linear image inverses via flows. arXiv preprint arXlv:231000442,, 2023.
[49] Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency policy:
Accelerated visuomotor policies via consistency distillation. arXiv preprint arXiv:2405.07503. 2024.
[50] Md Masudur Rahman and Yexiang Xue. Robust policy optimization in deep reinforcement learning. arXiv preprint arXiv:2212.075533 2022.
[51] J.B. Rawlings, D.Q. Mayne, and M. Diehl. Model Predictive Control: Theory, Computation, and Design. Nob Hill Publishing, 2017. ISBN 9780975937730. URL hittps://bbbat google
ch/books ? 3i=MMrctAAMAAAAMM
[52] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2222.000
2022.
[53] Tim Salzmann, Elia Kaufmann, Jon Arrizabalaga, Marco Pavone, Davide Scaramuzza, and Markus Ryll. Real-time neural mpe: Deep learning model predictive control for quadrotors and agile robotic platforms. IEEE Robotics and Automation Letters, 8(4)) -2004, 2023.
[54] Erik Schuitema, Lucian Buşoniu, Robert Babuška, and Pieter Jonker. Control delay in rein- forcement learning for real-time dynamic systems: A memoryless approach. 2010 JEEE/RSJ international conference intelligent robots and systems, pages 3226-3231. IEEE, 2010.
[55] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinvers--guided diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
[56] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[57] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint 180iv:1804..10 10332, 2018
[58] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et Gemini robotics: Bringing ai into the physical world. arXiv preprint arXivv2503.20020, 2025.
[59] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, al. Octo: An open-source generalist robot policy. arXiv preprinl arXiv:2405.122: 12213, 2024.
[60] Waywe Research Team et al. Lingo-2: Driving with natural language, 2024.
[61] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34: 24261-24272, 2021.
[62] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridge v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723-1736 PMLR, 2023.
[63] Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Planning and learning in environments with delayed feedback. In Machine Learning: ECML 2007: 18th European Conference on Machine Learning, Warsaw, Poland, September 17-21, 2007. Proceedings 18, pages 442-453. Springer, 2007.
[64] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06793 2022.
[65] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516 2025.
14

---

[66] Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine. Julian Ibarz, Karol Hausman, and Alexander Herzog. Thinking while moving: Deep reinforcement learning with concurrent control. arXiv preprint arXiv:2004.06089, 2020.
[67] Bin Xu, Farzam Malmir, Dhruvang Rathod, and Zoran Filipi. Real-time reinforcement learning optimized energy management for a 48v mild hybrid electric vehicle. Technical report, SAE Technical Paper, 2019.
[68] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[69] Tony Z Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity, arXiv preprint 2rX1v:2410.13/26, 2024.
[70] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d saiion-lanuaage-cttin generative world model, arXiv preprint 2024.
[71] Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III. Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporia awareness for generalist robotic policies. arXiv preprint 2rXiv:24/2.10344 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Yes, the abstract and introduction accurately reflect our contributions. The abstract claims that we present method for asynchronous execution of action chunking policies that improves speed and performance, which demonstrated through extensive experiments in both simulation and real-world settings. The introduction expands on these claims while clearly stating limitations, such as the method only being applicable to diffusion and flow-based policies. All claims are supported by our experimental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the work are discussed in Sec. which include the scope (diffusion- and flow-based policies only), computational efficiency, and shortcomings of the experiments.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides a detailed algorithm Algorithm thhat can be used to reproduce the experimental results real robots using any pre-trained flow-based policy. The full code to reproduce the simulated benchmark and its results are provided in the supplemental material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes] and [No]
Justification:
The full code and instructions reproduce the simulated benchmark and its
results are provided in the supplemental material. The data used to train Wo.5. as well as the real robot runtime code, are released as these are proprietary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cor
Beallic/guiees/Cee (odeeSSumissiiooool111. httpii//uuidss/dddettbssssssssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
pciips.ccpubllicguui..es sew.eddeee/ooeeSeiiiiiiieeeeeeee wwt.ssubbisssiniin ssinnooiccoyyyiccooooooooopoiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and testing hyperparameters are provided in the supplemental material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are reported, and described, everywhere in the experiments section where mean of multiple data points is reported.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides compute details in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps:/nnerrpps wwtpe/uubbl.eeeeeettttueeeeeeeee
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics. This work aims primarily to advance the state of the art in end-to-end robot learning. While there is always cause farm-for example, in military applications the
the potential for technology authors believe that this work does not contribute unduly to these risks. The particular applications considered in the experiments of this work are focused household robotics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
19

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: A broader impacts statement is included in the supplemental material. Also see the response to the previous question.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not release data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original creators models (To.s 244 and simulators (Kinetix 43 that we use the paper. The Kinetix software is released under the MIT license.
Guidelines:
20

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The full code reproduce the simulated benchmark and its results are provided in the supplemental material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
21

---

Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The paper does not use LLMs important, original, or non-standard component the core methods.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
wwweeriisscccooofeeeee /2/Connferencce/22255//////////e /LLM
22

---

40
0.9
Delay
Solve Rate
0.8
valtee
0.7
0.6
4.25
0
0.00
0.25
0.50
0.75
1.00
T
Max Acceleration
2.5
rad
rad
Prev. chunk
2.0
steps
100 steps
1.5
d = 15
49
5
20
50
150
Step
B
Figure 7: Top left: The graph of the value ..?? from Eq. which we clip at 3. At = 0. clipping is needed to make the value finite. With 5 denoising steps. if 2 4.25, the clipping only determines the guidance weight for the first step no marginal benefit beyond
0). Top right: An ablation of 3 in simulated benehmark. Increasing provides
Bottom left: Example real robot action chunks generated from the same noise with denoising steps (n = 5) and 100 denoising steps (n = 100), with lower opacities corresponding to higher guidance weight clipping (33 = (5. 20, 50, 150)). With 5 denoising steps, the generated action chunks diverge when 3 is too high. Bottom right: / vs. maximum acceleration (second discrete difference) for batch of 325 action chunks generated with d 15 and 5. Higher / leads jerkiness, proxy for di-tr---ittiittiinn actions.
A
Appendices
A.1 Broader Impacts
The goal of our work is to improve the speed and performance of learned policies for control tasks, and our experiments primarily deal with household robots. This technology has great potential to improve lives, e.g., by automating dangerous and difficult jobs, or assisting the disabled and elderly. Like any technology, also has the potential for harm-e.g. military applications, or by displacing physical labor.
A.2 The Necessity of Guidance Weight Clipping (g)
In Section 3.1 we describe how we adapt the inpainting algorithm from Pokle [48] and Song
et al. [55] to our setting. One modification we make is to add a clipping value, B, which limits weight
applied to the guidance term (Eq. 2). and necessary to make the weight finite at т image inpainting typically uses high number of denoising steps (e.g., n 100 in 48 control problems often use very few steps (e-g., n in our experiments). this case, we found that high guidance weights led to diverging action chunks, as shown in Figure 7 bottom left. Based on simulated ablation (Figure 7 top right), we set B to a conservative value
While
An alternative approach to avoid the infinite weight at 0 is to start denoising from > 0, used in 48 which we did not try.
23

---

A.3 Latency Measurements
Method
Latency
RTC (ours)
97ms
BID with N 16 (no forward model)
115ms
BID with 16 (shared backbone)
169ms
BID with N 16 (full)
223ms
Vanilla #T.55
76ms
Table I: Latency measurements for various inference-timg methods applied to wo.s [24] Numbers include on-GPU neural network inference only, and are averaged over 10 inference calls after 5 warmup calls. Inference IVIDIA RTX 4090 GPU using bfloatló precision and down inference due sampling batches of actions, whereas RTC slows down inference due backpropagating through each denoising step. BID (no forward contrast) refers to a version of BID without the forward contrast loss, which elides the need for second model. BID (shared hackhone) refers to a version of BID optimized specifically for the architecture, where the VLM backbone (3B parameters) is shared between the strong and weak model, so only two copies of the action expert (300M parameters) needed. Full BID requires two copies of entire model.
5 denoising steps. BID 39 slows
Component
Time (mobile)
Time (non-mobile)
Model
96.89 0.16ms
97.43 + 0.28ms
Network
21.20 = 3.12ms
6.89 = 2.39ms
Image resize
11.22 5.00ms
1.44 + 0.27ms
Other
9.67 3.20ms
3.00 = 0.68ms
Total
138.98 6.71ms
108.76 + 2.34ms
Table 2: Breakdown of total inference latency by component for RTC. The image resizing component happens on the CPU of the rohot computer. the mobile manipulation case, this computer is an Intel NUC portable computer with 1 12th Gen Intel i7-1260P processor. In the non-mobile case, this computer is desktop workstation with AMD Ryzen 7950X processor. hoth cases, the model runs separate workstation with NVIDIA RTX 4090 GPU; the robot computer and inference workstation are both connected to same LAN via a wired Ethernet connection, and communication happens via the WebSocket protocol. Model inference uses 66 precision and 5 denoising steps. Measurements are taken from 50 inference calls during real episode rollout, and + standard deviation is shown.
Component
Time (no RTC)
Time (with RTC)
Image encoders (SigLIP) LLM prefill (Gemma 2B)
18ms
18ms
44ms
44ms
Denoising step (x5)
14ms
35ms
Total
76ms
97ms
Table 3: Breakdown of model inference latency by component for vanilla mo.s and RTC. Measurements are taken from a single profiling trace for each method, run on RTX 4090 GPU. RTC incurs 2.5x latency increase per denoising step.
24

---

The code for the simulated experiments is available at chwnkinggkiinnn A.4 Additional Simulated Ablations
rar
cietpole
rataguil
chetpoll
rateddi
cateher
grasp sway
unieyele
Junar
grasp sher
MP unieycle
banl
Average
my:
myjt
Average
mnje
088
Salve Rate
0.7
Esponendal decny (puit))
waller
0.7
waller
tramppolize
96
Lincue doouy
96
RDD (lix at all
Diffsse ijannes tapoe
05
doeay (hard masking
Inference Delay,
Delay
Figure 8: Left: Simulated ablation over different schedules for soft masking weights (Eq. 5 Expomential decay performs the best overall, although linear decay is very close behind. Right: Comparison with the inpainting algorithm from Diffuser [26). which overwrites portion of the action chunk with the desired actions each denoising step. While this simpler (and cheaper) inpainting method still provides some benefit, it outperformed by our guidance-based approach.
A.5 Hyperparameters
Hyperparameter
Description
Simulation
Real-world
n
Denoising steps
H
Prediction horizon
50
Smin
Minimum execution horizon
25
Guidance weight clipping
Delay buffer size
10
Table 4: Hyperparameters used for RTC (Algorithm I
In simulation, d is held constant for each experiment, Smin and are needed. Additional hyperparameters for the simulated experiments can found in the code.
A.6 Code Release
https://github.com
A.7 Compute Resources
All the experiments in this work use no more than 8 NVIDIA H100 GPUs (one NVIDIA DGX server)
at a time. H100s are used via cloud provider.
Simulated experiments. Training expert policies with RPO 50 with 6 seeds x 12 environments takes approximately hours on 4xH100s. Generating data from those policies takes approximately 20 minutes on 6xH1 00s. Training imitation learning policies with flow matching for each environment takes approximately 1.5 hours on 2xH100s. Evaluating the policies for 2048 trials per environment takes approximately 5 minutes on 6xH100s.
Real-worlde experiments. We policies fine-tuned from the mo.5 24 base model. Each fine-tuning run takes approximately 24 hours on 8xH100s. All of our real-world inference is done on a single NVIDIA RTX 4090 GPU in workstation in the same building as the robots.
25


Reviews:
[{"review_id": "pwSVt30zv0", "reviewer": "Reviewer_jJyz", "summary": "This paper address the problem of how to accelerate the execution of policies encoded using action chunking. The approach uses conditional flow matching, and rather than perform synchronous action chunk generation, where the next action chunk is generated after the previous action chunk completes, the approach generates the next action chunk during the previous action chunk execution, and uses inpainting to ensure that the generated action chunk is consistent with the action chunk being generated. The experimental results show that in some cases the execution of the full policy is faster and is sometimes able to complete the task more reliably.", "questions": "- Why does the performance drop at d = 4? Why do the results not go to d = 6?\n- What happens in this paper when the flow matching returns quickly?\n- How sensitive is the performance of the model to changes in the exponential weight decay function?", "limitations": "The limitations section is exceedingly short. The authors should be encouraged to put more careful analysis into their technique.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The problem, as posed, is fairly clear and relevant, and the paper is reasonably clearly written and the technique is sound.\n\nThe primary difficulties I have with this paper are the problem setting and the experimental evaluation. Firstly, the paper describes the proposed approach as real-time, but real-time has a specific meaning, which is that there is a guaranteed response time. This paper accelerates execution but there is no discussion of a guarantee on response time, or a response time to what. It is not clear it is possible to guarantee that a flow matching technique can be made real time, in that if a given denoising step takes longer than the required response time for whatever reason, the policy cannot return early. This might seem like a quibble to the authors and can probably be addressed by changing the title and framing of the paper, but it is an important point in how the authors take about their work.\n\nThe lack of clarity also shows up in the experimental results. The authors describe a minimum delay of somewhere between 76 and 98ms, plus 25ms of transmission time over the LAN, for a minimum of $d \\approx 6ms$. However, we see in figure 5 that the performance drops substantially up to $d = 4$, and results are not given beyond that. The authors then state \"we would like to understand how the system behaves with higher inference latencies,\" and inject latencies of 100ms and 200ms, which are large (arguably unrealistic?) latencies given the stated properties of the existing system. \n\nThe authors are not especially careful in presenting their results. For instance, \"we find that hard masking somewhat underperforms soft masking, particularly when d is smaller.\" The fact that there are no error bars raises the question of how significant this is, but the authors need to be careful to point out that hard masking *overperforms* soft masking when d is large in the cartpole thrust problem. The text of the paper barely mentions that in both figure 5 and figure 7, we see that no technique is able to solve the problems with a high rate of success (solve rate is not defined. I assume this is the proportion of trials where the task was completed correctly.), and the differences are relatively modest -- naive async is 70%ish vs RTC at about 78 or 79% averaged across sim environments. For the real results, only for dishs_in_sink and shirt_folding do we see what appears to be 100% completion (the 100% line is not labelled). And for dishes_in_sink, only TE seems to achieve completion. \n\nThe approach has some parameters, and it would be helpful to know how sensitive performance is to the choice of parameters (number of denoising steps, faster denoising vs open-loop performance. Equation 5 gives the exponential decay of the weights during in-painting -- if the exponential were encouraged to delay faster, or more slowly, how sensitive is the performance of the model? \n\nThe authors are also not rigorous in the related work in terms of MPC techniques -- the issue of executing a partial control and then generating a new one is a very well-studied problem in control -- there are principled techniques from people like Chris Gerdes who have examined when to to switch to a new control given a partial plan. What happens in this paper when the flow matching returns quickly? This can lead to thrashing, and is not discussed in this paper. \n\nI dislike the term incremental, in that incremental results are important, but the relative modest technical contribution of this paper puts a lot of onus on careful experimental evaluation, which is not present in this paper.", "was_revised": true, "final_justification": "I believe that the authors have made a good faith effort to address my concerns. I am willing to overlook the unfortunate response, which the authors did attempt to walk back. However, my original criticism stands, and I am not supportive of the paper.  I really wish that the authors had taken the opportunity to take a step back at the onset of the research, and identify what is already known about problems where the system will evolve out from under the decision making system. This is a well-studied problem and the authors do themselves and the community a disservice by not doing that kind of scholarship first. \n\nI understand that the specific problem of how to deal with this in the context of diffusion, and the effect of smoothed inpainting, may not have been tried before (it would be hard to imagine it had been tried before, given how new diffusion is as a practical tool). But, I am arguing for going beyond simply trying something and reporting its results, however carefully analysed those results are.", "rebuttal": "Thank you for your in-depth review! We agree that framing our claims properly is very important, and we appreciate your help in treating this subject carefully. We will attempt to address your concerns by: (1) expanding and clarifying our usage of the term “real-time”, and being more explicit about our assumptions; (2) adding more simulated ablations; and (3) clarifying a number of points that we believe may stem from misunderstandings, regarding which we will improve intelligibility in a future revision. Please let us know if any further issues remain!\n\n> Firstly, the paper describes the proposed approach as real-time, but real-time has a specific meaning, which is that there is a guaranteed response time. This paper accelerates execution but there is no discussion of a guarantee on response time, or a response time to what.\n\nThis is a great point! We had a discussion among the authors about this exact point, and ultimately decided on “real-time” for the following reason: we can think of each low-level controller timestep as an “event”, and the procurement of the next action as the “response”. Prior synchronous action chunking strategies are not real-time because, every $s$ steps (the execution horizon), the controller pauses and waits for model inference, meaning the event-response time is suddenly and unexpectedly many times higher. In RTC, every response comes immediately (in $< \\Delta t$ seconds) because model inference is done asynchronously with controller execution.\n\nWe agree that this detail is important, and we will update the paper to explain how RTC satisfies the real-time guarantee, namely that it always returns an action response within the $\\Delta t$ deadline. As with any real-time system, it is possible for this guarantee to fail in cases of catastrophic failure, e.g., a temporary network outage that causes $d$ to exceed $H - s$ (this never happened throughout our 480 evaluation episodes). However, we will also clearly state this possibility as a limitation.\n\n> The authors describe a minimum delay of somewhere between 76 and 98ms, plus 25ms of transmission time over the LAN, for a minimum of $d \\approx 6ms$. However, we see in figure 5 that the performance drops substantially up to $d = 4$, and results are not given beyond that.\n\nWe believe that there are a few misunderstandings here, and we will clarify the writing on this point in a future revision. 76ms is the model inference delay for the baselines, and 98ms is the model inference delay for RTC (line 263). $d$ is in units of controller steps, not milliseconds, and is calculated using the equation on line 129: $d = \\lfloor(98ms + 25ms) / (20ms)\\rfloor = 6$. We will expand that section to clarify the distinction.\n\nFigure 5 describes the simulated experiments, not real-world experiments, meaning that we control $d$ directly independent of model latency. $d = 4$ is the maximum because we train with a horizon of $H = 8$, and the maximum possible $d$ is $\\lfloor H / 2 \\rfloor$ (mentioned on line 238). This means that our simulated experiments test very high relative latencies — up to half of the prediction horizon — which is why performance drops so much across the board (but less so for RTC).\n\n> and inject latencies of 100ms and 200ms, which are large (arguably unrealistic?) latencies given the stated properties of the existing system\n\nThese are very realistic latencies. Many VLAs are already much slower than Pi05: “Kim et al. [30], who optimize the 7B OpenVLA model [29] specifically for inference speed, achieve no better than 321ms of latency on an NVIDIA A100 GPU” (line 145). If the scaling properties from vision and language hold in robotics, we should expect the best-performing VLAs to increase in size as time goes on (mentioned in line 37). Additionally, as mentioned throughout the paper, our experiments were conducted in ideal network conditions with powerful workstations; weaker devices and/or slow networks could easily add latency in the hundreds of milliseconds. These justifications for studying injected latency are mentioned on lines 265-266 of the paper.\n\n> For instance, \"we find that hard masking somewhat underperforms soft masking, particularly when d is smaller.\" The fact that there are no error bars raises the question of how significant this is\n\nFigure 5, which compares hard and soft masking in the simulated benchmark, has 95% Wilson score intervals shaded in (as stated in the figure caption).\n\n\n> solve rate is not defined. I assume this is the proportion of trials where the task was completed correctly\n\nThe Figure 5 caption mentions that each Kinetix environment “involves getting a green object on the left to touch a blue one on the right.” This is what counts as a “success”; we will update the Figure 5 caption to clarify this point and explicitly define the solve rate.\n\n> Equation 5 gives the exponential decay of the weights during in-painting -- if the exponential were encouraged to delay faster, or more slowly, how sensitive is the performance of the model?\n\nWe have performed an additional simulated ablation that we will add to the paper (unfortunately, the rebuttal response does not allow images). We compared 4 schedules: exponential decay (our default), linear decay, constant ones (no decay), and constant zeros (hard masking; already in the paper). Averaged over environments, exponential performs the best, followed by linear, followed by hard masking, followed by no decay. Exponential and linear both outperform BID by a large margin.\n\n\n\n> The authors are also not rigorous in the related work in terms of MPC techniques -- the issue of executing a partial control and then generating a new one is a very well-studied problem in control -- there are principled techniques from people like Chris Gerdes who have examined when to to switch to a new control given a partial plan\n\nCould you point us to some more specific references? We are not aware of any particular techniques from the MPC literature that would be applicable out-of-the-box to our setting, but we would be happy to add any relevant references.\n\n> What happens in this paper when the flow matching returns quickly? This can lead to thrashing, and is not discussed in this paper.\n\nHow often the chunk changes can be controlled independently of how quickly the flow matching returns, and is controlled by the hyperperameter $s_\\text{min}$ (see Algorithm 1, line 13, as well as lines 210-212 of the text). If the flow matching returns quickly, it allows for a lower execution horizon, but does not require it (see line 211 of the text). However, RTC can take advantage of a lower execution horizon due to its consistency across chunks (Figure 5, lower left).\n\nIf by “thrashing”, you are referring to the phenomenon where the robot switches between different plans too quickly, this problem is discussed throughout the paper and is at the core of the motivation for RTC (lines 45, 154-164, 253-255, Figure 2). Apologies if we are misunderstanding your terminology."}, {"review_id": "t1JyC1VyXD", "reviewer": "Reviewer_f8Dc", "summary": "This paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. The proposed method real-time chunking (RTC) is applicable to any diffusion- or flow-based VLA out of the box with no re-training. To test RTC, the authors introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate favorable performance.", "questions": "1. How robust is the proposed method to sudden perturbations?\n2. How sensitive is the overall system performance to the choice of inpainting algorithm?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Timely Contribution to Real-Time Control with Large Models: The application of large models to real-time control is a topic of growing importance in the robot learning community, especially given the inherent latency and computational demands of such models. Addressing this challenge requires effective asynchronous mechanisms. The authors’ proposed method, RTC, is a compelling step in this direction. Figure 1 clearly illustrates RTC’s ability to generate significantly smoother trajectories compared to existing baselines, highlighting its practical benefit. Furthermore, the accompanying demonstration video effectively showcases RTC’s real-time responsiveness and stability in dynamic scenarios.\n\n2. Strong Empirical Validation Across Simulated and Real-World Settings:\nThe authors introduce a new benchmark built on the Kinetix simulator, featuring 12 diverse and highly dynamic tasks spanning both manipulation and locomotion. This benchmark offers a meaningful testbed for evaluating real-time performance under challenging conditions. In addition, the real-world experiments—conducted on 6 complex bimanual manipulation tasks using the π₀.₅ VLA as the base policy—provide strong empirical support for the effectiveness of RTC. Across both domains, the results consistently show that RTC achieves high performance with low latency, underscoring its practical viability for real-time deployment.\n\nWeaknesses:\n1. Clarifying the Broader Impact of Computational Overhead: This work presents a compelling method for enabling real-time control using large models, supported by strong empirical results. However, in the limitations section, the authors primarily acknowledge the added computational overhead introduced by their method. I encourage the authors to further examine and report on whether this overhead affects other critical aspects of system performance—such as latency variability, responsiveness to dynamic changes, or robustness under constrained compute conditions. Understanding the broader implications of this overhead is essential for assessing the method’s practical deployment in resource-limited settings, such as on embedded platforms or mobile robots. One common evaluation scenario is testing robustness to sudden perturbations. It would be valuable to understand how the proposed method performs under such conditions and whether the increased computational cost compromises its responsiveness or stability.\n\n2. Role and Generality of the Inpainting Algorithm: The proposed framework relies heavily on an inpainting algorithm to recover missing observations, which plays a crucial role in enabling effective asynchronous control. In this study, the authors adopt the method by Pokle et al. [46]. However, it remains unclear how sensitive the overall system performance is to this choice. Could alternative inpainting algorithms lead to better, worse, or more efficient outcomes? A comparison or ablation study exploring different inpainting strategies would greatly enhance the generality and reproducibility of the proposed approach. It would also clarify whether the strength of the results stems primarily from the RTC framework itself or is heavily dependent on the chosen inpainting technique.", "was_revised": true, "final_justification": "I appreciate the authors’ efforts in addressing the concerns raised in my original review. I have also carefully considered the comments provided by the other reviewers.\n\nWith respect to my own feedback, I find that the authors have satisfactorily addressed the key issues in their response. Please ensure that the additional results mentioned are incorporated into the final manuscript.\n\nRegarding the comments from Reviewer jJyz, I agree that the manuscript would benefit from a more comprehensive discussion of the existing literature in control, particularly with respect to explaining why those approaches are not applicable to the present work. Additionally, the connection and relevance of this work to cascade control systems should be clarified. Thus, I suggest the authors to tone down sentences like \"this is a machine learning paper in submission at a machine learning conference, not a controls conference.\"\n\nWhile the manuscript lacks a thorough discussion of the control literature, I believe the work maintains a clear focus on addressing a specific problem, i.e., executing action chunking using diffusion policies. Overall, I find it to be a solid empirical study with demonstrated performance, and I recommend that the paper be accepted.", "rebuttal": "Thank you for your in-depth review! We have attempted to address your question about inpainting strategies by adding a simulated ablation: it shows that RTC is still valuable with simpler inpainting strategies, but the work we have done to adapt and tune a state-of-the-art method is critical for maximum performance. Additionally, we have clarified the impact of additional computational overhead and sudden perturbations. Please let us know if you have any other concerns!\n\n> However, in the limitations section, the authors primarily acknowledge the added computational overhead introduced by their method. I encourage the authors to further examine and report on whether this overhead affects other critical aspects of system performance—such as latency variability, responsiveness to dynamic changes, or robustness under constrained compute conditions.\n\nIn Table 2 of the Appendix, we report mean and standard deviations of inference times for each component of the system. We find that even with RTC, latency variance for model inference is very low (<1ms); most of the variance comes from the network component, which is unrelated to RTC. Regarding responsiveness, RTC is necessary to enable responsiveness under delays (see our answer below), so it is difficult to divorce the negative impact of the additional computational overhead from the positive impact of the method itself. Regarding constrained compute conditions, RTC is targeted primarily at offboard inference scenarios, where compute is plentiful, but the network adds inevitable latency. However, if one were to use RTC in onboard inference, we do not see any reason why the backpropagation in RTC would disproportionately affect a smaller GPU (the relative overhead would be the same as on a powerful GPU).\n\n> How robust is the proposed method to sudden perturbations?\n\nUnfortunately, we cannot run more real-world experiments, but we can point out that in the simulated benchmark, the actions are perturbed with random Gaussian noise at every step (lines 231-232). Our results (Figure 5) show that RTC is more robust than other methods, and that it strictly benefits from increasingly closed-loop control (Figure 5, bottom left).\n\n> A comparison or ablation study exploring different inpainting strategies would greatly enhance the generality and reproducibility of the proposed approach.\n\nWe have performed an additional simulated ablation comparing to the inpainting strategy from Diffuser [1], which is simpler and does not include a gradient-based correction. Unfortunately, we cannot attach images to the rebuttal, but the inpainting strategy from Diffuser performs worse than standard RTC across the board; it performs worse than BID at $d = 1$ and $d = 2$, and better than BID at $d = 3$ and $d = 4$; and it performs much better than naive asynchronous and temporal ensembling. Our conclusion is that our full method — with a state-of-the-art inpainting algorithm and soft masking — is important for maximum performance, but that a simpler inpainting strategy can still provide some benefit.\n\n\n\n[1] “Planning with Diffusion for Flexible Behavior Synthesis.” Janner et al."}, {"review_id": "qQIgkzBqAX", "reviewer": "Reviewer_rbym", "summary": "This paper address a practical tension between the unavoidable inference latency of robot foundation models and the desired real-time control on real hardware. \n\n---\n\n## Problem Setting\nThe authors illustrate that for existing VLA models, the first few time steps in the produced action sequence has passed when the inference is done. The deviation in initial condition can cause jerky motions or completely off trajectories depending on the underlying controller.\n\nKey assumptions:\n- Builds from a trained diffusion or flow model that prediction action chunks.\n- Asynchronous execution: the algorithm generates the next action chunk while the current one is still being executed.\n\n---\n\n## Method: Real-Time Chunking (RTC)\n\n- The high level idea is to pose asynchronous action chunking as an inpainting problem.\n- RTC aims to force the first $d$ steps from the new chunk (those executed during the inference delay) to match the corresponding actions from the previous chunk, and then inpaints the rest of the new chunk.\n- This is done by adding a guidance term to the velocity field of the VLA. The guidance term can be computed with back propagation.\n- The authors also propose soft masking: steps past the first $d$ also get exponentially decaying weights to match the previous chunk.\n\n---\n\n## Experiments\n- The authors propose a suite of simulated task using force control. These tasks help highlight the effect of inference delay.\n- In these simulated tasks, RTC is the most robust to inference delay - the task success rate drops the slowest as the inference delay increases. It also performs the best at different execution horizons.\n- In real robot experiments, RTC achieved the highest task throughput across different levels of injected delay. Moreover, unlike the baselines, there is no performance drop as the delay increases.", "questions": "- Why does soft masking match the first $d$ steps better than hard masking? This is a bit counter-intuitive. Doesn't soft masking add more constraints? Also, because soft masking is encouraging closer match, why wouldn't it end up slowing down task progress?\n- Would other types of low level controllers, such as more compliant controllers benefit more or less from this method?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n- No training required.\n- This work addresses a timely and realistic issue with VLAs. The paper provides clear illustrations and real robot rollouts to motivate the problem.\n- The experiment results are strong, showing that the method is able to avoid performance degradation at 200ms injected delay, more than the typical inference delay.\n\n---\n\n## Weaknesses:\n- Limited to flow or diffusion models.\n- This work applies an existing algorithm from image generation to robotics and falls short on novelty.\n- Innovations like soft masking seems to work well empirically but lacks mathematical justification.", "was_revised": true, "final_justification": "I don't have major concern with the paper as a practical fix for flow-based VLA models with empirical justifications. I'll keep my original rating.", "rebuttal": "Thank you for your in-depth review! We appreciate your acknowledgement of the timeliness and practicality of our problem setting. Regarding novelty, we would emphasize that we are the first work to study this problem setting (real-time execution with VLAs when the model latency exceeds a single controller timestep), that our baselines have not been applied to this problem setting before, and that we still provide an effective inference-only solution that outperforms these baselines.\n\nWe have attempted to answer your questions about soft masking, and we will revise the paper to clarify these points, namely its role as an empirical (but not theoretically-motivated) modification to the existing inpainting algorithm. Please let us know if you have any other concerns!\n\n> Innovations like soft masking seems to work well empirically but lacks mathematical justification.\n\nThat is a valid point, and we will add it to our limitations section.\n\n> Why does soft masking match the first $d$ steps better than hard masking?\n\nIn the paper, we mention that “a small $d$ leads to a weak guidance signal and approximation errors can still cause discontinuities” (line 195). What we mean is that the inpainting algorithm is fundamentally a best-effort approximation, and in practice, we found that only using the first $d$ steps as guidance led to a poor match for small $d$. As you pointed out, this is theoretically unsatisfying, but it is an empirical reality.\n\n> Also, because soft masking is encouraging closer match, why wouldn't it end up slowing down task progress?\n\nThe level of matching is a spectrum: from no matching at all, which is so bad that it is unusable on a real robot (see Figure 2, Figure 5, and the supplemental video), to perfectly matching a very long chunk of future actions, which could slow down task progress by preventing closed-loop correction. The right level of matching is an empirical question, but we found that with existing VLA hyperparameters (an action horizon of ~1 second), our soft masking approach with exponential decay performs well.\n\n> Would other types of low level controllers, such as more compliant controllers benefit more or less from this method?\n\nIn theory, a more compliant controller could be more amenable to real-time execution, as the impact of sudden discontinuities between chunks is lessened. However, without RTC, the controller would merely provide a smoothing effect similar to that of temporal ensembling, which we have shown performs very poorly both in simulation and in reality."}, {"review_id": "fMcZU8gvih", "reviewer": "Reviewer_QwTe", "summary": "This paper presents Real-Time Chunking (RTC), a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies for robot control. The key innovation is framing real-time execution as an inpainting problem, where the next action chunk is generated while executing the current one, \"freezing\" actions guaranteed to execute and \"inpainting\" the rest. The method is applicable to any diffusion- or flow-based vision-language-action (VLA) model without retraining. The authors validate RTC on 12 dynamic simulation tasks in Kinetix and 6 real-world bimanual manipulation tasks using the π0.5 VLA, demonstrating improved performance and robustness to inference delays up to 300ms.", "questions": "Can you provide detailed timing breakdowns comparing RTC to standard inference? How does the overhead scale with action dimension, prediction horizon, and number of denoising steps?\n\nWhy do RTC and hard masking perform similarly at d=0 and d=4 in Figure 5? What does this tell us about when soft masking is beneficial?\n\nUnder what specific conditions does RTC fail or perform poorly? Are there particular motion patterns, task characteristics, or environmental factors that challenge the inpainting approach?\n\nWhat are the fundamental barriers to applying RTC to autoregressive VLAs? Have you explored any modifications that could extend the approach beyond diffusion/flow models?", "limitations": "The authors provide a reasonable discussion of limitations in Section 6, acknowledging the architectural constraints and computational overhead. However, the discussion would benefit from more in-depth analysis of computational costs.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\nThis work addresses a practical and timely challenge: how to deploy powerful but slow neural policies in real-time robotic control. As large VLAs and diffusion-based controllers become more common, the latency issue could hinder their real-world use. RTC provides a solution that could be widely adopted to improve responsiveness. \n\nThe inpainting approach is intuitive and well-motivated. The soft masking extension (Section 4.2) that uses exponentially decaying weights for cross-chunk continuity is a clever contribution that improves upon naive hard masking.\n\nThe paper includes both simulated benchmarks (12 dynamic tasks) and real-world experiments (6 bimanual manipulation tasks), showing the effectiveness of the model in multiple scenarios.\n\nThe paper is well-written with effective visualizations that clearly illustrate the problem and solution.\n\n**Weakness:**\n\nWhile the paper acknowledges the computational cost of the inpainting procedure, it lacks detailed analysis. \n\nIn Figure 5, RTC and RTC with hard masking show similar performance when inference delay is zero or high (d=4). Why does soft masking provide no benefit in these cases? This pattern suggests potential limitations of the approach that are not adequately discussed.\n\n The paper provides limited discussion of failure modes or scenarios where RTC might struggle. Understanding when and why the method fails would help practitioners decide when to apply it and guide future improvements.\n\nThe comparison set is relatively narrow. While temporal ensembling and BID are relevant baselines, the paper would benefit from comparisons to other real-time execution strategies such as the hierarchical VLA designs mentioned in the related work.", "was_revised": true, "final_justification": "I am generally satisfied with the authors' response, which has addressed my initial concerns. I am willing to maintain my accept rating to this paper.", "rebuttal": "Thank you for your in-depth review! We have attempted to address your concern about computational costs by providing more details, which we will also add to the Appendix in a future revision (in addition to the existing latency breakdown in the Appendix). We have also provided some clarifications about the experiments. Please let us know if you have any other concerns!\n\n> In Figure 5, RTC and RTC with hard masking show similar performance when inference delay is zero or high (d=4). Why does soft masking provide no benefit in these cases?\n\nWhen $d = 0$, hard masking is equivalent to disabling RTC entirely, which performs well because there is no delay. When $d = 4$, hard masking is equivalent to soft masking, because the size of the region where soft masking would normally apply is $H - s - d = 0$.\n\n> The paper provides limited discussion of failure modes or scenarios where RTC might struggle. Understanding when and why the method fails would help practitioners decide when to apply it and guide future improvements.\n\nAcross our simulated and real tasks, we did not find scenarios where RTC struggles (compared to synchronous baselines), so we would be confident recommending it as a reasonable default strategy.\n\n> Can you provide detailed timing breakdowns comparing RTC to standard inference? How does the overhead scale with action dimension, prediction horizon, and number of denoising steps?\n\nThe RTC overhead comprises an approximate 50% slowdown of each denoising step. Thus, the overhead is linear in the number of denoising steps, independent of action dimension and prediction horizon. The total overhead depends on the model architecture — in Pi05, with 5 denoising steps, the denoising loop is approximately 20% of the total inference time without RTC, and 30% with RTC. We will add these details to the Appendix, as well as a more detailed table breaking down how much latency each component of the Pi05 architecture contributes (with and without RTC).\n\n> What are the fundamental barriers to applying RTC to autoregressive VLAs? Have you explored any modifications that could extend the approach beyond diffusion/flow models?\n\nThe fundamental barrier is the lack of inpainting ability. If the actions were represented in standard chronological order, then one could simply “prompt” the model with the first $d$ frozen actions. However, [1] found that this standard action representation does not work well for high-frequency control, and their solution foils a naive prompting strategy. It is possible that alternative inpainting methods for autoregressive models exist, but we have not explored this direction yet.\n\n[1] “FAST: Efficient Action Tokenization for Vision-Language-Action Models.” Pertsch et al."}]

Author rebuttal:

Meta-review: After carefully considering the mixed reviews and rebuttal, I find the contribution technically sound and impactful. RTC introduces a practical inference-time algorithm for real-time execution of diffusion/flow-based VLA policies, showing strong improvements across both simulation and real-world robotics tasks with robustness to latency. Positive reviews (f8Dc, rbym, QwTe) highlight the method’s timeliness, broad applicability without retraining, and convincing empirical validation. The main negative review (jJyz) raised concerns about framing of “real-time,” insufficient positioning in the control literature, and presentation care, though some initial misunderstandings were later clarified. For the camera-ready, I encourage the authors to: (i) revise the framing of “real-time” and clarify guarantees, (ii) expand discussion of control-related works as Reviewer jJyz emphasized, (iii) provide fuller analysis of computational overhead and trade-offs, and (iv) explicitly discuss limitations, sensitivity to inpainting strategies, and potential failure modes.
Decision: Accept (poster)