Title: Real-Time Execution of Action Chunking Flow Policies
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion policy

flow matching

realtime

real-time

action chunking

VLA

vision-language-action

robotics

control

manipulation
Abstract: Modern AI systems, especially those interacting with the physical world, increasingly require real-time performance. However, the high latency of state-of-the-art generalist models, including recent vision-language-action models (VLAs), poses a significant challenge. While action chunking has enabled temporal consistency in high-frequency control tasks, it does not fully address the latency problem, leading to pauses or out-of-distribution jerky movements at chunk boundaries. This paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. Our method, real-time chunking (RTC), is applicable to any diffusion- or flow-based VLA out of the box with no retraining. It generates the next action chunk while executing the current one, "freezing" actions guaranteed to execute and "inpainting" the rest. To test RTC, we introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate that RTC is fast, performant, and uniquely robust to inference delay, significantly improving task throughput and enabling success in precise tasks --- such as lighting a match --- even in the presence of extreme latency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Real-Time Chunking (RTC), a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies for robot control. The key innovation is framing real-time execution as an inpainting problem, where the next action chunk is generated while executing the current one, \"freezing\" actions guaranteed to execute and \"inpainting\" the rest. The method is applicable to any diffusion- or flow-based vision-language-action (VLA) model without retraining. The authors validate RTC on 12 dynamic simulation tasks in Kinetix and 6 real-world bimanual manipulation tasks using the π0.5 VLA, demonstrating improved performance and robustness to inference delays up to 300ms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\nThis work addresses a practical and timely challenge: how to deploy powerful but slow neural policies in real-time robotic control. As large VLAs and diffusion-based controllers become more common, the latency issue could hinder their real-world use. RTC provides a solution that could be widely adopted to improve responsiveness. \n\nThe inpainting approach is intuitive and well-motivated. The soft masking extension (Section 4.2) that uses exponentially decaying weights for cross-chunk continuity is a clever contribution that improves upon naive hard masking.\n\nThe paper includes both simulated benchmarks (12 dynamic tasks) and real-world experiments (6 bimanual manipulation tasks), showing the effectiveness of the model in multiple scenarios.\n\nThe paper is well-written with effective visualizations that clearly illustrate the problem and solution.\n\n**Weakness:**\n\nWhile the paper acknowledges the computational cost of the inpainting procedure, it lacks detailed analysis. \n\nIn Figure 5, RTC and RTC with hard masking show similar performance when inference delay is zero or high (d=4). Why does soft masking provide no benefit in these cases? This pattern suggests potential limitations of the approach that are not adequately discussed.\n\n The paper provides limited discussion of failure modes or scenarios where RTC might struggle. Understanding when and why the method fails would help practitioners decide when to apply it and guide future improvements.\n\nThe comparison set is relatively narrow. While temporal ensembling and BID are relevant baselines, the paper would benefit from comparisons to other real-time execution strategies such as the hierarchical VLA designs mentioned in the related work.\n\nQUESTIONS:\nCan you provide detailed timing breakdowns comparing RTC to standard inference? How does the overhead scale with action dimension, prediction horizon, and number of denoising steps?\n\nWhy do RTC and hard masking perform similarly at d=0 and d=4 in Figure 5? What does this tell us about when soft masking is beneficial?\n\nUnder what specific conditions does RTC fail or perform poorly? Are there particular motion patterns, task characteristics, or environmental factors that challenge the inpainting approach?\n\nWhat are the fundamental barriers to applying RTC to autoregressive VLAs? Have you explored any modifications that could extend the approach beyond diffusion/flow models?\n\nLIMITATIONS:\nThe authors provide a reasonable discussion of limitations in Section 6, acknowledging the architectural constraints and computational overhead. However, the discussion would benefit from more in-depth analysis of computational costs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper address a practical tension between the unavoidable inference latency of robot foundation models and the desired real-time control on real hardware. \n\n---\n\n## Problem Setting\nThe authors illustrate that for existing VLA models, the first few time steps in the produced action sequence has passed when the inference is done. The deviation in initial condition can cause jerky motions or completely off trajectories depending on the underlying controller.\n\nKey assumptions:\n- Builds from a trained diffusion or flow model that prediction action chunks.\n- Asynchronous execution: the algorithm generates the next action chunk while the current one is still being executed.\n\n---\n\n## Method: Real-Time Chunking (RTC)\n\n- The high level idea is to pose asynchronous action chunking as an inpainting problem.\n- RTC aims to force the first $d$ steps from the new chunk (those executed during the inference delay) to match the corresponding actions from the previous chunk, and then inpaints the rest of the new chunk.\n- This is done by adding a guidance term to the velocity field of the VLA. The guidance term can be computed with back propagation.\n- The authors also propose soft masking: steps past the first $d$ also get exponentially decaying weights to match the previous chunk.\n\n---\n\n## Experiments\n- The authors propose a suite of simulated task using force control. These tasks help highlight the effect of inference delay.\n- In these simulated tasks, RTC is the most robust to inference delay - the task success rate drops the slowest as the inference delay increases. It also performs the best at different execution horizons.\n- In real robot experiments, RTC achieved the highest task throughput across different levels of injected delay. Moreover, unlike the baselines, there is no performance drop as the delay increases.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n- No training required.\n- This work addresses a timely and realistic issue with VLAs. The paper provides clear illustrations and real robot rollouts to motivate the problem.\n- The experiment results are strong, showing that the method is able to avoid performance degradation at 200ms injected delay, more than the typical inference delay.\n\n---\n\n## Weaknesses:\n- Limited to flow or diffusion models.\n- This work applies an existing algorithm from image generation to robotics and falls short on novelty.\n- Innovations like soft masking seems to work well empirically but lacks mathematical justification.\n\nQUESTIONS:\n- Why does soft masking match the first $d$ steps better than hard masking? This is a bit counter-intuitive. Doesn't soft masking add more constraints? Also, because soft masking is encouraging closer match, why wouldn't it end up slowing down task progress?\n- Would other types of low level controllers, such as more compliant controllers benefit more or less from this method?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. The proposed method real-time chunking (RTC) is applicable to any diffusion- or flow-based VLA out of the box with no re-training. To test RTC, the authors introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate favorable performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Timely Contribution to Real-Time Control with Large Models: The application of large models to real-time control is a topic of growing importance in the robot learning community, especially given the inherent latency and computational demands of such models. Addressing this challenge requires effective asynchronous mechanisms. The authors’ proposed method, RTC, is a compelling step in this direction. Figure 1 clearly illustrates RTC’s ability to generate significantly smoother trajectories compared to existing baselines, highlighting its practical benefit. Furthermore, the accompanying demonstration video effectively showcases RTC’s real-time responsiveness and stability in dynamic scenarios.\n\n2. Strong Empirical Validation Across Simulated and Real-World Settings:\nThe authors introduce a new benchmark built on the Kinetix simulator, featuring 12 diverse and highly dynamic tasks spanning both manipulation and locomotion. This benchmark offers a meaningful testbed for evaluating real-time performance under challenging conditions. In addition, the real-world experiments—conducted on 6 complex bimanual manipulation tasks using the π₀.₅ VLA as the base policy—provide strong empirical support for the effectiveness of RTC. Across both domains, the results consistently show that RTC achieves high performance with low latency, underscoring its practical viability for real-time deployment.\n\nWeaknesses:\n1. Clarifying the Broader Impact of Computational Overhead: This work presents a compelling method for enabling real-time control using large models, supported by strong empirical results. However, in the limitations section, the authors primarily acknowledge the added computational overhead introduced by their method. I encourage the authors to further examine and report on whether this overhead affects other critical aspects of system performance—such as latency variability, responsiveness to dynamic changes, or robustness under constrained compute conditions. Understanding the broader implications of this overhead is essential for assessing the method’s practical deployment in resource-limited settings, such as on embedded platforms or mobile robots. One common evaluation scenario is testing robustness to sudden perturbations. It would be valuable to understand how the proposed method performs under such conditions and whether the increased computational cost compromises its responsiveness or stability.\n\n2. Role and Generality of the Inpainting Algorithm: The proposed framework relies heavily on an inpainting algorithm to recover missing observations, which plays a crucial role in enabling effective asynchronous control. In this study, the authors adopt the method by Pokle et al. [46]. However, it remains unclear how sensitive the overall system performance is to this choice. Could alternative inpainting algorithms lead to better, worse, or more efficient outcomes? A comparison or ablation study exploring different inpainting strategies would greatly enhance the generality and reproducibility of the proposed approach. It would also clarify whether the strength of the results stems primarily from the RTC framework itself or is heavily dependent on the chosen inpainting technique.\n\nQUESTIONS:\n1. How robust is the proposed method to sudden perturbations?\n2. How sensitive is the overall system performance to the choice of inpainting algorithm?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper address the problem of how to accelerate the execution of policies encoded using action chunking. The approach uses conditional flow matching, and rather than perform synchronous action chunk generation, where the next action chunk is generated after the previous action chunk completes, the approach generates the next action chunk during the previous action chunk execution, and uses inpainting to ensure that the generated action chunk is consistent with the action chunk being generated. The experimental results show that in some cases the execution of the full policy is faster and is sometimes able to complete the task more reliably.\n\nSTRENGTHS AND WEAKNESSES:\nThe problem, as posed, is fairly clear and relevant, and the paper is reasonably clearly written and the technique is sound.\n\nThe primary difficulties I have with this paper are the problem setting and the experimental evaluation. Firstly, the paper describes the proposed approach as real-time, but real-time has a specific meaning, which is that there is a guaranteed response time. This paper accelerates execution but there is no discussion of a guarantee on response time, or a response time to what. It is not clear it is possible to guarantee that a flow matching technique can be made real time, in that if a given denoising step takes longer than the required response time for whatever reason, the policy cannot return early. This might seem like a quibble to the authors and can probably be addressed by changing the title and framing of the paper, but it is an important point in how the authors take about their work.\n\nThe lack of clarity also shows up in the experimental results. The authors describe a minimum delay of somewhere between 76 and 98ms, plus 25ms of transmission time over the LAN, for a minimum of $d \\approx 6ms$. However, we see in figure 5 that the performance drops substantially up to $d = 4$, and results are not given beyond that. The authors then state \"we would like to understand how the system behaves with higher inference latencies,\" and inject latencies of 100ms and 200ms, which are large (arguably unrealistic?) latencies given the stated properties of the existing system. \n\nThe authors are not especially careful in presenting their results. For instance, \"we find that hard masking somewhat underperforms soft masking, particularly when d is smaller.\" The fact that there are no error bars raises the question of how significant this is, but the authors need to be careful to point out that hard masking *overperforms* soft masking when d is large in the cartpole thrust problem. The text of the paper barely mentions that in both figure 5 and figure 7, we see that no technique is able to solve the problems with a high rate of success (solve rate is not defined. I assume this is the proportion of trials where the task was completed correctly.), and the differences are relatively modest -- naive async is 70%ish vs RTC at about 78 or 79% averaged across sim environments. For the real results, only for dishs_in_sink and shirt_folding do we see what appears to be 100% completion (the 100% line is not labelled). And for dishes_in_sink, only TE seems to achieve completion. \n\nThe approach has some parameters, and it would be helpful to know how sensitive performance is to the choice of parameters (number of denoising steps, faster denoising vs open-loop performance. Equation 5 gives the exponential decay of the weights during in-painting -- if the exponential were encouraged to delay faster, or more slowly, how sensitive is the performance of the model? \n\nThe authors are also not rigorous in the related work in terms of MPC techniques -- the issue of executing a partial control and then generating a new one is a very well-studied problem in control -- there are principled techniques from people like Chris Gerdes who have examined when to to switch to a new control given a partial plan. What happens in this paper when the flow matching returns quickly? This can lead to thrashing, and is not discussed in this paper. \n\nI dislike the term incremental, in that incremental results are important, but the relative modest technical contribution of this paper puts a lot of onus on careful experimental evaluation, which is not present in this paper.\n\nQUESTIONS:\n- Why does the performance drop at d = 4? Why do the results not go to d = 6?\n- What happens in this paper when the flow matching returns quickly?\n- How sensitive is the performance of the model to changes in the exponential weight decay function?\n\nLIMITATIONS:\nThe limitations section is exceedingly short. The authors should be encouraged to put more careful analysis into their technique.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}