Title: FreqPolicy: Frequency Autoregressive Visuomotor Policy with Continuous Tokens
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robot Action Generation

Visuomotor Policy Learning

Frequency-domain Modeling

Continuous Latent Representation
Abstract: Learning effective visuomotor policies for robotic manipulation is challenging, as it requires generating precise actions while maintaining computational efficiency. Existing methods remain unsatisfactory due to inherent limitations in the essential action representation and the basic network architectures. We observe that representing actions in the frequency domain captures the structured nature of motion more effectively: low-frequency components reflect global movement patterns, while high-frequency components encode fine local details. Additionally, robotic manipulation tasks of varying complexity demand different levels of modeling precision across these frequency bands. Motivated by this, we propose a novel paradigm for visuomotor policy learning that progressively models hierarchical frequency components. To further enhance precision, we introduce continuous latent representations that maintain smoothness and continuity in the action space. Extensive experiments across diverse 2D and 3D robotic manipulation benchmarks demonstrate that our approach outperforms existing methods in both accuracy and efficiency, showcasing the potential of a frequency-domain autoregressive framework with continuous tokens for generalized robotic manipulation.Code is available at https://github.com/4DVLab/Freqpolicy


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on an important, often overlooked, aspect of learning robotic policies: *how to represent actions*. Mainly, transforming the actions to the frequency domain provides a certain structure that is effective in identifying global motions (low frequency) and local, fine, motions (high frequency). The proposed action representation and method, FreqPolicy, is composed of hierarchical frequency components (progressively learned), implemented as an autoregressive-diffusion model, and is evaluated on multiple benchmarks resulting in better performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n* Well-motivated: the analysis performed on robotic actions and motions justifies the proposed approach.\n* Extensive benchmark.\n* Faster inference than previous diffusion-based policies.\n* Some generalization capabilities (though the overall performance is not great, i.e. <30).\n* Real-world demonstration.\n* Very detailed appendix.\n* A promise for an open-source code.\n\n**Weaknesses**:\n* I find the clarity of the method lacking and the entirety of Sections 3.4-3.5 hard to follow. I’m confused regarding the stages and the notations. I would suggest adding a proper example to clarify the sampling/generation pipeline. E.g., “Consider a 3-dimensional action space and a trajectory of 5 timesteps. We apply DCT for $k=...$ levels… For this case, $x_i$ is… $y_i$ is … For hierarchical generation, consider $l_i \\in {...}$ and we perform 4 iterations resulting in…”. See “Minor” for more.\n* The reported performance improvement seems marginal compared to DP3/Mamba Policy in Tables 1-3\n* Misleading results: in Section 4.2 - “Inference Efficiency” - the authors mention that the method is faster compared to diffusion policies, but in Figure 4, to outperform DP3, the method requires 3-4 iterations, which has longer inference time than DP3. I would like a clarification for that.\n* The method includes several algorithmic components (e.g., masking, diffusion,...) which might make it a bit complex to implement and tune (though the authors report the hyper-parameters on the appendix).\n* No ablations in the main text (only in the appendix). I recommend writing something about it in the main text and short conclusions/insights.\n* Limitations: very sparse limitations section in the appendix. I can think of some limitations that arise from the complexity involved in synching all the components together (masking and diffusion). I also recommend adding a word or two on this in the main text (e.g., in the Conclusions section).\n\n**Minor**:\n* Introduction - line 44 - you introduce the term “energy” but do not explain its meaning. I also recommend making the caption of Figure 1 a bit more detailed and guide the reader what to look for in the figure (as explained in the main text). In general, this term is used extensively throughout this text and it should be explained in the context of this work or there should be a reference to where it is explained at the very least.\n* Figure 2: “We” -> “It”/”The model” (or change all the verbs, i.e, “transforms”, “learns”, “reconstructs”). Also, maybe add numbers to the steps in the figure so the reader can understand where the pipeline starts.\n* Please add variable dimensions to section 3.3 (e.g., $x \\in \\mathbb{R}^{a \\times b}$). Also, according to the notation prior to Equation 1, shouldn’t $n$ run from 1 to $N$? There is no $x_0$ in your notation.\n* Section 3.5: I’m a bit confused regarding the notations for the index. What is the relationship between $k$ and $l_i$?\n* Line 298: “For all simulation experiments, we use 4 iterations, whereas for real-world experiments, only 1 iteration is used”. Iterations of what? Diffusion? Which variable is that referring to?\n* Table 4: I assume the numbers are success rates? Please explicitly mention thai in the caption/table.\n\nQUESTIONS:\nQuestions:\n* I wrote my questions under “Weaknesses” and “Minor”.\n\nSuggestions:\n* I wrote my suggestions under “Weaknesses” and “Minor”.\n\n\nI'm willing to increase my score given a sufficient response to my concerns (mainly, the clarity issue) and upon reading the other reviews.\n\nLIMITATIONS:\nVery sparse limitations section in the appendix. I can think of some limitations that arise from the complexity involved in synching all the components together (masking and diffusion). I also recommend adding a word or two on this in the main text (e.g., in the Conclusions section).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method, FreqPolicy, aiming to learn motions progressively with respect to different frequency. FreqPolicy uses a conditioned diffusion model to iteratively generate motions at different levels. The authors compare the method on several simulators to prove its efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- I really like the visualization and analysis of section 3.2. The motivation of the paper is clear, and is not a pure combination of different methods and tricks, which is insightful.\n\nWeaknesses:\n\n- The writing of the method section is little confusing. It seems that some information in figure 2 is not presenting in the paper, and it took me a while to understand the whole pipeline.\n\n- Comparing with FAST [26] as a baseline seems to be necessary since the similarity of these two methods (both operate in the frequency domain, both use DCT, both try to decompose the actions, but FAST is discrete while FreqPolicy is continuous).\n\nIn general, I like this paper, and I'm willing to raise the score if the authors can solve the problems.\n\nQUESTIONS:\n1.\tI think the major problem is the lack of comparison with FAST. FAST also use DCT and decompose the action sequence in the frequency domain. Both the motivation and the method part is very similar. The authors claim that FAST leads to “significant information loss” (L70). It would be definitely helpful if the authors can compare the FAST as a baseline on success rates, reconstruction loss on evaluation set, etc.\n\n2.\tFor the robomimic benchmark (L166), can you explain the composition of 10 degrees of freedom for robot arm with parallel grippers?\n\n3.\tFor figure 3(b), what is the x-axis (frequency domain scale)? For frequency domain scale = $1.00$, why the success rates for most of the tasks are still not close to 1.0? How lossy is the reconstruction?\n\n4.\tHow to represent $y_k$ in the discrete space?\n\n5.\tFor the experiment results Figure 4, it seems that more iteration (>=4) will lead to a fluctuated success rate (4 > 6 > 7 > 5, etc). If would be better if the author can shed more lights on this. For instance, showing the reconstruction loss of different iterations, etc., will be helpful.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FreqPolicy, a novel visuomotor policy learning framework that represents robot actions in the frequency domain and generates continuous-action trajectories via a hierarchical, autoregressive diffusion model. By decomposing trajectories into low-to-high frequency components using DCT and mapping them into continuous latent tokens, FreqPolicy progressively refines coarse global motions into fine local details. Extensive simulation results on Adroit, DexArt, Meta-World and RoboTwin benchmarks demonstrate state-of-the-art success rates and a favorable inference–performance trade-off.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Frequency-Domain Representation: Modeling actions via DCT naturally separates global versus fine-grained motion, yielding smoother low-frequency priors and targeted high-frequency refinements—a clear conceptual advance over purely time-domain AR or diffusion approaches.\n\n2. Rich Simulation Evaluation: The authors benchmark on 48 simulation tasks across multiple datasets, outperforming strong baselines (e.g., DP3, Mamba Policy) on 8 of 10 Adroit tasks and averaging 67.9 % success versus 65.8 % from competing methods.\n\n3. Efficiency & Real-Time Potential: FreqPolicy’s single-iteration inference runs at 0.21 s per trajectory (≈ 70 FPS) and demonstrates a qualitative handover on real hardware, highlighting its suitability for latency-sensitive control.\n\nWeaknesses:\n\n1. Lack of Quantitative Real-Robot Metrics: The “real-world” section reports only inference speed and a qualitative handover snapshot, without reporting success rates or robustness on the physical platform.\n\n2. Missing Baseline with Frequency-Tokenizers: Despite the frequency focus, there is no comparison against other frequency-aware action tokenizers such as FAST + DP or DP3 variants that explicitly compress frequency components.\n\n3. Marginal Success-Rate Improvements: Although FreqPolicy outperforms baselines, the absolute gains in success rate are modest (e.g., from 65.8 % to 67.9 % on average), raising questions about the practical significance of the frequency-domain representation.\n\nQUESTIONS:\n1. Can you provide concrete success‐rate and robustness metrics from your “real-world” experiments rather than only qualitative snapshots?\n\n2. How does FreqPolicy compare against other frequency‐aware action tokenizers such as FAST+DP or DP3 on your core benchmarks?\n\n3. Because of marginal success‐rate gains, what additional evaluation metrics or experiments (e.g., trajectory smoothness in the frequency spectrum, robustness to noise, sample efficiency, or control bandwidth analysis) would you recommend to more convincingly validate the benefits of a frequency-domain approach?\n\nLIMITATIONS:\nPlease see the weaknesses. If you add more experiments to solve my concerns, I would like to raise my scores.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes FreqPolicy, a new approach to teaching robots how to move by analyzing actions through their frequency components. The key insight is that robotic movements have natural structure - broad, overall patterns appear in low frequencies while precise, detailed movements show up in high frequencies.\nThe method works by breaking down actions into these frequency layers and learning them step-by-step in a hierarchical manner. Instead of forcing actions into discrete categories, FreqPolicy employs smooth, continuous representations, combined with a diffusion-based system, to generate more natural and precise movements.\nThe main innovations include creating a frequency-based framework that captures the inherent structure of how robots move, developing a continuous representation system that avoids the limitations of categorizing actions into fixed bins, and demonstrating superior performance on robot manipulation tasks with better accuracy and faster computation compared to existing approaches.\nEssentially, by considering robot actions like how we think about music or signals, with different frequency components conveying distinct types of information, this method achieves more effective and efficient robot learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n  * **Originality:** FreqPolicy introduces a novel and well-motivated approach by representing actions in the frequency domain, addressing limitations of existing methods. The progressive modeling with continuous latent representations is a unique contribution.\n  * **Significance:** It significantly advances robotic manipulation by achieving state-of-the-art performance and faster inference, making it highly practical for real-time applications. The analysis of task-dependent frequency requirements is a valuable insight.\n  * **Quality:** Extensive experiments across diverse benchmarks consistently demonstrate superior accuracy and computational efficiency. The frequency domain analysis empirically justifies the core idea.\n  * **Clarity:** The paper is generally well-written, clearly explaining the problem, method, and results with effective figures.\n\n**Weaknesses:**\n\n  * **Clarity - Specifics on Diffusion-Based Decoding:** More detail on the diffusion-based decoding and its integration in the main text would improve understanding.\n  * **Clarity - Masked Generative Strategy:** A slightly more detailed explanation of its benefits (cost reduction, diversity) beyond mask ratios is needed.\n  * **Quality - Ablation on Frequency Levels:** The paper lacks explicit discussion or ablation studies on the selection and progression of frequency levels for autoregressive iterations.\n  * **Limitations Section:** A dedicated limitations section in the main paper would be beneficial for critical self-reflection.\n\nQUESTIONS:\n1.  How does FreqPolicy handle the trade-off between computational efficiency and action precision, especially given that high-frequency components encode fine local details?\n2.  What are the specific mechanisms or architectural components within FreqPolicy that enable the \"coarse-to-fine\" generation process for hierarchical frequency components?\n3.  The paper mentions real-world experiments with only 1 iteration for efficiency. How does this single-iteration performance compare to higher iteration counts in simulated environments, and what are the implications for deploying FreqPolicy in other real-time robotic applications?\n\nLIMITATIONS:\nPlease check the above weakness section", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}