Fast Visuomotor Policy for Robotic Manipulation

## Abstract

Abstract We present a fast and effective policy framework for robotic manipulation, named Energy Policy , designed for high-frequency robotic tasks and resource-constrained systems. Unlike existing robotic policies, Energy Policy natively predicts multimodal actions in a single forward pass, enabling high-precision manipulation at high speed.
The framework is built upon two core components. First, we adopt the energy score as the learning objective to facilitate multimodal action modeling. Second, we introduce an energy MLP to implement the proposed objective while keeping the architecture simple and efficient.
We conduct comprehensive experiments in both simulated environments and real-world robotic tasks to evaluate the effectiveness of Energy Policy. The results show that Energy Policy matches or surpasses the performance of state-of-the-art manipulation methods while significantly reducing computational overhead. Notably, on the MimicGen benchmark, Energy Policy achieves superior performance with at a faster inference compared to existing approaches.

## 1 Introduction

Policy learning from demonstrations has emerged as a powerful paradigm for enabling robots to acquire complex skills. It is typically formulated as a supervised regression task, where observations are mapped to actions. To achieve high-precision action regression, incorporating generative models into policy learning has become a dominant approach across various robotic tasks.

Recent research has aimed to enhance policy learning by introducing various generative modeling techniques. Adopting Autoregressive Modeling (AM) from large language models (kim2024openvla,; brohan2022rt,; brohan2023rt,; qu2025spatialvla,) has proven to be a powerful solution, owing to its scalability, flexibility, and mature exploration. However, like language models, AM uses discrete action tokens for action prediction, which can sacrifice fine-grained action details. Recently, there has been considerable research into continuous action representations. While directly applying L1 or L2 regression (kim2025fine,; su2025dense,) to continuous action prediction offers a straightforward way to improve action precision, it struggles with multimodal action distributions due to its uni-modal modeling approach. Diffusion Modeling (DM) (chi2023diffusion,; ze20243d,; wang2024sparse,; team2024octo,; black2024pi_0,; intelligence2025pi_,) provides a promising alternative by learning multimodal distributions through modeling the gradient of the action score function. However, DM requires multiple denoising steps, making it computationally prohibitive for real-time robotic tasks.

In this paper, we propose a novel and efficient approach to policy learning that natively predicts multimodal continuous actions in a single forward pass. Specifically, during training, we utilize energy score (brier1950verification,; gneiting2007strictly,) as the learning objective to minimize the distributional difference between the predicted actions and the ground truth. The energy score provides a rigorous measure of whether predictions match the underlying distribution, making it a natural choice for multimodal action modeling. To fully exploit this objective, we propose an energy MLP, a dedicated module that explicitly parameterizes energy score modeling. This design is central to our method, as it enhances representational expressiveness, allowing the energy score to serve as an effective supervisory signal for complex multimodal distributions. During inference, we can directly sample continuous actions from the model’s distribution prediction, avoiding the need for multiple forward passes as in Diffusion Modeling. In addition, we incorporate parallel decoding, which generates all actions simultaneously and enables efficient action chunking (zhao2023learning,).

We conduct extensive experiments to demonstrate the effectiveness of our proposed method. Across a range of simulated robotic manipulation benchmarks, such as Robomimic (mandlekar2021matters,) and MimicGen (mandlekar2023mimicgen,), our method achieves high task success rates and fast inference speeds. Notably, it outperforms CARP (gong2024carp,) across all benchmarks, with a $2.3\times\sim 7\times$ faster inference speed, and further surpasses existing efficient policies on the PushT task. We also evaluate our approach on real-world tasks under compute-constrained conditions. Compared to baseline methods, our method exhibits a higher success rate and faster inference, underscoring its suitability for real-time robotic applications.

In summary, our contributions are as follows: First, we present a novel approach to policy learning that models multimodal continuous actions. Second, the proposed method offers faster inference speeds, making it suitable for real-time robotic tasks. Third, extensive experiments validate the effectiveness of our method in both simulated and real-world robotic manipulation tasks.

## 2 Related Work

##### Learning Robotic Manipulation from Demonstrations.

Imitation learning enables robots to learn to perform tasks demonstrated by experts. Recently, there are various approaches to be developed for policy learning with different task constraints and control modalities. Autoregressive Modeling (AM) (kim2024openvla,; brohan2022rt,; brohan2023rt,; qu2025spatialvla,) provides next-token prediction paradigm and use discrete action representation for manipulation learning. RT2 (brohan2023rt,) takes language instructions and visual observations as input, and outputs discrete action tokens in an auto-regressive manner. For high precision manipulation, enormous works (kim2025fine,; su2025dense,; chi2023diffusion,; ze20243d,; wang2024sparse,; team2024octo,; black2024pi_0,; intelligence2025pi_,) explore continuous action representations.  (kim2025fine,; su2025dense,) applies L1 or L2 objectives to learn to predict continuous action. However, these methods struggle with multimodal action distributions due to their uni-modal nature. Diffusion Policy (chi2023diffusion,) is proposed to handle multimodal action distributions by adopting a conditional denoising diffusion process, which involves multiple denoising steps. Unlike existing works, our method employs energy score to learn to predict continuous multimodal action.

##### Fast Visuomotor Policy Learning.

In addition to manipulation precision, inference speed is another critical aspect of robotic policy. For AM-based models, integrating the KV-Cache technique can significantly enhance inference speed. Recent work such as Fast (pertsch2025fast,) introduces compressed action tokens to further improve runtime, while CARP (gong2024carp,) employs a next-scale autoregressive paradigm to shorten prediction horizons. Diffusion-based approaches typically rely on action chunking (zhao2023learning,) to achieve higher action throughput, and can leverage distillation techniques to reduce denoising steps (prasad2024consistency,; wang2024one,), though often at the cost of action accuracy. Unlike these diffusion-based pipelines and their distilled variants, our approach natively predicts continuous actions in a single forward pass. This fundamental distinction eliminates the need for iterative refinement and avoids accuracy-compromising distillation, enabling our approach to achieve both high precision and fast inference simultaneously.

## 3 Method

In this section, we start by focusing on preliminaries, including problem formulation and existing works. Then we propose an energy-based learning objective to avoid these limitations. Finally, we demonstrate the details about network architecture to implement our method.

Figure: Figure 2: Comparisons of existing Polies. (a) Autoregressive policy predicts discrete tokens in an autoregressive manner. (b) L1/L2 policy predicts continuous actions but struggles with multimodal distribution modeling. (c) Diffusion policy generates multimodal continuous actions through multiple denoising steps. (d) Our Energy Policy produces multimodal continuous actions in a single forward pass.
Refer to caption: x2.png

### 3.1 Preliminaries

##### Problem Formulation.

For a task $\mathcal{T}$, there are $N$ expert demonstrations $\{\pi_{i}\}_{i=1}^{N}$. Each demonstration $\pi_{i}$ consists of a sequence of state-action pairs $\{o_{t},a_{t}\}_{t=1}^{T}$, where $a_{t}$ denotes the action, $o_{t}$ represents the observation, $T$ is the action sequence length. We formulate robot policy learning as an action sequence prediction problem. The aim is to train a model to minimize the error in future actions conditioned on historical states. Specifically, policy learning minimize the imitation learning loss $\mathcal{L}_{im}$ formulated as

$$ $\mathcal{L}_{im}=\mathbb{E}_{\pi_{i}\sim\mathcal{T}}\left[\sum_{t=0}^{T}\mathcal{L}\left(f_{\theta}(a_{t:t+H-1}|o_{t^{\prime}<t}),a_{t:t+H-1}\right)\right]$ (1) $$

where $H$ is the prediction horizon, $t$ and $t’$ denote the current and previous time step, respectively. $\mathcal{L}$ represents a supervised action prediction loss, and $\theta$ represents the learnable parameters of the policy network $f_{\theta}$. Based on above problem formulation, the existing works (see Figure [2](https://arxiv.org/html/2510.12483v1#S3.F2)) mainly differ in how the $\mathcal{L}$ and $H$ are defiend, discussed as follows.

##### Autoressive Policy.

By default, autoregressive-based policies predict action sequences in an autoregressive manner, with $H$ set to 1. Additionally, due to the discrete nature of action tokens, cross-entropy loss is typically used as the default objective $\mathcal{L}$. However, using discrete action tokens often sacrifices fine-grained action details, making it challenging for robotic tasks that require high-precision control.

##### L1/L2 Policy.

To circumvent the use of discrete action tokens, L1/L2 policies have been proposed. By using L1/L2 loss as the objective $\mathcal{L}$, these methods can predict continuous actions. This makes the learned policy well-suited for high-precision manipulation tasks. However, it struggles with multimodal action distributions due to its unimodal modeling characteristics.

##### Diffusion Policy.

In diffusion-based policies, action prediction is modeled as a denoising process, which makes it easier to handle multimodal action distributions. It uses denoising loss as the objective $\mathcal{L}$ and predicts a sequence of actions with $H>1$ simultaneously. However, diffusion-based policies suffer from the need for multiple denoising steps, making them less suitable for high-frequency robotic tasks.

Figure: Figure 3: Overview of Energy Policy. (a) The architecture of the energy policy primarily consists of a transformer decoder and an energy MLP. The transformer decoder takes observations and learnable action tokens as input, producing a sequence of vectors $\{z_{t}\}_{t=1}^{H}$. The energy MLP then predicts the action sequence $\{\hat{a}_{t}\}_{t=1}^{H}$ by taking $\{z_{t}\}_{t=1}^{H}$ as input, conditioned on noise samples. (b) The energy loss is computed based on two sampled actions, $\hat{a}_{t}^{1}$ and $\hat{a}_{t}^{2}$, along with the ground-truth action $a_{t}$. These two action samples are generated from the same $z_{t}$ using different noise inputs. (c) The energy MLP is composed of several residual blocks, each incorporating adaLN for noise injection and modulation.
Refer to caption: x3.png

### 3.2 Energy Policy

To address the limitations of existing approaches, we propose an energy-based policy that uses energy scores as the learning objective.
With energy scores, the model can learn multimodal action distributions during training.
During inference, continuous actions can be sampled in a single forward pass.
Additionally, we introduce a decoder-only transformer specifically designed for energy policy.

#### 3.2.1 Energy Loss

Consider two probability distributions $p$ and $q$ in $\mathbb{R}^{d}$.
A scoring rule is a function $S(p,y)$ that assigns a score to the distribution $p$ based on the observed data $y\sim q$ (Gneiting2007StrictlyPS,).
The expected score $S(p,q)$ is defined as $S(p,q):=\mathbb{E}_{y\sim q}[S(p,y)]$.
A scoring rule is called strictly proper if the expected score is minimized if and only if $p=q$.

A commonly used class of strictly proper scoring rules is the energy scores (szekely03estatistics,), defined as

$$ $S(p,y)=-\mathbb{E}[\|x_{1}-x_{2}\|^{\alpha}]+2\mathbb{E}[\|x-y\|^{\alpha}]$ (2) $$

where $x_{1},x_{2},x\in\mathbb{R}^{d}$ are independent samples drawn from the model distribution $p$, and $\|\cdot\|$ denotes the Euclidean norm.
By allowing for an index $\alpha\in(0,2)$, $S(p,q)$ is minimized if and only if $p=q$, which implies that the model distribution is consistent with the data distribution.

For action prediction, the model takes an observation as input and outputs a prediction distribution $p$ for the action $a_{t}$.
To obtain an unbiased estimate of the energy score $S(p,q)$, two independent samples, $\hat{a}_{t}^{1}$ and $\hat{a}_{t}^{2}$, are drawn from the model distribution $p$.
The energy loss for action prediction is then defined as:

$$ $\mathcal{L}(p,a_{t})=\|\hat{a}_{t}^{1}-a_{t}\|^{\alpha}+\|\hat{a}_{t}^{2}-a_{t}\|^{\alpha}-\|\hat{a}_{t}^{1}-\hat{a}_{t}^{2}\|^{\alpha}$ (3) $$

This loss objective incentivizes the model to generate samples close to the target action, while maintaining the diversity between independent samples.
To implement the energy loss, we introduce an energy-based MLP within the transformer architecture.

#### 3.2.2 Transformer with Energy MLP

As shown in Figure [3](https://arxiv.org/html/2510.12483v1#S3.F3), our model takes observations and learnable action tokens as input. These input tokens are also combined with learnable position tokens. The decoder-only transformer then generates a sequence of vectors $\{z_{t}\}_{t=1}^{H}$, each corresponding to a learnable action token. For each vector $z_{t}$, the corresponding continuous ground-truth action is denoted as $a_{t}$.

Given ground-truth action $a_{t}$, we introduce a dedicated energy MLP specifically designed for the energy loss. This network takes $z_{t}$ as input, together with two random noise samples drawn from a uniform distribution, and outputs two candidate actions $\hat{a}_{t}^{1}$ and $\hat{a}_{t}^{2}$. The energy MLP consists of several residual blocks (he2016deep,). Each block sequentially applies LayerNorm (LN) (ba2016layer,), a linear layer, SiLU (elfwing2018sigmoid,), another linear layer, and a residual connection. To inject stochasticity, we adopt adaLN-Zero blocks (peebles2022scalable,), which condition on noise inputs and perturb the predictions $z_{t}$ through shift, scale, and gate operations. This design enables effective conditioning on noise, which not only introduces richer stochasticity but also enhances the model’s expressive power.

At inference time, this energy MLP takes $z_{t}$ and a single noise sample to directly predict the corresponding continuous action, ensuring consistency between training and deployment while avoiding iterative refinement.

## 4 Evaluation

We conduct a comprehensive evaluation of our method across a diverse set of robotic tasks in both simulated and real-world environments. These tasks include both single-task and multi-task settings, utilizing image-based and state-based observations. We compare our approach against several state-of-the-art baselines, including both diffusion-based and auto-regressive methods. The evaluation considers key metrics such as success rate, inference time, and model size. Our experimental study aims to address the following research questions:

- •
How does our method compare to state-of-the-art approaches in terms of inference speed and task success rate?
- •
How well does our method perform robotic tasks in real-world environments?
- •
Can our energy loss effectively learn the multimodal action distribution?

### 4.1 Simulation Environment

We evaluate our method across diverse settings, including single-object manipulation, long-horizon planning, high-precision control, and dual-arm coordination, using Robomimic (mandlekar2021matters,), Franka Kitchen (gupta2019frank,), MimicGen (mandlekar2023mimicgen,), and PushT (chi2023diffusion,). Unless otherwise specified, the models are trained for 400 epochs with a batch size of 1024 and $\alpha=1.0$, using an energy MLP head with a depth of 3 and a width of 512. At inference, the model predicts an action sequence of length 16, from which the first 8 actions are executed. Baseline models are trained and evaluated following their original protocols. Performance is reported as the average success rate over the best three checkpoints. Inference efficiency is measured on an NVIDIA RTX 4090 GPU by averaging the runtime for generating 8 executable actions, with each test repeated three times.

#### 4.1.1 Single Object Manipulation on Robomimc

##### Setup.

Robomimic offers a diverse set of tasks; in this section, we focus on three representative single-object manipulation tasks: Lift, Can, and Square. For each task, we use 200 expert demonstrations collected via teleoperation in simulation and evaluate performance under two observation modalities: image-based (RGB inputs from eye-in-hand and third-person views) and state-based (low-dimensional privileged information). We compare against three baselines: Implicit Behavior Cloning (IBC) (Florence2021ibc,), Diffusion Policy (DP-C, DP-T) (chi2023diffusion,), and CARP (gong2024carp,).

##### Result.

As shown in Table [1](https://arxiv.org/html/2510.12483v1#S4.T1), our model matches or surpasses the baselines in both state-based and image-based settings, while also delivering substantially faster inference. Specifically, our approach is $3.7\times\sim 7.0\times$ faster than the autoregressive baseline (CARP) and $22.3\times\sim 70.0\times$ faster than the diffusion-based baselines (DP-C and DP-T). Although achieving a low latency, IBC exhibits near-zero success rates on several tasks. In contrast, our method maintains high success rates comparable to the strongest baselines while eliminating the costly iterative sampling steps of diffusion. This demonstrates both efficiency and robustness across single-object manipulation tasks.

**Table 1: Comparison of policy performance and efficiency across state-based and image-based Robomimic (mandlekar2021matters,) tasks.**
| Policy | State-based | Image-based |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Lift-ph | Can-ph | Square-ph | Params(M) | Speed(s) | Lift-ph | Can-ph | Square-ph | Params(M) | Speed(s) |  |
| IBC (Florence2021ibc,) | 0.79 | 0.00 | 0.00 | 3.20 | 0.03 | 0.94 | 0.08 | 0.03 | 3.44 | 0.10 |
| DP-C (chi2023diffusion,) | 1.00 | 0.94 | 0.94 | 65.88 | 0.70 | 1.00 | 0.97 | 0.92 | 255.61 | 0.72 |
| DP-T (chi2023diffusion,) | 1.00 | 1.00 | 0.88 | 8.97 | 0.64 | 1.00 | 0.98 | 0.86 | 9.01 | 0.67 |
| CARP (gong2024carp,) | 1.00 | 1.00 | 0.98 | 0.65 | 0.07 | 1.00 | 0.98 | 0.88 | 7.58 | 0.11 |
| Ours | 1.00 | 1.00 | 0.97 | 0.73 | 0.01 | 1.00 | 0.98 | 0.95 | 11.51 | 0.03 |

#### 4.1.2 Long Horizon Planning on Franka Kitchen

**Table 2: Performance and efficiency of different policies on Franka-Kitchen (gupta2019frank,) tasks.**
| Policy | p1 | p2 | p3 | p4 | Params(M) | Speed(s) |
| --- | --- | --- | --- | --- | --- | --- |
| IBC (Florence2021ibc,) | 0.99 | 0.87 | 0.61 | 0.24 | 3.28 | 0.05 |
| DP-C (chi2023diffusion,) | 1.00 | 1.00 | 1.00 | 0.96 | 66.94 | 0.91 |
| DP-T (chi2023diffusion,) | 1.00 | 0.99 | 0.98 | 0.96 | 80.42 | 0.84 |
| CARP (gong2024carp,) | 1.00 | 1.00 | 0.98 | 0.98 | 3.88 | 0.08 |
| Ours | 1.00 | 1.00 | 1.00 | 0.96 | 5.06 | 0.02 |

##### Setup.

To assess long-horizon, multi-task learning, we evaluate on the Franka Kitchen environment, which involves interaction with seven objects and 566 human demonstrations, each completing four tasks in arbitrary order. Only state-based inputs are used. We compare against three baselines: Implicit Behavior Cloning (IBC) (Florence2021ibc,), Diffusion Policy (DP-C, DP-T) (chi2023diffusion,), and CARP (gong2024carp,).

##### Result.

As shown in Table [2](https://arxiv.org/html/2510.12483v1#S4.T2), our model attains success rates that are comparable to or exceed those of the baselines across all tasks, while also maintaining the fastest inference speed. Specifically, our approach runs $4\times$ faster than the autoregressive baseline (CARP) and $42\times\sim 45\times$ faster than the diffusion-based baselines (DP-C and DP-T). These results highlight the efficiency and scalability of our design in complex, long-horizon environments.

#### 4.1.3 Multi-task Learning on MimicGen

##### Setup.

We evaluate our model on MimicGen (mandlekar2023mimicgen,), a large-scale imitation learning benchmark, which extends Robomimic (mandlekar2021matters,) by providing 1K–10K demonstrations per task and broader initial state distributions. MimicGen comprises 12 MuJoCo (todorov2012mujoco,)-based robosuite tasks and 4 high-precision tasks from Isaac Gym Factory (makoviychuk2021isaac,). Following prior work (Wang2024sdp,; gong2024carp,), we evaluate on 8 robosuite tasks, each with 1K demonstrations. The baselines include two diffusion-based multitask models—Task-Conditioned Diffusion (TCD) (liang2023tcd,) and Sparse Diffusion Policy (SDP) (Wang2024sdp,), the latter employing a Transformer with a Mixture of Experts (MoE) (shazeer2017moe,)—as well as the autoregressive multitask variant of CARP (gong2024carp,). All models are conditioned on task labels to enable multitask learning.

##### Result.

As shown in Table [3](https://arxiv.org/html/2510.12483v1#S4.T3), our method achieves substantial improvements over the diffusion baseline, with about a 10% higher success rate and at least a $16.6\times$ faster inference. Moreover, it matches the success rate of the autoregressive baseline while delivering a $2.3\times$ faster inference. These results further support our first research question.

**Table 3: Performance and efficiency comparison of different policies across 8 manipulation tasks in MimicGen (Chi2024umi,).**
| Policy | Coffee | Hammer | Mug | Nut | Square | Stack | Stack 3 | Threading | Avg. | Params (M) | Speed (s) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TCD (liang2023tcd,) | 0.77 | 0.92 | 0.53 | 0.44 | 0.63 | 0.95 | 0.62 | 0.56 | 0.68 | 156.11 | 1.33 |
| SDP (Wang2024sdp,) | 0.82 | 1.00 | 0.62 | 0.54 | 0.82 | 0.96 | 0.80 | 0.70 | 0.78 | 159.85 | 1.53 |
| CARP (gong2024carp,) | 0.86 | 0.98 | 0.74 | 0.78 | 0.90 | 1.00 | 0.82 | 0.70 | 0.85 | 16.08 | 0.18 |
| Ours | 0.89 | 0.98 | 0.69 | 0.86 | 0.89 | 0.99 | 0.75 | 0.79 | 0.86 | 17.85 | 0.08 |

#### 4.1.4 Comparison with Other Efficient Policies

##### Setup.

To further evaluate the performance of our method, we compare it against recent single-step robotic policies on more challenging tasks, including PushT, Square-mh, Square-ph, ToolHang-ph, Transport-mh, and Transport-ph, which challenge the model’s ability to learn dual-arm and high-precision manipulation. The baselines include One-step Diffusion Policy (OneDP) (wang2024one,), IMLE Policy (IMLE) (rana2025imle,), Consistency Policy (CP) (prasad2024consistency,), and VQ-BeT (lee2024behavior,). These methods are designed to improve the efficiency of robotic policy inference by eliminating or reducing reliance on multi-step sampling procedures.

##### Result.

As summarized in Table [4](https://arxiv.org/html/2510.12483v1#S4.T4), our Energy Policy consistently outperforms all baselines in average success rate across diverse tasks. It achieves an average success rate of $0.87$, clearly surpassing OneDP, IMLE, CP, and VQ-BeT, while maintaining competitive inference speed. Although VQ-BeT attains the lowest latency, this comes at the cost of a substantial drop in task performance, underscoring the advantage of our approach in balancing efficiency and effectiveness.

**Table 4: Comparison of different efficient policies across tasks. * For fair comparisons, we report the model parameter counts and inference speeds on the PushT task using an NVIDIA RTX 4090 GPU.**
| Policy | PushT | Square-mh | Square-ph | Toolhang-ph | Transport-mh | Transport-ph | Avg. | Params(M)* | Speed(ms)* |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ours | 0.85 | 0.85 | 0.95 | 0.92 | 0.70 | 0.95 | 0.87 | 7.73 | 7.02 |
| OneDP-S (wang2024one,) | 0.82 | 0.86 | 0.93 | 0.85 | 0.69 | 0.91 | 0.84 | 251.51 | 9.33 |
| IMLE (rana2025imle,) | 0.59 | – | 0.82 | 0.81 | – | 0.90 | – | 75.75 | 7.63 |
| CP (prasad2024consistency,) | 0.82 | – | 0.92 | 0.70 | – | – | – | 255.18 | 15.23 |
| VQ-BeT (lee2024behavior,) | 0.68 | – | – | – | – | – | – | 4.37 | 4.09 |

Figure: Figure 4: Real-World Experiment Setup. (a) Left: equipment and objects used in the static tasks, with each item labeled by a circled number in the upper-right corner. Middle: desired manipulation behavior for Task Rabbit. Right: desired manipulation behavior for Task Cup . (b) Left: equipment and objects used in the dynamic task, with each item labeled by a circled number in the upper-right corner. Right: desired manipulation behavior for Task Catch.
Refer to caption: x4.png

### 4.2 Real-World Environments

We evaluate our model on two robotic platforms to assess real-world manipulation performance. To systematically test different capabilities, we design three tasks: two static tasks (Cup and Rabbit) that assess precision and target selection under distraction, and one dynamic task (Catch) that evaluates real-time interaction. As the baseline, we use the CNN-based Diffusion Policy (chi2023diffusion,). Each model is evaluated from a single checkpoint, with success measured as the number of successful trials out of 20 per task. For speed evaluation, both models are deployed on the same machine with an NVIDIA RTX 1080Ti GPU, and we report the average inference time for generating 8 executable action steps.

#### 4.2.1 Static Tasks

Setup. As shown in Figure [4](https://arxiv.org/html/2510.12483v1#S4.F4) (a, left), an Emergen CR3 robotic arm^\raisebox{-0.9pt}{1}⃝ is equipped with a 3D-printed Universal Manipulation Interface (UMI) gripper^\raisebox{-0.9pt}{3}⃝ (Chi2024umi,) and a wrist-mounted GoPro Hero 9 camera^\raisebox{-0.9pt}{2}⃝ with a fisheye lens. we collect 150 human demonstration trajectories per task using the UMI gripper^\raisebox{-0.9pt}{8}⃝. As illustrated in Figure [4](https://arxiv.org/html/2510.12483v1#S4.F4) (a, center and right), we design the following real-world tasks: Cup. The workspace contains a green cup^\raisebox{-0.9pt}{4}⃝ and a red block^\raisebox{-0.9pt}{6}⃝ on a tabletop. The robot must grasp the block and place it inside the cup. While the cup remains fixed, the block’s initial position is perturbed to introduce variability. This task evaluates precise pick-and-place under positional uncertainty. Rabbit. The workspace includes a soft rabbit toy^\raisebox{-0.9pt}{5}⃝, a red rectangular block^\raisebox{-0.9pt}{6}⃝, and a blue triangular block^\raisebox{-0.9pt}{7}⃝, which are randomly assigned to three predefined locations with small perturbations. The robot must identify and grasp the rabbit among distractors, testing the robustness to spatial variation and target ambiguity. As the baseline, we use the CNN-based Diffusion Policy (DP-UMI) (Chi2024umi,).

**Table 5: Comparison of policy performance and efficiency across real-world tasks.**
| Static Tasks | Dynamic Task |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Policy | Cup | Rabbit | Params(M) | Speed(s) | Policy | Catch | Params(M) | Speed(s) |
| DP-UMI (Chi2024umi,) | 16/20 | 19/20 | 85.47 | 0.34 | DP-C (chi2023diffusion,) | 8/20 | 64.87 | 0.10 |
| Ours | 17/20 | 20/20 | 68.65 | 0.10 | Ours | 13/20 | 11.50 | 0.02 |

##### Results.

As shown in Table [5](https://arxiv.org/html/2510.12483v1#S4.T5), our model achieves a slightly higher success rate than the baseline, while maintaining a $3.4\times$ inference speedup. Although the perturbations introduced during evaluation are relatively small, DP-UMI (Chi2024umi,) occasionally fails, primarily due to slight gripping inaccuracies when the state deviates from the training distribution. In contrast, Energy Policy executes more reliably under the same conditions, underscoring its robustness in real-world robotic tasks.

#### 4.2.2 Dynamic Task

Setup. As shown in Figure [4](https://arxiv.org/html/2510.12483v1#S4.F4) (b, left), a UR5 robotic arm^\raisebox{-0.9pt}{1}⃝ is equipped with a Robotiq 2F-85 adaptive gripper^\raisebox{-0.9pt}{3}⃝ and a wrist-mounted Intel RealSense D435i camera^\raisebox{-0.9pt}{2}⃝. An additional RealSense D435i camera^\raisebox{-0.9pt}{6}⃝ mounted on a fixed stand provides a third-person view. We collect 100 teleoperated demonstrations using a 3D SpaceMouse. As illustrated in Figure [4](https://arxiv.org/html/2510.12483v1#S4.F4) (b, right), we design the following task: Catch. The workspace contains a red block^\raisebox{-0.9pt}{5}⃝ attached to a string and a ring^\raisebox{-0.9pt}{4}⃝ held by the gripper. A human drags the block across the table at random speeds from right to left, and the robot must intercept it with the ring before it exits the camera’s field of view. This task challenges the policy’s inference speed and its ability to interact with fast-moving objects in real time. As the baseline, we use the CNN-based Diffusion Policy (DP-C) (chi2023diffusion,).

##### Results.

As shown in Table [5](https://arxiv.org/html/2510.12483v1#S4.T5), our model outperforms the CNN-based Diffusion Policy on the dynamic Catch task (13/20 vs. 8/20) with a $5\times$ faster inference speed (0.02s vs. 0.10s). The performance gap is more pronounced than in static tasks, since multi-step diffusion models struggle with latency. In contrast, our low-latency predictions enable reliable performance under time-sensitive conditions.

### 4.3 Modeling Multi-Modal Behavior

To evaluate the model’s ability to capture multimodal behavior, we design an initial state in the PushT environment where the task can be successfully completed by moving either left or right. We then sample 50 trajectories from the Energy Policy to examine its ability to model multi-modal behavior. The visualizations are shown in Figure [5](https://arxiv.org/html/2510.12483v1#S4.F5), Energy Policy generates smooth trajectories covering both modes.

Figure: Figure 5: Rollout of the first 40 steps of the samples.

### 4.4 Ablation Study

We conduct an ablation study to evaluate the contributions of the newly introduced components: the Energy MLP and the Energy Loss. All experiments are conducted on the Square-mh task from Robomimic (mandlekar2021matters,). Models are trained for 400 epochs with a batch size of 1024. For experiments involving the energy loss, we set the weighting coefficient $\alpha=1.0$. Performance is assessed by averaging the success rates of the top three checkpoints. We investigate: (i) the impact of the Energy MLP channel width, (ii) the effect of the coefficient $\alpha$, (iii) the role of adaLN, and (iv) the effect of different noise injection distributions.

Figure: Figure 6: Ablation Study Results. (a) Success rate as a function of MLP channel width. (b) Success rate as a function of $\alpha$. (c) Success rate with and without AdaLN. (d) Success rate under different noise injection distributions.
Refer to caption: x6.png

##### Channel Size of Energy MLP.

We investigate the impact of the Energy MLP channel width on model performance. Starting with a width equal to the token embedding size (256), we progressively increase the width to 512 and 1024. Performance improves from 0.83 to 0.85 when increasing the width to 512, but further increasing it to 1024 results in a decline to 0.80. We hypothesize that this degradation is due to overfitting. Based on these results, we adopt 512 as the default MLP width for all evaluations in the image-based simulation environment.

##### Choice of the coefficient α \alpha .

In our main experiments, we set $\alpha$ to $1.0$ empirically. To assess the sensitivity to this hyperparameter, we conduct an ablation study by varying its value. As shown in the Square-mh task, performance remains stable when reducing $\alpha$ from $1.0$ to $0.5$ (success rate $0.85$), but degrades when increasing it to $1.5$ ($0.73$) or $2.0$ ($0.71$). Thus, we adopt $\alpha=1.0$ as the default setting in all experiments.

##### Role of Adaptive Layer Normalization (adaLN).

To validate its effectiveness, we compare our approach with a baseline that simply concatenates Gaussian noise with the output vector from the Transformer decoder, instead of applying adaptive layer normalization. We evaluate the success rate on the Square-mh task. Removing adaLN leads to a drastic performance drop from $0.85$ to $0.31$, highlighting that adaLN is crucial for effectively modeling noise conditioning.

##### Choice of Noise Distribution.

By default, the noise injected into the Energy MLP is sampled from a uniform distribution in the range $[-0.5,0.5]$. We also evaluated two alternative noise sources: a uniform distribution over $[-1.0,1.0]$ and a Gaussian distribution $\mathcal{N}(0,1)$. In all cases, the resulting success rates differ by less than 1%, indicating that our method is robust to variations in the noise injection distribution.

## 5 Conclusion

Energy Policy offers a novel approach to multimodal action modeling, making it well-suited for robotic tasks that demand high-precision manipulation. Moreover, thanks to its underlying mechanism, it can generate a sequence of continuous actions in a single forward pass, resulting in significantly faster inference. Our strong performance across a variety of robotic tasks demonstrates both the effectiveness and efficiency of the proposed method.

## References

- [1]
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song.
Diffusion policy: Visuomotor policy learning via action diffusion.
The International Journal of Robotics Research, page 02783649241273668, 2023.
- [2]
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al.
Openvla: An open-source vision-language-action model.
arXiv preprint arXiv:2406.09246, 2024.
- [3]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.
Rt-1: Robotics transformer for real-world control at scale.
arXiv preprint arXiv:2212.06817, 2022.
- [4]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.
Rt-2: Vision-language-action models transfer web knowledge to robotic control.
arXiv preprint arXiv:2307.15818, 2023.
- [5]
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al.
Spatialvla: Exploring spatial representations for visual-language-action model.
arXiv preprint arXiv:2501.15830, 2025.
- [6]
Moo Jin Kim, Chelsea Finn, and Percy Liang.
Fine-tuning vision-language-action models: Optimizing speed and success.
arXiv preprint arXiv:2502.19645, 2025.
- [7]
Yue Su, Xinyu Zhan, Hongjie Fang, Han Xue, Hao-Shu Fang, Yong-Lu Li, Cewu Lu, and Lixin Yang.
Dense policy: Bidirectional autoregressive learning of actions.
arXiv preprint arXiv:2503.13217, 2025.
- [8]
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu.
3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.
arXiv preprint arXiv:2403.03954, 2024.
- [9]
Yixiao Wang, Yifei Zhang, Mingxiao Huo, Ran Tian, Xiang Zhang, Yichen Xie, Chenfeng Xu, Pengliang Ji, Wei Zhan, Mingyu Ding, et al.
Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learning.
arXiv preprint arXiv:2407.01531, 2024.
- [10]
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al.
Octo: An open-source generalist robot policy.
arXiv preprint arXiv:2405.12213, 2024.
- [11]
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.
\pi_0: A vision-language-action flow model for general robot control.
arXiv preprint arXiv:2410.24164, 2024.
- [12]
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.
\pi_ {0.5}: a vision-language-action model with open-world generalization.
arXiv preprint arXiv:2504.16054, 2025.
- [13]
Glenn W Brier.
Verification of forecasts expressed in terms of probability.
Monthly weather review, 78(1):1–3, 1950.
- [14]
Tilmann Gneiting and Adrian E Raftery.
Strictly proper scoring rules, prediction, and estimation.
Journal of the American statistical Association, 102(477):359–378, 2007.
- [15]
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.
Learning fine-grained bimanual manipulation with low-cost hardware.
arXiv preprint arXiv:2304.13705, 2023.
- [16]
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín.
What matters in learning from offline human demonstrations for robot manipulation.
arXiv preprint arXiv:2108.03298, 2021.
- [17]
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox.
Mimicgen: A data generation system for scalable robot learning using human demonstrations.
arXiv preprint arXiv:2310.17596, 2023.
- [18]
Zhefei Gong, Pengxiang Ding, Shangke Lyu, Siteng Huang, Mingyang Sun, Wei Zhao, Zhaoxin Fan, and Donglin Wang.
Carp: Visuomotor policy learning via coarse-to-fine autoregressive prediction.
arXiv preprint arXiv:2412.06782, 2024.
- [19]
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine.
Fast: Efficient action tokenization for vision-language-action models.
arXiv preprint arXiv:2501.09747, 2025.
- [20]
Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg.
Consistency policy: Accelerated visuomotor policies via consistency distillation.
arXiv preprint arXiv:2405.07503, 2024.
- [21]
Zhendong Wang, Zhaoshuo Li, Ajay Mandlekar, Zhenjia Xu, Jiaojiao Fan, Yashraj Narang, Linxi Fan, Yuke Zhu, Yogesh Balaji, Mingyuan Zhou, et al.
One-step diffusion policy: Fast visuomotor policies via diffusion distillation.
arXiv preprint arXiv:2410.21257, 2024.
- [22]
Tilmann Gneiting and Adrian E. Raftery.
Strictly proper scoring rules, prediction, and estimation.
Journal of the American Statistical Association, 102:359 – 378, 2007.
- [23]
Gábor J. Székely.
E-statistics: The energy of statistical samples.
Technical Report 3.05, Bowling Green State University, Department of Mathematics and Statistics, 2003.
- [24]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- [25]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization.
arXiv preprint arXiv:1607.06450, 2016.
- [26]
Stefan Elfwing, Eiji Uchibe, and Kenji Doya.
Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.
Neural networks, 107:3–11, 2018.
- [27]
William S Peebles and Saining Xie.
Scalable diffusion models with transformers. 2023 ieee.
In CVF International Conference on Computer Vision (ICCV), volume 4172, 2022.
- [28]
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman.
Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.
arXiv preprint arXiv:1910.11956, 2019.
- [29]
Peter R. Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian S. Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson.
Implicit behavioral cloning.
In Conference on Robot Learning, 2021.
- [30]
Emanuel Todorov, Tom Erez, and Yuval Tassa.
Mujoco: A physics engine for model-based control.
2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012.
- [31]
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, N. Rudin, Arthur Allshire, Ankur Handa, and Gavriel State.
Isaac gym: High performance gpu-based physics simulation for robot learning.
ArXiv, abs/2108.10470, 2021.
- [32]
Yixiao Wang, Yifei Zhang, Mingxiao Huo, Ran Tian, Xiang Zhang, Yichen Xie, Chenfeng Xu, Pengliang Ji, Wei Zhan, Mingyu Ding, and Masayoshi Tomizuka.
Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learning.
In Conference on Robot Learning, 2024.
- [33]
Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo.
Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution.
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16467–16476, 2023.
- [34]
Noam M. Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
ArXiv, abs/1701.06538, 2017.
- [35]
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song.
Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots.
ArXiv, abs/2402.10329, 2024.
- [36]
Krishan Rana, Robert Lee, David Pershouse, and Niko Suenderhauf.
Imle policy: Fast and sample efficient visuomotor policy learning via implicit maximum likelihood estimation.
arXiv preprint arXiv:2502.12371, 2025.
- [37]
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto.
Behavior generation with latent actions.
arXiv preprint arXiv:2403.03181, 2024.
- [38]
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu.
3d diffusion policy.
arXiv preprint arXiv:abs/2403.03954, 2024.
- [39]
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan C. Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
arXiv preprint arXiv:1910.10897, 2019.

## Appendix A Appendix

### A.1 Manipulation Tasks on Meta-World

##### Setup.

We further extend our evaluation by adapting our method to the 3D Diffusion Policy [38] framework and testing on the Meta-World benchmark [39]. Following prior work [38], we evaluate five challenging tasks: Disassemble, Pick Place Wall, Shelf Place, Stick Pull, and Stick Push. This setting validates the ability of our model to handle complex, high-dimensional observation spaces.

##### Implementation.

We adopt the same observation setup, encoder, and network architecture as 3D Diffusion Policy (DP3), with one key modification: the diffusion timestep embedding is removed, and a two-layer energy MLP (width 256) is added as the output head. Both models are trained for 1000 epochs with a batch size of 256.

##### Results.

Table [6](https://arxiv.org/html/2510.12483v1#A1.T6) summarizes the results. Our method achieves a higher average success rate than DP3 ($0.71$ vs. $0.70$), while reducing inference latency by more than an order of magnitude ($7.72$ ms vs. $73.33$ ms). These results highlight that our approach maintains competitive performance while delivering substantial efficiency gains in high-dimensional observation spaces settings.

### A.2 Experiments Implementation Details

#### A.2.1 Single-Task Simulation (Robomimic, Franka Kitchen, PushT)

We adopt the observation space configuration from Diffusion Policy [1]. For image-based tasks, we use an observation horizon of 2, consisting of multiview RGB images and proprioceptive states. For state-based tasks, the observation horizon is 2 (Robomimic [16]) or 4 (Franka Kitchen [28]), using low-dimensional object state vectors as input. The action prediction horizon is set to 16, with only the first 8 actions executed during the evaluation. Our model, a decoder-only transformer with an energy-based MLP head, is trained with a batch size of 1024 for 400 epochs (image-based) or 600 epochs (state-based) using $\alpha=1.0$. For the decoder-only Transformer, we use the same depth and token embedding dimension as DP-T. For the task involving two arms, we double the energy MLP width. Baseline models are trained and evaluated following the original protocols.

#### A.2.2 Multi-Task Simulation (MimicGen)

We follow the observation space setup from Sparse Diffusion Policy [32], using an observation horizon of 2 for image and robot pose input. Our model employs the same architecture as in the single-task evaluation, augmented with a task-class token and doubled network depth. Training is performed for 400 epochs with a batch size of 1024 and $\alpha=1.0$. Baseline models are trained and evaluated according to their respective protocols. As in the single-task setting, the model predicts an action sequence of length 16, from which the first 8 actions are executed.

**Table 6: Evaluation on Meta-World.**
| Policy | Disassemble | Pick Place Wall | Shelf Place | Stick Pull | Stick Push | Avg. | Speed (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Energy Policy (Ours) | 0.95 | 0.85 | 0.30 | 0.70 | 0.75 | 0.71 | 7.72 |
| DP3 | 0.95 | 0.90 | 0.25 | 0.60 | 0.80 | 0.70 | 73.33 |

#### A.2.3 Real-World Experiments

For all real-world tasks, we use an observation horizon of 2, incorporating camera RGB images and proprioceptive states.

For the static tasks (Cup and Rabbit), our model retains the simulation architecture of a single task with modifications to address real-world complexity and noise: the transformer embedding size is tripled and the MLP channel width is doubled. Training is carried out for 150 epochs with a batch size of 64 and $\alpha=1.0$. The baseline model follows its default architecture and training configuration. Both models predict an action sequence of length 16, with the first 8 actions executed.

For the dynamic task (Catch), our model uses the same architecture as in the single-task simulation setting without modification to the Transformer embedding size or MLP channel width. Training is carried out for 200 epochs with a batch size of 64
and $\alpha=1.0$. The baseline model follows the configuration specified in its original implementation for real-world PushT.

### A.3 Execution Visualization

We provide qualitative visualizations of successful rollouts across all benchmarks. In each figure, a row corresponds to a task rollout, with the leftmost frame showing the initial state and the rightmost frame showing the final state.

#### A.3.1 Robomimic

Figure: Figure 7: Visualization of Robomimic experiments.
Refer to caption: x7.png

#### A.3.2 Franka Kitchen

Figure: Figure 8: Visualization of Franka Kitchen subtasks.
Refer to caption: x8.png

#### A.3.3 MimicGen

Figure: Figure 9: Visualization of MimicGen multi-task experiments.
Refer to caption: x9.png

#### A.3.4 Meta-World

Figure: Figure 10: Visualization of Meta-World experiments.
Refer to caption: x10.png

#### A.3.5 Real World (Static)

Figure: Figure 11: Visualization of real-world static task experiments.
Refer to caption: x11.png

#### A.3.6 Real World (Dynamic)

Figure: Figure 12: Visualization of real-world dynamic task experiments.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: * in section 3.2.1, in equation (3) why isn’t a 3rd independent sample drawn, e.g. to match the formulation in (2)? X1, x2, and y are used in (2).
* Why wasn’t the sampling done at the input to the Transformer? Instead of in a separate MLP head.
* Can the authors please describe, at a fundamental level, how this differs from a standard CVAE architecture? What would happen if the uniform noise was switched to Gaussian, and the loss became a standard ELBO loss? Is there any benefit from using the Energy loss in that case?
soundness: 3
strengths: * Decreases inference time substantially, making their method more suitable for real-time inference
* Explores a novel loss formulation that I haven’t seen before in this context.
* Many experiments in simulation and real are provided, showing that the method is applicable in multiple scenarios.
* Well-written and clear.
summary: In this work, the authors consider policy learning in a continuous action space for robotic manipulation tasks. Specifically, the authors propose a method for learning multi-modal action distributions using an architecture which has a lower inference time than other multi-modal policy learning techniques (including most “fast” ones). The key idea here is to predict a distribution representation with a transformer, and then sample actions from an Energy MLP conditioned on uniform random noise. This Energy MLP is trained using the Energy loss. When trained on existing benchmark tasks, this method matches the performance of other SOTA methods while reducing the parameter count and inference time.
weaknesses: * The benchmark tasks chosen are in some ways too easy - meaning their distributions are relatively narrow and not particularly multimodal. It’s not clear whether the proposed MLP-based architecture scales to rather complex distribution
* Missing an ablation where parameter size is held constant compared to some of the larger models - e.g. could the larger models be reduced in parameter size with the same training technique but yield the same results? Speedup is presumably from that inference time reduction due to fewer parameters.
* The multimodal behavior “T” example is not really sufficient to demonstrate multimodality, as the task is a bit too simple (2 modes)
* Necessity of the energy loss (as compared to other distribution losses) is not well-motivated
* nitpick: flow-matching is not mentioned in the related work

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Could you elaborate on the relationship between Energy Policy and the broader category of implicit generative models? The method of using a noise-conditioned MLP seems to be a form of implicit generator. How does training with the energy score (Eq 3) compare conceptually to other implicit objectives like IMLE?

2. Following up on the multimodality claim, can you provide more insight into the capacity of the noise-injection mechanism? Have you tested its ability to model more complex, multi-modal distributions (e.g., 3+ discrete, valid solutions)?

3. For the real-world experiments, why was the comparison limited to diffusion policies? A comparison against a fast autoregressive baseline like CARP would have been more comprehensive, as CARP was a strong competitor in simulation.

4. Do you have a hypothesis for the performance degradation when $\alpha > 1.0$ (Fig 6b)? Is this due to training instability, or a more fundamental property of the energy score in this application?

5. In your dynamic "Catch" experiment (Sec 4.2.2), you demonstrate a key application for low-latency policies. Could you please briefly discuss relevant concurrent work in this specific area, such as "Latent Adaptive Planner for Dynamic Manipulation" (Noh et al., 2025)? It would be valuable to contrast your single-step action policy with their trajectory-level planning approach for this class of dynamic task.
soundness: 3
strengths: 1. Significant Performance (Speed): The primary claim of the paper is its speed, and it delivers unequivocally. The ability to generate continuous, multimodal actions in a single pass is a major advantage. The reported inference times (e.g., 7.02ms on PushT in Table 4, 3-7x faster than the next-fastest baseline CARP, and 20-70x faster than diffusion policies in Table 1) are a substantial contribution for real-world robotics, where low latency is paramount. The real-world "Catch" task (Sec 4.2.2) provides excellent practical validation of why this inference speed matters.

2. Excellent Performance (Success Rate): The method does not sacrifice accuracy for speed. It demonstrates state-of-the-art or comparable success rates against a very strong and recent set of baselines (DP, CARP, OneDP, CP, etc.). The comparison in Table 4 is particularly strong, showing that while another fast policy (VQ-BeT) exists, it suffers a massive performance drop (0.87 vs 0.68 avg. success), whereas Energy Policy maintains high performance.

3. Novelty and Elegance: The core idea of using an energy score to train an implicit generative model (the Energy MLP + noise injection) is an elegant solution to the problem. It cleverly sidesteps the iterative sampling of diffusion and the imprecision of autoregressive tokenization, effectively getting the best of both worlds. The architecture is simple and the method is well-motivated.

4. Comprehensive Evaluation: The experimental validation is thorough.
summary: This paper introduces "Energy Policy," a novel visuomotor policy designed to address the critical trade-off between inference speed, action precision, and the ability to model multimodal action distributions. The authors identify key limitations in existing approaches: autoregressive models sacrifice fine-grained precision by discretizing actions, simple L1/L2 regression policies are inherently unimodal (failing in tasks with multiple valid solutions), and diffusion models, while effectively multimodal, are computationally expensive due to their iterative denoising process.

The proposed Energy Policy tackles this by making two primary contributions:

Energy Score as Objective: It employs the energy score, a strictly proper scoring rule, as its learning objective. This allows the model to learn complex, multimodal distributions without an explicit density function.

Energy MLP Architecture: It introduces a simple yet effective "Energy MLP" head. This module takes the output of a standard Transformer decoder (a latent vector $z_t$) and a random noise vector $\epsilon$ as input. By training this implicit generator (EnergyMLP($z_t, \epsilon$)) with the energy score loss, the model learns to produce action samples from the desired multimodal distribution.

Crucially, this design allows for action generation in a single forward pass during inference by simply providing one noise sample. The authors conduct a comprehensive evaluation across a wide array of simulated benchmarks (Robomimic, Franka Kitchen, MimicGen, PushT) and two real-world robotic setups. The results are compelling: Energy Policy consistently matches or exceeds the success rates of state-of-the-art baselines (including strong diffusion and autoregressive models) while being significantly faster, achieving speedups ranging from 3x to 70x.
weaknesses: 1. Missing Context in Related Work (Implicit Models): The core mechanism—a deterministic function $f(z, \epsilon)$ that maps a latent code $z$ and a noise vector $\epsilon$ to an output, trained with a loss that compares samples—is the definition of an implicit generative model. A discussion comparing the energy score loss to other implicit losses would significantly strengthen the paper's positioning.

2. Depth of Multimodality Analysis: The paper claims to model multimodal distributions, and provides visual evidence in Figure 5 for a bimodal task (go left or go right). This is good, but the analysis is somewhat superficial. The "mode" is generated by a simple uniform or Gaussian noise vector. What is the capacity of this mechanism? Can it learn to represent, for example, three or four distinct, sharp modes (e.g., three different valid grasps for an object) with the correct probabilities?

3. Baseline Selection in Real-World Tasks: In the real-world evaluation (Sec 4.2), the authors compare their method only against diffusion policies (DP-UMI and DP-C). While they convincingly win on speed and success, this is a slightly weak comparison, as diffusion is known to be slow. A much more compelling experiment would have included a comparison against the strongest fast baseline from simulation.

4. Analysis of $\alpha$ Hyperparameter: The ablation on the $\alpha$ parameter for the energy loss (Fig 6b) is interesting. The paper notes that $\alpha=1.0$ is used empirically, and that performance degrades for $\alpha=1.5$ or $\alpha=2.0$. The paper would be strengthened by a brief hypothesis for why this is the case. Is it an issue of gradient stability during training when using an L1.5 or L2 norm in this specific loss formulation (Eq 3)?

---

confidence: 5
contribution: 3
initial_rating: 2
presentation: 2
questions: - Improvements on single-task policies may not be enough at current timesteps. Since the architecture is quite like some VLA models, do you have any plans to apply the energy policy to them to replace the flow matching method during post-training?
- Could you please provide more experimental results involving more complex manipulation tasks during the rebuttal stage?
- Can you provide some insights regarding your selection of the coefficient $\alpha$ from the perspective of data distribution other than just experience?
soundness: 2
strengths: - The whole idea and implementation of the Energy Policy is simple but quite effective. Besides, it naturally requires only one single pass to predict actions without the need for distillation.
- Using the energy score as the learning objective, the model can not only generate samples close to the target action but also maintain a certain level of diversity between independent samples.
- Experiments are conducted in both simulation and real-world settings. Simulation experiments include RoboMimic, Franka Kitchen, MimicGen, and PushT. Real-world experiments also have more than one setting. The speed advantage of Energy Policy is demonstrated through these settings.
summary: In this work, the authors introduced a visuomotor manipulation policy named Energy Policy. Contrary to previous works like distilled diffusion policy or other autoregressive methods, Energy Policy naturally only requires one single forward pass to predict multi-modal actions. The whole framework is lightweight and simply implemented with an energy MLP and a transformer decoder, which are optimized by the energy score. Experiments are conducted in both simulation and real-world settings, through which Energy Policy exhibits its comparable performance and efficiency to some degree.
weaknesses: - Although experiments are conducted in various settings, the tasks that the authors choose are quite simple, and most of them are just some basic atomic skills. For example, in the tasks from RoboMimic, state-based methods have already achieved 1.0 success rates. Thus, they cannot truly reflect the proposed methods’ effectiveness. So are the cases for Franka Kitchen. 
- The selection of baselines across different environments is not consistent, and many comparisons are not complete. For instance, why are the original DP’s performances not shown in Table 4? Besides, multiple positions are blank and filled with dashes.
- Similarly, the real-world experiments also have such kinds of problems. The so-called dynamic tasks are actually still static ones.
- The writing of this work requires improving. Some expressions are just too over, e.g., in line 60 “...making it computationally prohibitive…”

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: Can you provide a deeper discussion based on your results, on what settings most favor your method, and which settings can be leading to drop in performance? to rephrase, do you have an intuition on what to blame the drop in performance in certain tasks on?
soundness: 4
strengths: - The paper is very well written and is an enjoyable read
- Finding methods that appropriately tradeoff inference speed and performance is very important for policy learning, specially in robotic manipulation
- The proposed approach is very simple making it an elegant solution to the problem
- The results are quite promising, almost consistently showing a mostly equivalent (sometimes worse, sometimes better) performance in comparison to the baselines, while improving on inference speed
- Experiments include ablations of the introduced hyperparameters. This makes building on top of this work by properly understanding a lot better.
summary: The paper proposes using energy score-based policies as a way to obtain multi-modal continuous actions while keeping a reasonable inference speed. The main motivation is to bypass the need for lengthy diffusion online or the loss of accuracy via distillation or loss of granularity and accuracy based on discretization. The proposed approach is compared to multiple baselines in single and mutli-task settings in simulation as well as 3 real-world tasks.
weaknesses: - Novelty is limited
- Table 4 is missing some values without an explanation for why this is the case
- The papers lacks a deeper analysis of the experimental results, for instance in some cases the proposed approach outperforms the baselines in others it doesn't. It would be interesting if the authors can attempt to provide an understanding of which settings are most and least suited for their method


Decision: Reject