AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulation

## Abstract

Abstract Vision-language-action (VLA) models have shown promise on task-conditioned control in complex settings such as bimanual manipulation. However, the heavy reliance on task-specific human demonstrations limits their generalization and incurs high data acquisition costs. In this work, we present a new notion of task-agnostic action paradigm that decouples action execution from task-specific conditioning, enhancing scalability, efficiency, and cost-effectiveness. To address the data collection challenges posed by this paradigm—such as low coverage density, behavioral redundancy, and safety risks—we introduce ATARA (Automated Task-Agnostic Random Actions), a scalable self-supervised framework that accelerates collection by over 30 × 30\times 30 × compared to human teleoperation. To further enable effective learning from task-agnostic data, which often suffers from distribution mismatch and irrelevant trajectories, we propose AnyPos , an inverse dynamics model equipped with Arm-Decoupled Estimation and a Direction-Aware Decoder (DAD). We additionally integrate a video-conditioned action validation module to verify the feasibility of learned policies across diverse manipulation tasks. Extensive experiments show that the AnyPos–ATARA pipeline yields a 51% improvement in test accuracy and achieves 30–40% higher success rates in downstream tasks such as lifting, pick-and-place, and clicking, using replay-based video validation.

## 1 Introduction

Building general-purpose embodied agents that can perceive, reason, and act in complex physical environments is a fundamental task. Recent progress in vision-language-action (VLA) models—such as RT-X rtx, Octo octo, RDT liu2024rdt, and OpenVLA openvla—has advanced this goal by learning task-conditioned action policies grounded in visual observations and natural language instructions. These models have demonstrated promising performance on manipulation tasks, including object picking, placement, and instruction following openvla; liu2024rdt.

Despite these advances, a core bottleneck lies in the lack of scalable, diverse, and semantically aligned action data. Unlike the vision and language domains with internet-scale datasets, robotic data collection remains inherently expensive, time-consuming, and tightly coupled to specific hardware platforms. Most existing datasets are task-specific–they are collected explicitly to solve a particular robotic task, such as stacking blocks or opening doors, within a narrow operational context and embodiment.
While the datasets can be effective for learning isolated skills, they offer poor coverage across different robot morphologies (i.e., embodiment gap (rtx,)), lack behavioral diversity, and are rarely reusable across tasks or domains. Benchmarks such as ManiSkill2 gumaniskill2, RT-X rtx, and RoboVerse geng2025roboverse underscore the limitations of current data resources. Even large-scale efforts like Bridge Data bridge reveal the high annotation burden required to curate task-specific data, further compounding the cost and complexity of dataset creation at scale.

In this work, we first present a new notion of task-agnostic action paradigm. We then introduce a method for automated, scalable collection of task-agnostic random action data (ATARA). Building on this, we propose AnyPos—a high-precision prediction model that trains on task-agnostic action data.

Task-Agnostic Action Data.
To address the fundamental data bottleneck in Embodied AI, we argue that decoupling actions from tasks is a key step towards building scalable and generalizable embodied agents.
Specifically, we advocate for the use of task-agnostic action data—trajectories collected without conditioning on specific task instructions or reward signals. This shift brings several advantages: (1) task-agnostic data is much easier and more scalable to collect, particularly through autonomous exploration or simulation; (2) it supports cross-task reuse, enabling general-purpose models to transfer across downstream tasks; and (3) it reduces human-supervision and annotation costs, as it does not require success labels or goal alignment.

To collect and utilize such task-agnostic action data effectively for real-world manipulation, we have to address several issues. First, naïve random exploration often leads to inefficient coverage, redundant behaviors, collisions, and irrelevant interactions. Second, achieving diverse and physically grounded $\langle\text{image},\text{action}\rangle$ pairs without task supervision remains difficult.
To address this, we introduce Automated Task-Agnostic Random Actions (ATARA)—a scalable data collection pipeline that uses scripted policies to uniformly explore the cubic workspace of robotic manipulators.
ATARA enables efficient and reproducible collection of diverse trajectories without human intervention, significantly reducing the cost ($30\times$ acceleration and no labor costs) and bias of traditional demonstration-based datasets.

Task-agnostic data cannot be directly applied after collection because it lacks the semantic alignment typically available in goal-conditioned demonstrations, making direct action modeling challenging.
To bridge this gap, we propose a two-stage learning framework that combines internet-scale videos with ATARA-generated data.
A semantic video generation model provides high-level perceptual context, while an inverse dynamics model (IDM) acts as a robot-specific adapter that maps image frames to precise actions.
This modular design leverages physical priors from large-scale videos and allows scalable supervision via task-agnostic behaviors—without relying on explicit goal annotations.

Training with Task-Agnostic Data. Learning inverse dynamics models (IDMs) from task-agnostic data is challenging, primarily due to the fine-grained precision required for robotic control, and secondarily because of distributional mismatch and noisy behaviors.
To address these challenges,
we introduce AnyPos—a robot-specific image-to-action model trained entirely on task-agnostic trajectories by incorporating structural priors and architectural biases that improve data efficiency and model generalization. AnyPos integrates two key techniques: (1) Arm-Decoupled Estimation: In multi-arm settings, shared visual features often lead to cross-arm entanglement and inaccurate predictions. We apply a spatial decomposition algorithm based on flood-fill and symmetry-line segmentation to isolate per-arm regions and eliminate irrelevant joint dimensions. This reduces the effective hypothesis space and improves action prediction accuracy by about 20%. (2) Direction-Aware Decoder (DAD): To further inject physical priors—such as joint angles and link orientations—we design a decoder that aligns visual features (e.g., from DINOv2 dino; dinoreg) with plausible motion directions. This increases robustness to noise and yields an additional 20% improvement in downstream task performance.

Together, ATARA and AnyPos constitute a fully task-agnostic framework for training IDMs without goal supervision. By combining scalable unsupervised data collection with physically informed learning architectures, our approach demonstrates that task-agnostic action data can serve as a practical and powerful foundation for generalizable manipulation.

Figure: Figure 2: AnyPos illustration. We obtain a task-agnostic training dataset covering the entire cubic workspace of dual robotic arms using ATARA. Input to AnyPos: An image containing the robotic arms. Output of AnyPos: The action/joint position values inferred from the image.
Refer to caption: x2.png

Results.
We newly collect both training data (i.e., automated task-agnostic random actions) with 610k image-action pairs and real-world manipulation test datasets with 2.5k image-action pairs.
Our AnyPos method achieves 57.13% action prediction accuracy on the test set, which includes unseen skills and objects, surpassing previous approaches (naïve ResNet+MLP used in (unipi,; unisim,; robodreamer,; susie,)) by 51%.
In real-world robot replay tests, AnyPos-ATARA demonstrates 92.59% task success rate, representing a 33% improvement over the human-collected dataset, and surpassing previous approaches by 44%.
Furthermore, by integrating diffusion-based video generation models with AnyPos for inference, we successfully accomplish manipulation tasks, including lifting the basket, clicking, and picking up and placing various objects.

## 2 Related Work

Embodied Data Collection.
The data collection for embodied AI mainly falls into three categories: simulator data, real-robot data, and internet video data. Simulator-based methods like RoboTwin (robotwin,), ManiBox (manibox,), and AgiBot DigitalWorld (contributors2025agibotdigitalworld,) enable scalable data collection but face Sim2Real gaps and physical fidelity limitations in complex tasks.
Real-world approaches including Diffusion Policy (dp,), Mobile Aloha (mobile_aloha,), VLAs (liu2024rdt,; rtx,; openvla,), and manipulation datasets (droid,; bridge,; wu2024robomind,; rtx,; contributors2025agibotworld,) show practical capabilities but struggle with action space generalization due to task-specific data bottlenecks and high costs.
Internet videos offer rich physical and motion priors, with early work (unipi,; vpp,; gr2,; robodreamer,) showing their potential. However, effectively connecting video data to high-precision action generation remains understudied.

Embodied Policies and VLAs.
Recent embodied manipulation policies, such as ACT (mobile_aloha,) and Diffusion Policy (dp,; dp3,; dppo,), show success in real-world applications. These methods train policies using real-world data to map visual inputs to action trajectories. However, they focus on single tasks and lack explicit language understanding or multi-task generalization.
To overcome this, vision-language-action (VLA) models (liu2024rdt,; rt2,; rt1,; octo,; openvla,; liu2025hybridvla,; ding2025humanoidvla,; li2024cogact,; rtx,; pertsch2025pi0fast,; black2024pi0,) use language instructions as task conditions, enabling multi-task learning for manipulation tasks like pick-and-place and pouring. Yet, this approach requires large-scale task-conditioned action data for each robotic embodiment. Current datasets are small, and the need for task-specific actions creates embodiment gaps and generalization challenges for embodied AI (rtx,).

Video Generation Model and Inverse Dynamics Models for Embodied AI.
Recent studies explore video generation for manipulation, which typically combine a visual generation model for near-future prediction and a downstream IDM for action execution. Current methods mostly use decoupled IDM approaches, such as SuSIE (susie,), UniPi (unipi,), RoboDreamer (robodreamer,), UniSim (unisim,), and Gen2Act (gen2act,). These methods rely on generated observations (e.g., videos or images) to estimate actions but struggle with unseen tasks (unipi,), high-precision action generation, or task-specific action data. Other models, such as Seer (seer,), VPP (vpp,), and UVA (uva,), jointly predict images and actions but still face challenges such as limited task-specific action data and embodiment gaps from inconsistent action spaces.

## 3 Methods

In this section, we first formulate task-agnostic action data and then introduce our automated task-agnostic random actions (ATARA) and AnyPos. AnyPos includes arm decoupled estimation to reduce the hypothesis space and a direction-aware decoder for action estimation.

### 3.1 Task-Agnostic Action Data

Consider the language-conditioned robotic control problem with the following components: image $\bm{x}\in\mathcal{X}$, language instruction $\mathbf{l}\in\mathcal{L}$, action $\bm{a}\in\mathcal{A}$, for a 6-DOF dual-arm manipulator with two grippers, $\mathcal{A}\subseteq\mathbb{R}^{14}$.
The agent learns a policy $\pi$ that takes images and a language command as input to roll out actions $\bm{a}$ to complete the corresponding manipulation task.
Standard VLA models learn temporally extended policies $p_{\theta}(\bm{a}_{T+1:T+t}|\bm{x}_{T-H+1:T},\mathbf{l})$ (^1^11For clarity, we denote the model’s action at timestep $i-1$ as $\bm{a}_{i}$, which corresponds to the joint position at timestep $i$.),
where $\theta$ represents the parameters of the VLA policy, $T$ is the current timestep and $H$ denotes the history window size, mapping observation histories and language commands to action sequences.
Given an expert dataset $D_{\text{expert}}$, the training objective of VLAs is to maximize the likelihood:

$$ $\max_{\theta}\mathbb{E}_{\bm{a}_{T+1:T+t},\bm{x}_{T},\mathbf{l}\sim D_{\text{ expert}}}\quad p_{\theta}(\bm{a}_{T+1:T+t}|\bm{x}_{T},\mathbf{l}).$ (1) $$

However, due to the high-dimensional nature of $\mathcal{L}$ and $\mathcal{A}^{t}$, VLA models suffer from higher training complexity and greater action data requirements. Therefore, we decompose the above equation by integrating over all possible future observations:

$$ $p(\bm{a}_{T+1:T+t}|\bm{x}_{T},\mathbf{l})=\int p(\bm{x}_{T+1:T+t}|\bm{x}_{T}, \mathbf{l})\cdot p(\bm{a}_{T+1:T+t}|\bm{x}_{T+1:T+t})\,d\bm{x}_{T+1:T+t},$ (2) $$

where $d\bm{x}_{T+1:T+t}$ denotes the multiple integral of $\bm{x}_{T+1},\dots,\bm{x}_{T+t}$. Since the robot’s action space typically consists of expected joint positions, in fully observed scenarios, $\bm{x}_{i}$ already includes information about $\bm{a}_{i}$. Thus, we can decompose $p(\bm{a}_{T+1:T+t}\mid\bm{x}_{T+1:T+t})$ as:

$$ $p(\bm{a}_{T+1:T+t}|\bm{x}_{T},\mathbf{l})=\int p(\bm{x}_{T+1:T+t}|\bm{x}_{T}, \mathbf{l})\cdot\prod_{i=T+1}^{T+t}p(\bm{a}_{i}|\bm{x}_{i})\,d\bm{x}_{T+1:T+t}.$ (3) $$

Even if the action space includes joint velocities, we only need to add $\bm{x}_{i-1}$ to the conditioning terms.
In this way, modeling task-specific actions can be decomposed into two components: task-specific predicted images and task-agnostic actions:

$$ $\underbrace{p(\bm{a}_{T+1:T+t}|\bm{x}_{T},\mathbf{l})}_{\text{Task-Specific Actions}}=\mathbb{E}_{\bm{x}_{T+1:T+t}\sim p(\bm{x}_{T+1:T+t}|\bm{x}_{0}, \mathbf{l})}\left[\prod_{i=T+1}^{T+t}\underbrace{p(\bm{a}_{i}|\bm{x}_{i})}_{ \text{Task-Agnostic Actions}}\right].$ (4) $$

To address the challenges of grounding task semantics in action prediction while enabling generalization from task-agnostic data, we adopt a modular framework that decouples high-level semantic understanding from low-level physical control. Specifically, we employ a video generation model $\mathcal{M}_{x}$ and an inverse dynamics model (IDM) $\mathcal{F}_{\delta}$, each trained independently to leverage their respective strengths. The video model $\mathcal{M}_{x}:\mathcal{L}\times\mathcal{X}\rightarrow\mathcal{X}^{N}$ generates a sequence of future visual states $\bm{x}_{T+1:T+N}$ conditioned on an initial observation $\bm{x}_{T}$ and a natural language instruction $\mathbf{l}$. This aligns the generated trajectory with the intended task semantics, serving as a semantic prior over feasible image sequences.

The inverse dynamics model $\mathcal{F}_{\delta}$ maps each image $\bm{x}_{i}$ in the generated sequence to a corresponding action $\bm{a}_{i}$, ensuring physical plausibility of the resulting behavior. Crucially, the IDM is trained exclusively on task-agnostic random exploration data $\mathcal{D}_{\text{agnostic}}=\{(\bm{x}_{i},\bm{a}_{i})\}$, where actions $\bm{a}_{i}$ are obtained via motor babbling or unsupervised exploration. This setup decouples the IDM from any specific task or language supervision, making it broadly generalizable. Formally, the IDM is trained by minimizing the error between predicted and observed actions:

$$ $\min_{\delta}\,\,\mathbb{E}_{(\bm{x}_{i},\bm{a}_{i})\sim\mathcal{D}_{\text{ agnostic}}}d(\bm{a}_{i},\mathcal{F}_{\delta}(\bm{x}_{i})),$ $$

where $d:\mathcal{A}\times\mathcal{A}\rightarrow\mathbb{R}^{+}$ is a distance metric in the action space. This objective encourages the model to learn state-conditioned action priors that exhibit smooth, physically valid behaviors (e.g., collision avoidance, stable grasps), independent of downstream task semantics.

The task-agnostic action paradigm has three main advantages:
(1) Data Efficiency and Reusable Motor Skills: Task-agnostic training avoids costly task-specific demonstrations, enabling large-scale unsupervised data collection. The inverse dynamics model (IDM), which learns a universal action prior $p(\bm{a}_{i}\mid\bm{x}_{i})$, acts as a shared motor skill library for diverse tasks.
(2) Zero-Shot Task Generalization: The IDM models task-independent action priors, allowing generalization to new tasks by adapting only the video generation model (e.g., via language prompts) without IDM retraining.
(3) Decoupled Planning and Low-Level Control: High-level planning (e.g., "open the drawer") is handled by a video generation model, while the IDM executes the visual trajectories. This modular approach simplifies policy design by framing manipulation as an image-space prediction problem.

### 3.2 ATARA: RL-Based Task-Agnostic Action Generation

Generating task-agnostic random actions that robustly cover manipulation-relevant behaviors is non-trivial.
As shown in Fig. [2](https://arxiv.org/html/2507.12768v1#S1.F2), naïve joint-space sampling often results in inefficient coverage of reachable states, redundant or degenerate motions (e.g., arms exiting the field of view), and frequent self-collisions. To address these limitations, we propose ATARA (Automated Task-Agnostic Random Actions), a reinforcement learning framework that constructs a coverage-aware mapping from end-effector space to joint space.

Single-Arm Workspace Coverage.
We define the reachable end-effector workspace as a bounded cubic volume $\mathcal{W}\subset\mathbb{R}^{3}$. Our goal is to learn a mapping:
$f_{\text{RL}}:\mathcal{W}\rightarrow\mathcal{A}$,
where $\mathcal{A}\subset\mathbb{R}^{d}$ is the action space (i.e., joint position space in our setting) of the robotic arm.
Because traditional inverse kinematics (IK) cannot calculate joint position when only eef position is given and not eef orientation, while a random orientation will result in an unsolvable IK, we train a policy $\pi_{\theta}$ via PPO (ppo,) in simulation environments (mittal2023orbit,; makoviychuk2021isaac,) to predict joint positions that bring the end-effector to a target point $\mathbf{w}\in\mathcal{W}$. The reward function is defined as:

$$ $r(\bm{a};\mathbf{w})=-\left\|\operatorname{EEF_{POS}}(\bm{a})-\mathbf{w}\right \|_{2}^{2},$ $$

where $\operatorname{EEF_{POS}}(\bm{a})$ denotes the eef position of joint position $\bm{a}$, encouraging proximity between the current end-effector position and the sampled goal.

Orientation Augmentation.
To enhance motion diversity, we sample orientation-related joints (e.g., wrist roll/pitch/yaw) from a distribution $\mathcal{A}_{\text{wrist}}$, and append them to the RL-predicted configuration as
$\bm{a}_{\text{aug}}=\left[f_{\text{RL}}(\mathbf{w})\,\|\,\bm{a}_{\text{wrist}}\right]$,
yielding physically valid and orientation-diverse action samples.

Bimanual Extension with Collision Avoidance.
We extend ATARA to bimanual settings by partitioning the workspace with a virtual random boundary plane $\mathcal{B}$, yielding left/right regions $\mathcal{W}_{L}$ and $\mathcal{W}_{R}$ without intersection for collision avoidance. For each arm, we independently sample:

$$ $\mathbf{w}_{L}\sim\mathcal{U}(\mathcal{W}_{L}),\quad\mathbf{w}_{R}\sim\mathcal {U}(\mathcal{W}_{R}),$ $$

where $\mathcal{U}$ denotes uniform distribution, and compute corresponding joint configurations:

$$ $\bm{a}_{L}=f_{\text{RL}}(\mathbf{w}_{L}),\quad\bm{a}_{R}=f_{\text{RL}}(\mathbf {w}_{R}).$ $$

Embodiment-Aware Reusability.
ATARA is fully defined by the robot’s URDF and kinematic model. It does not depend on camera intrinsics or extrinsics. When the visual configuration changes, ATARA can regenerate new task-agnostic trajectories by replaying the workspace sampling process, preserving embodiment-specific motion constraints.

In this case, ATARA provides scalable, collision-free action sampling across diverse robot configurations. It enables efficient task-agnostic data generation that preserves inherently encoded robot embodiment information and broad behavioral coverage, serving as a reusable prior for downstream policy learning.

### 3.3 Training with Task-Agnostic Data

Let $\bm{x}\in\mathcal{X}$ be the observation containing the robotic arm (e.g., RGB images from a high camera, a left-wrist camera, and a right-wrist camera)
and let $\bm{a}=(a_{1},\ldots,a_{14})\in\mathbb{R}^{14}$ denote the full joint configuration (7 DoF per arm). Estimating $\bm{a}$ from $\bm{x}$ in a bimanual setting presents unique challenges: the output space doubles in dimensionality, the joint hypothesis space grows quadratically, cross-arm visual interference introduces estimation noise detrimental to precision, and high-precision actions are crucial in robotic manipulation—failure occurs otherwise, as shown in Fig. [2](https://arxiv.org/html/2507.12768v1#S1.F2).

To address these issues, we propose a two-stage estimation strategy (see Fig. [2](https://arxiv.org/html/2507.12768v1#S1.F2)):

$$ $\bm{x}\xrightarrow{\Phi}(\bm{x}_{L},\bm{x}_{R})\xrightarrow{f_{L},f_{R}}\hat{ \bm{a}}=\left[f_{L}(\bm{x}_{L});f_{R}(\bm{x}_{R})\right],$ $$

where $\Phi$ segments the input image into left and right arm views, and $f_{L},f_{R}$ are independent inverse dynamics regressors. We further introduce a Direction-Aware Decoder (DAD) for high-precision joint estimation under pose variations.

#### 3.3.1 Arm-Decoupled Estimation

We observe that when estimating $\bm{a}_{L}$ (left arm joints), the model often attends to visual features of the right arm, and vice versa (see Appendix [A.1](https://arxiv.org/html/2507.12768v1#A1.SS1)). To mitigate this, we isolate the input features per arm.

Segmentation. We define a heuristic segmentation operator $\Phi:\bm{x}\rightarrow(\bm{x}_{L},\bm{x}_{R})$ to split the left and right arms in the observation $\bm{x}$, using a flood-fill algorithm initialized at fixed pedestal joints. If occlusion or overlap occurs, fallback split lines are used to ensure clean separation.

Independent Regression. Two sub-networks $f_{L},f_{R}:\mathbb{R}^{h\times w\times 3}\rightarrow\mathbb{R}^{6}$ estimate joint positions for each arm independently. Gripper poses are estimated by specialized heads operating on wrist-centric crops. The final output is the concatenation:

$$ $\hat{\bm{a}}=\left[f_{L}(\bm{x}_{L});f_{R}(\bm{x}_{R})\right].$ $$

This decoupling reduces the visual hypothesis space, improves estimation accuracy, and enables specialization per arm. For specific technical details, please refer to App. [B.2.1](https://arxiv.org/html/2507.12768v1#A2.SS2.SSS1).

#### 3.3.2 Direction-Aware Decoder (DAD)

For networks $f_{L},f_{R}$, we choose DINOv2 with register (DINOv2-Reg) (dinoreg,) as the visual encoder because it removes artifacts from feature maps compared to DINOv2 (dino,), better meeting our requirements for pixel-level regression and high-precision prediction.
Given DINOv2-Reg features $\bm{Y}\in\mathbb{R}^{C\times H\times W}$, our goal is to estimate joint values with sub-0.06 accuracy (empirical value, on a 3.0 unit scale), which is critical for successful replay. To meet this high-precision requirement, DAD integrates three core components to improve directional sensitivity and spatial generalization.

(1) Multi-Scale Dilated Convolutions.
We extract multi-scale directional features via parallel convolutions:
$F_{d}=\sigma\left(\mathcal{C}_{d}(\bm{Y})\right),\quad d\in\mathcal{D}$,
where $\mathcal{C}_{d}$ is a dilated convolution with rate $d$, and $\sigma$ is a nonlinearity (e.g., GELU). The outputs are aggregated as $F=\bigoplus_{d\in\mathcal{D}}F_{d}$, capturing spatial cues at multiple receptive fields.

(2) Deformable Convolutions.
To account for articulation-dependent deformations, we apply:

$$ $(\Delta p,m)=\phi(F),\quad\bm{Y}^{\prime}=\mathcal{C}_{\text{def}}(F;\Delta p, m),$ $$

where $\Delta p\in\mathbb{R}^{2k}$ are spatial offsets and $m\in[0,1]^{k}$ are learned modulation masks (dai2017deformable,).

(3) Angle-Sensitive Pooling.
We rotate features across multiple orientations to capture features in different directions:

$$ $P=\bigoplus_{\theta\in\Theta}\mathcal{P}\left(\mathcal{R}_{\theta}(\bm{Y}^{ \prime})\right),$ $$

where $\mathcal{R}_{\theta}$ denotes rotation by angle $\theta$, and $\mathcal{P}$ is a spatial pooling function (e.g., average pooling). This produces an orientation-aware feature representation.

Joint Prediction.
The pooled feature $P$ is projected to joint outputs via a linear head: $\hat{\bm{a}}=\mathrm{MLP}(P)$.

Training Objective.
We train the full model whose all trainable parameters are denoted by $\delta$ on a task-agnostic dataset $\mathcal{D}_{\text{agnostic}}=\{(\bm{x}^{(i)},\bm{a}^{(i)})\}$ by minimizing the weighted smooth L1 error:

$$ $\mathcal{L}(\delta)=\mathbb{E}_{(\bm{x},\bm{a})\sim\mathcal{D}_{\text{agnostic }}}d(\hat{\bm{a}}(\bm{x};\delta),\bm{a}).$ $$

Compared to a monolithic regression baseline, Arm-Decoupled Estimation improves action prediction accuracy by about 20%, and the DAD yields a further 20% improvement. The combined system meets the 0.06 precision threshold required for successful video-based manipulation replay.

## 4 Experiments

Figure: Figure 3: The schematic of the dual-arm setup. The red box is added manually, not model input. The bottom-left/right subfigures display left/right grippers. The top subfigure depicts the 2 lightweight 6-DOF robotic arms, each comprising 2 base joints, 1 elbow joint, and 3 high-precision wrist joints.
Refer to caption: extracted/6629930/figs/sample_image_100_edit.png

We evaluate AnyPos through three progressively rigorous experiments:
(a) Action Prediction Accuracy: We compare AnyPos-ATARA’s performance against standard baselines (ResNet and task-specific datasets) on a unified test benchmark to assess its high-precision prediction capability.
(b) Real-World Replay: We test AnyPos-ATARA’s robustness on common and novel long-horizon tasks by executing its predictions through ground-truth videos, comparing success rates with baselines.
(c) Real-World Deployment: AnyPos-ATARA consistently completes diverse tasks using generated (non-real) video inputs.

### 4.1 Experimental Setup

Real Robot.
Mobile ALOHA (mobile_aloha,) is a commonly used mobile dual-arm robot for manipulation tasks. Each 6-DoF arm has a gripper, creating a 14-dimensional action space for various tasks. We modify it with three RGB cameras: two wrist-mounted and one rear-mounted elevated camera to observe the workspace. This setup provides complete visual data for IDMs’ qpos predictions. The model uses this input to predict all 14 joint positions. The red box in Fig. [3](https://arxiv.org/html/2507.12768v1#S4.F3) (added manually, not part of model input) emphasizes the wrist joint details, which are crucial for high-precision tasks.

Training Dataset: We collect 610k image-action pairs using ATARA, along with human-teleoperation training data for comparison. Fig. [7](https://arxiv.org/html/2507.12768v1#A1.F7) shows ATARA’s task-agnostic action coverage across all action dimensions in the test dataset.

Evaluation Method: We evaluate prediction accuracy (Sec. [4.2](https://arxiv.org/html/2507.12768v1#S4.SS2)) using 13 teleoperated manipulation tasks (2.5k image-action pairs) with unseen skills/objects. For real-world tasks, we assess AnyPos’s success rate with ground-truth videos (Sec. [4.3](https://arxiv.org/html/2507.12768v1#S4.SS3)) and demonstrate 14 tasks with AI-generated videos (Sec. [4.4](https://arxiv.org/html/2507.12768v1#S4.SS4)).

### 4.2 Evaluation of Action Prediction

**Table 1: The Accuracy Benchmark on Manipulation Test Dataset. All models are trained on the 610k ATARA dataset. We report the test accuracy as the predictions are deterministic.**
| Test Acc. | ResNet50 + MLPs | DINOv2-Reg + MLPs | DINOv2-Reg + DAD |
| --- | --- | --- | --- |
| w/o Arm-Decoupling | 5.83% (Baseline) | 23.96% | 34.64% |
| w/ Arm-Decoupling | 22.23% | 36.20% | 57.13% (Ours) |

To evaluate IDMs’ action prediction accuracy under real-world manipulation task distributions, we collect human demonstrations of image-action pairs and build a test benchmark with 2.5k samples. We use ResNet (he2016resnet,)+MLP (common in IDMs for  (unipi,; unisim,; robodreamer,; susie,)) and DINOv2-Reg (dino,; dinoreg,) as baselines. We compare their performance with and without Arm-Decoupled Estimation to assess AnyPos’s ability to train on task-agnostic random actions and predict precise actions for unseen tasks. Empirical results show that successful execution requires predicted joint positions (qpos) to meet an accuracy threshold of 0.06 (except the gripper, which allows 0.5). We use these thresholds to evaluate IDMs’ action prediction accuracy.

As shown in Tab. [1](https://arxiv.org/html/2507.12768v1#S4.T1), our AnyPos (i.e., DINOv2-Reg + DAD, enhanced by Arm-Decoupled Estimation), trained on ATARA data, significantly outperforms other approaches. The Arm-Decoupled Estimation alone improves accuracy by about 20%, while DAD further boosts it by about 21%. Compared to the simple ResNet + MLP used in  (unipi,; unisim,; robodreamer,; susie,), our method achieves a 56% higher accuracy.

The results highlight that AnyPos achieves a significantly higher accuracy in high-precision action prediction compared to previous IDM methods.

**Table 2: The Comparison of Human Data (human-collected manipulation data) and ATARA (Task-Agnostic Actions) method. SR denotes the success rate.**
|  | Test Acc. | Replay SR | Collection Time | Dataset Size | Manpower? |
| --- | --- | --- | --- | --- | --- |
| Human Data | 57.78% | 59.26% | $\sim$ 2 days (16h) | 33k | Yes |
| ATARA | 57.13% | 92.59% | $\sim$ 10h | 610k | Automatic |

Additionally, we collect a human-teleoperated training dataset with 33k image-action pairs of manipulation tasks.
This data collection process is labor-intensive and time-consuming, taking 2 days to complete.
As shown in Tab. [2](https://arxiv.org/html/2507.12768v1#S4.T2), AnyPos trained on the 610k ATARA dataset matches the test accuracy of the human-collected dataset while speeding up data collection by $30\times$ without human labor.

Since the human-collected training and test sets follow nearly identical distributions, while the ATARA data is randomly gathered, this demonstrates the significant efficiency and performance advantages of our ATARA data collection method.

### 4.3 Evaluation of Real-World Replay

Figure: Figure 4: The results of AnyPos with video replay to accomplish various manipulation tasks.
Refer to caption: x3.png

**Table 3: The Success Rates Benchmark of Video Replay. Refer to App. [A.4](https://arxiv.org/html/2507.12768v1#A1.SS4) for specific task demonstrations and statistical information.**
| Model | Success Rate |
| --- | --- |
| ResNet50+MLPs-ATARA (Baseline) | 48.15% |
| AnyPos-Human (trained on human-collection data) | 59.26% |
| AnyPos-ATATA (Ours) | 92.59% |

Besides testing the action prediction accuracy of AnyPos, we also assess whether this accuracy reflects real-world task success rates and long-horizon manipulation robustness. To evaluate IDMs’ manipulation success rates under controlled conditions, we use the video replay method. First, human operators record robot-view videos of teleoperated task executions. The environment is then reset to the initial state shown in the video. Next, we feed each frame of these ground truth videos to the IDMs, execute the generated actions, and observe whether the robot completes the tasks successfully under the same initial conditions.

Our real-robot replay tasks consist of 10 bimanual tasks across 18 objects.
Each manipulation task consists of multiple finer sub-steps to evaluate the stability of AnyPos in long-horizon execution.

Fig. [4](https://arxiv.org/html/2507.12768v1#S4.F4) and Tab. [3](https://arxiv.org/html/2507.12768v1#S4.T3) show AnyPos significantly outperforming both the baseline and AnyPos-Human (trained on human data) in replay tests, completing nearly 100% of task steps. Failures mainly occur due to reset errors or limited error tolerance in teleoperation data. Despite only 57% action prediction accuracy, AnyPos achieves high real-world success because few critical actions need high precision while others are more forgiving. Experiments demonstrate AnyPos reliably reproduces human behaviors from the replay video.

These results show that even 610k steps of automated random action collection (collected in 10 hours) can effectively enable AnyPos to generalize across diverse and long-horizon manipulation tasks.

### 4.4 Real-World Deployment with Video Generation Model

Figure: Figure 5: The results of AnyPos collaborating with video generation models to accomplish various manipulation tasks.
Refer to caption: x4.png

To evaluate the potential of IDMs and video models for joint action prediction in real-world manipulation tasks, we finetune a video generation model and combine its outputs with IDM predictions. The video model takes the current RGB observation and generates a predicted arm trajectory. AnyPos then processes each video frame to infer actions, which the robot executes.

As shown in Fig. [5](https://arxiv.org/html/2507.12768v1#S4.F5), our AnyPos, when combined with video generation models, can successfully complete real-world tasks, such as lifting the basket, clicking, and picking up and placing various objects, even when the generated videos are non-real and slightly blurred (App. [A.5](https://arxiv.org/html/2507.12768v1#A1.SS5)).
This demonstrates the potential of integrating AnyPos with generated videos for real-world manipulation tasks.

## 5 Discussions

This work formally introduces task-agnostic actions, demonstrating their potential for general-purpose embodied manipulation and their advantages over task-specific actions in terms of efficiency, cost-effectiveness, and performance.
Our whole method introduces 2 components: (1) Task-agnostic Data: Efficiently and scalably collecting Task-agnostic Random Actions (ATARA) to mitigate action data scarcity in embodied AI,
(2) Model trained with Task-agnostic Data: AnyPos with Arm-Decoupled Estimation algorithm and Direction-Aware Decoder to effectively and robustly predict high-precision actions.
Experiments demonstrate that AnyPos-ATARA significantly outperforms previous methods in action prediction accuracy (+51%) and real-world dual-arm manipulation success rates (+30$\sim$40%).
Additionally, we validate the synergistic potential between video generation models and AnyPos for embodied AI applications.

##### Limitation and Discussion

In the future, we will improve background generalization, enhance ATARA’s randomized dataset, and expand the action space to support multiple robotic platforms and dynamic manipulation. This will enable AnyPos to serve as an adapter between general embodied models and robot-specific actions.

## References

- [1]
AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang, Shukai Yang, Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu.
Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.
arXiv preprint arXiv:2503.06669, 2025.
- [2]
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu.
Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models.
CoRR, abs/2405.04233, 2024.
- [3]
Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani.
Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation.
CoRR, abs/2409.16283, 2024.
- [4]
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.
A vision-language action flow model for general robot control.
arXiv preprint arXiv:2410.24164, 3(6), 2024.
- [5]
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine.
Zero-shot robotic manipulation with pretrained image-editing diffusion models.
CoRR, abs/2310.10639, 2023.
- [6]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.
Rt-1: Robotics transformer for real-world control at scale.
arXiv preprint arXiv:2212.06817, 2022.
- [7]
Chilam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, and Minzhao Zhu.
GR-2: A generative video-language-action model with web-scale knowledge for robot manipulation.
CoRR, abs/2410.06158, 2024.
- [8]
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song.
Diffusion policy: Visuomotor policy learning via action diffusion.
In Robotics: Science and Systems, 2023.
- [9]
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.
Deformable convolutional networks.
In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017.
- [10]
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski.
Vision transformers need registers.
In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
- [11]
Pengxiang Ding, Jianfei Ma, Xinyang Tong, Binghong Zou, Xinxin Luo, Yiguo Fan, Ting Wang, Hongchao Lu, Panzhong Mo, Jinxin Liu, et al.
Humanoid-vla: Towards universal humanoid control with visual integration.
arXiv preprint arXiv:2502.14795, 2025.
- [12]
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel.
Learning universal policies via text-guided video generation.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
- [13]
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine.
Bridge data: Boosting generalization of robotic skills with cross-domain datasets.
In Kris Hauser, Dylan A. Shell, and Shoudong Huang, editors, Robotics: Science and Systems XVIII, New York City, NY, USA, June 27 - July 1, 2022, 2022.
- [14]
Zipeng Fu, Tony Z Zhao, and Chelsea Finn.
Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.
arXiv preprint arXiv:2401.02117, 2024.
- [15]
Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, et al.
Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning.
arXiv preprint arXiv:2504.18904, 2025.
- [16]
Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R. Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine.
Octo: An open-source generalist robot policy.
In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado, editors, Robotics: Science and Systems XX, Delft, The Netherlands, July 15-19, 2024, 2024.
- [17]
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al.
Maniskill2: A unified benchmark for generalizable manipulation skills.
In The Eleventh International Conference on Learning Representations.
- [18]
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan.
Accelerate: Training and inference at scale made simple, efficient and adaptable.
[https://github.com/huggingface/accelerate](https://github.com/huggingface/accelerate), 2022.
- [19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- [20]
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen.
Video prediction policy: A generalist robot policy with predictive visual representations.
CoRR, abs/2412.14803, 2024.
- [21]
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R. Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail
O’Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J. Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn.
DROID: A large-scale in-the-wild robot manipulation dataset.
In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado, editors, Robotics: Science and Systems XX, Delft, The Netherlands, July 15-19, 2024, 2024.
- [22]
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al.
Openvla: An open-source vision-language-action model.
arXiv preprint arXiv:2406.09246, 2024.
- [23]
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al.
Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.
arXiv preprint arXiv:2411.19650, 2024.
- [24]
Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song.
Unified video action model.
CoRR, abs/2503.00200, 2025.
- [25]
Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, et al.
Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model.
arXiv preprint arXiv:2503.10631, 2025.
- [26]
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu.
Rdt-1b: a diffusion foundation model for bimanual manipulation.
arXiv preprint arXiv:2410.07864, 2024.
- [27]
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al.
Isaac gym: High performance gpu-based physics simulation for robot learning.
arXiv preprint arXiv:2108.10470, 2021.
- [28]
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, et al.
Orbit: A unified simulation framework for interactive robot learning environments.
IEEE Robotics and Automation Letters, 8(6):3740–3747, 2023.
- [29]
Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo.
Robotwin: Dual-arm robot benchmark with generative digital twins (early version).
CoRR, abs/2409.02920, 2024.
- [30]
Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alexander Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew E. Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Paul Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng,
Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su, Haoshu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu,
Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi Jim Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J. Joshi, Niko Sünderhauf, Ning Liu, Norman Di Palo, Nur Muhammad (Mahi) Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry
Moore, Shikhar Bahl, Shivin Dass, Shubham D. Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Liangwei Xu, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, and Zipeng
Lin.
Open x-embodiment: Robotic learning datasets and RT-X models : Open x-embodiment collaboration.
In IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024, pages 6892–6903. IEEE, 2024.
- [31]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.
Dinov2: Learning robust visual features without supervision.
Trans. Mach. Learn. Res., 2024, 2024.
- [32]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
Pytorch: An imperative style, high-performance deep learning library.
Advances in neural information processing systems, 32, 2019.
- [33]
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine.
Fast: Efficient action tokenization for vision-language-action models.
arXiv preprint arXiv:2501.09747, 2025.
- [34]
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz.
Diffusion policy policy optimization.
arXiv preprint arXiv:2409.00588, 2024.
- [35]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.
- [36]
Hengkai Tan, Xuezhou Xu, Chengyang Ying, Xinyi Mao, Songming Liu, Xingxing Zhang, Hang Su, and Jun Zhu.
Manibox: Enhancing spatial grasping generalization via scalable simulation data generation.
CoRR, abs/2411.01850, 2024.
- [37]
Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang.
Predictive inverse dynamics models are scalable learners for robotic manipulation.
CoRR, abs/2412.15109, 2024.
- [38]
Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al.
Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation.
arXiv preprint arXiv:2412.13877, 2024.
- [39]
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel.
Learning interactive real-world simulators.
In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
- [40]
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu.
3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.
In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, 2024.
- [41]
Jiyao Zhang, Mingjie Pan, Baifeng Xie, Yinghao Zhao, Wenlong Gao, Guangte Xiang, Jiawei Zhang, Dong Li, Zhijun Li, Sheng Zhang, Hongwei Fan, Chengyue Zhao, Shukai Yang, Maoqing Yao, Chuanzhe Suo, and Hao Dong.
Agibot digitalworld.
[https://agibot-digitalworld.com/](https://agibot-digitalworld.com/), 2025.
- [42]
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan.
Robodreamer: Learning compositional world models for robot imagination.
In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
- [43]
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al.
Rt-2: Vision-language-action models transfer web knowledge to robotic control.
In Conference on Robot Learning, pages 2165–2183. PMLR, 2023.

## Appendix A More Results

### A.1 Demonstration of Cross-Arm Interference

To investigate potential interference between the two arms during IDM inference, we visualize the attention maps derived from input image gradients.
Our analysis reveals that even when estimating the qpos of a single arm, the other arm still receives significant attention, demonstrating the presence of cross-arm interference in the model’s processing.

Figure: Figure 6: Attention heatmap of the input image. Here we only estimate the qpos of the left arm, but there is a clear attention focus on the right arm. demonstrating that the model can not fully distangle the two arm during inference.
Refer to caption: x5.png

### A.2 Distribution of ATARA and Test Dataset

To measure the coverage of the action space of our random actions, we evaluate the distribution of qpos on each dimension, shown in Figure [7](https://arxiv.org/html/2507.12768v1#A1.F7).

Figure: Figure 7: Qpos distribution of task-agnostic random actions and test dataset. The figure calculates the frequency distribution of qpos in 14 dimensions. We show that random action can cover all the possible qpos in each dimension. Note that the volume of ATARA’s data significantly exceeds that of the test dataset.
Refer to caption: x6.png

### A.3 Evaluation of Action Prediction

**Table 4: The Test Accuracy and Error Benchmark on Manipulation Test Dataset. Due to the gripper’s higher tolerance for errors, the gripper’s error significantly impacts the overall error. Therefore, the Test L1 Error in the table is calculated after excluding the gripper.**
| Methods | Arm-Decoupling? | Data | Test Acc. | Test L1 Error |
| --- | --- | --- | --- | --- |
| ResNet50 + MLPs | No | ATARA Data | 5.83% | 0.1022 |
| DINOv2-Reg + MLPs | No | ATARA Data | 23.96% | 0.0440 |
| DINOv2-Reg + DAD | No | ATARA Data | 34.64% | 0.0491 |
| ResNet50 + MLPs | Yes | ATARA Data | 22.23% | 0.0444 |
| DINOv2-Reg + MLPs | Yes | ATARA Data | 36.20% | 0.0352 |
| DINOv2-Reg + DAD | Yes | ATARA Data | 57.13% | 0.0282 |
| DINOv2-Reg + DAD | Yes | Human Data | 57.78% | 0.0203 |

### A.4 Evaluation of Real-World Video Replay

Fig. [8](https://arxiv.org/html/2507.12768v1#A1.F8), Fig. [9](https://arxiv.org/html/2507.12768v1#A1.F9), and Fig. [10](https://arxiv.org/html/2507.12768v1#A1.F10) show the detailed replay performance of AnyPos, baseline (ResNet+MLP), and AnyPos-Human (trained with human-collected data) on the manually collected real-world video replay dataset, respectively.

Figure: Figure 8: The results of AnyPos collaborating with video replay to accomplish various manipulation tasks.
Refer to caption: x7.png

Figure: Figure 9: The results of baseline (ResNet+MLP) collaborating with video replay to accomplish various manipulation tasks.
Refer to caption: x8.png

Figure: Figure 10: The results of AnyPos-Human (trained with human-collected data) collaborating with video replay to accomplish various manipulation tasks.
Refer to caption: x9.png

### A.5 Real-World Deployment with Video Generation Model

Fig. [11](https://arxiv.org/html/2507.12768v1#A1.F11) demonstrates how AnyPos collaborates with a video generation model in real-world deployment.
Especially when the robotic arm is a bit blurry in the generated video, AnyPos can still complete the manipulation task.
More detailed execution videos can be found in the supplementary materials.

Figure: Figure 11: Sampled results of AnyPos collaborating with generated video to accomplish various manipulation tasks. In tasks such as "Grasp the Blue Cube" and "Grab Bottle", the generated video frames on the left exhibit blurred wrist joint details of the robotic arm. Nevertheless, AnyPos successfully accomplishes the manipulation task under these conditions.
Refer to caption: x10.png

## Appendix B Implementation Details

### B.1 ATARA Dataset

Our PPO implementation is built on [rsl_rl](https://github.com/leggedrobotics/rsl_rl). Key settings of PPO and ATARA Dataset are summarized in Table [5](https://arxiv.org/html/2507.12768v1#A2.T5).

**Table 5: Parameters of PPO and ATARA Dataset.**
| Parameters of PPO | Value |
| --- | --- |
| Clip Param. of PPO | 0.2 |
| Value Function Clipping | True |
| Value Loss Coeff. | 1.0 |
| Desired KL Divergence | 0.01 |
| Entropy Coef. | 0.01 |
| gamma | 0.98 |
| GAE (lambda) | 0.95 |
| Gradient Clipping | 1.0 |
| Learning Rate | 0.001 |
| Mini-Batch | 4 |
| The Number of Steps per Env per Update | 24 |
| Learning Epochs | 5 |
| Schedule | adaptive |
| Empirical Normalization | True |
| Target EEF position Range of Left Arm | $x\in(0.36,0.7),y\in(-0.08,0.41),z\in(0.6,1.0)$ |
| Hidden Dim. of Actor | [512, 256, 128] |
| Hidden Dim. of Critic | [512, 256, 128] |
| Activation | Elu |
| Parameters of ATARA Dataset | Value |
| Dataset Size (steps) | 610k image-action pairs |
| Dataset Size (trajectories) | 638 |
| Input | Concatenated image of high, left-wrist, and right-wrist views |
| Image Resolution | 640*720 |
| Output | 14-dim joint position |
| Content | Task-agnostic random dual-arm trajectories collected by ATARA |
| Virtual Random Boundary Plane $\mathcal{B}$ | $y\in(-0.15,0.15)$ |
| Target EEF position Range of Left Arm | $x\in(0.36,0.7),y\in(-0.08,0.41),z\in(0.6,1.0)$ |
| Target EEF position Range of Right Arm | $x\in(0.36,0.7),y\in(-0.41,0.08),z\in(0.6,1.0)$ |
| Interval Threshold between Arms | 0.15 |

### B.2 AnyPos

The model configuration of Anypos and other models trained on ATARA is listed in Table [6](https://arxiv.org/html/2507.12768v1#A2.T6). The model accepts 4 images as input, two from the wrist cameras, and two from the front camera divided by the split-line algorithm.
The four images are resized to the same size of $518\times 518$ and normalized.

**Table 6: Configuration of Different Models trained on ATARA**
|  | Models | Value |
| --- | --- | --- |
| DINO-Reg | Hidden Size | 768 |
| Hidden Layers | 12 |  |
| Model Size | 86.6M params |  |
| Pretrained | Yes |  |
| MLP-regressor | Convolution | $1\times 1,(768,2)$ |
| MLP | $(2738,256),(256,14/6/1)$ |  |
| Activation Function | GELU |  |
|  | Model Size | 0.71M params |
| DAD | $\mathcal{D}$ | $\{1,2,3,6\}$ |
| $\Theta$ | $\{0^{\circ},45^{\circ},90^{\circ},135^{\circ}\}$ |  |
| MLP | $(256,512),(512,14/6/1)$ |  |
| Activation Function | GELU |  |
|  | Model Size | 2.96M params |
| ResNet50 | Input | $224\times 224$ |
| MLP | $(2048,14/6/1)$ |  |
| Model Size | 23.6M params |  |
| Pretrained | Yes |  |
| Training | Batchsize | 8 |
| Iteration | 96000 |  |
| Optimizer | AdamW, $\beta=(0.9,0.999),\epsilon=0.01$ |  |
| Learning Rate | $5\times 10^{-5}\text{ for DINO-Reg},5\times 10^{-4}\text{ for the rest}$ |  |
| Weight Decay | 0.01 |  |
| LR Scheduler | Cosine Scheduler |  |
| Warmup Steps | 9600 |  |
| Weighted Smooth L1 Loss | : $d(x,\hat{x})=\begin{cases}0.5\mathbf{w}\cdot\frac{(x-\hat{x})^{2}}{\beta}&<br>\text{if }|x-\hat{x}|<\beta\\<br>\mathbf{w}\cdot(|x-\hat{x}|-0.5\beta)&\text{otherwise}\end{cases}$ |  |
| $\beta$ | 0.1 |  |
| $\mathbf{w}$ | $w_{4,11}=2,w_{\{0,1,\dotsc,13\}-\{4,11\}}=1$ |  |
| Data Augmentation | ColorJitter | Brightness Range: (0.8, 1.2) |
|  | Contrast Range: (0.7, 1.3) |  |
|  | Saturation: (0.5, 1.5) |  |
|  | Hue: 0.05 |  |
| Randomize Background | Randomize pixels in non-arm-colored background. |  |
|  | Random Apply Probability: 0.4 |  |
| Random Adjust Sharpness | Sharpness Factor: 1.8 |  |
| Sharpness Probability | 0.7 |  |
| Resize | (518, 518) |  |
| Normalization | $mean=[0.485,0.456,0.406]$ |  |
|  |  | $std=[0.229,0.224,0.225]$ |

**Table 7: Composition of Different Models.**
| Model | Arm-Decoupling | Composition |
| --- | --- | --- |
| DINO + DAD (Anypos) | Yes | $(\times 2~{}\text{Arms})$ DINO-Reg + DAD |
|  |  | $(\times 2~{}\text{Wrists})$ DINO-Reg + MLP-regressor |
|  | No | DINO-Reg + DAD |
| DINO + MLP | Yes | $(\times 4~{}\text{Arms \& Wrists})$ DINO-Reg + MLP-regressor |
|  | No | DINO-Reg + MLP-regressor |
| ResNet50 + MLP | Yes | $(\times 4~{}\text{Arms \& Wrists})$ ResNet50 |
|  | No | ResNet50 |

For training on human-collected data, only replace the iteration to 48000, because human-collected data is smaller, thus the epoch will be larger. The model converges after 48000 iterations on human-collected data (validation accuracy: 97.8%).

#### B.2.1 Arm-Decoupled Estimation to Reduce Hypothesis Space

Our approach consists of two stages:
(1) Arm Segmentation:
Leveraging the fact that the pedestal joints remain stable and the robotic arms are uniformly black, we use the pedestal joint pixel as a seed point for flood-fill-based arm segmentation to calculate a split line for the image that divides two arms.
However, if the two arms overlap or part of the arm goes out of the picture, which causes the flood-fill algorithm to fail, we fall back to a default bounding box strategy, cropping the left or right 3/5 of the image based on arm position prior.
(2) Decoupled qpos Estimation:
The segmented left and right arm regions are fed into two independent sub-models, each predicting qpos for their respective arm excluding the gripper.
Specifically, Gripper states are estimated separately by two additional sub-models that take only the image of the left or right wrist as input.
Therefore, by combining split lines with four specialized sub-models, our method achieves arm-decoupled estimation, significantly improving qpos prediction accuracy compared to entangled bimanual approaches.

### B.3 Computation Resources

We conduct the training on a machine equipped with 8 * 80GB NVIDIA Hopper series GPUs, utilizing Accelerate (accelerate,) and Pytorch (paszke2019pytorch,) for multi-GPU parallelism. AnyPos-ATARA required 25 hours to train on 610k pairs of data for 96,000 iterations * 8 batch size * 8 GPUs.

## Appendix C Experimental Details

### C.1 Evaluation of Action Prediction

The parameters of evaluation of action prediction are shown in Table [8](https://arxiv.org/html/2507.12768v1#A3.T8).

**Table 8: Parameters of evaluation of action prediction.**
| Parameter | Value |
| --- | --- |
| Training Dataset | 610k ATARA or 33k Human-Collected Data |
| Test Dataset | 2.5k Manipulation Dataset |
| Evaluation Threshold on Test Dataset | $\text{for}~{}i=6,13,d(a_{i},\hat{a}_{i})<0.5$ |
|  | $\text{others:}~{}d(a_{i},\hat{a}_{i})<0.06$ |

### C.2 Evaluation of Real-World Video Replay

We design our Real-World Video Replay scenario to replicate the daily workspace setting, which includes a typical white laboratory desk, with cluttered objects on the desk, and several computer monitors in the background.
We manually collected 10 long-horizon robot manipulation tasks for real-world video replay, which represent ubiquitous daily household chores.
Each task exhibits sequential dependency, where successful completion of subsequent stages directly depends on the preceding stage’s achievement.

Our 10 tasks include the following tasks and stages:

- •
Make Toast: (1) Pick toast from plate, (2) Insert toast into toaster slot, (3) Push down the toasting lever.
- •
Serve Plates: (1) Grip plate with both hands, (2) Position plate forward on the table.
- •
Microwave Bread: (1) Open microwave door, (2) Retrieve baking tray with bread, (3) Place baking tray inside microwave, (4) Close microwave door.
- •
Organize Tableware: (1) Position bowl on plate, (2) Place fork on right side of plate, (3) Place spoon on left side of plate.
- •
Place carrot: (1) Pick up the carrot, (2) Place the carrot in the basket.
- •
Trash Cubes: (1) Select cube from right side, (2) Dispose cube in trash bin, (3) Select cube from left side, (4) Dispose cube in trash bin.
- •
Fold Clothes: (1) Fold pants by waistband and hem, (2) Fold pants using waistband grip.
- •
Water Plants: (1) Hold water-filled cup, (2) Tilt cup to irrigate plant.
- •
Scrub Plates: (1) Simultaneously grasp sponge and plate, (2) Scrub plate with leftward sponge motion, (3) Scrub plate with rightward sponge motion.
- •
Wipe Table: (1) Maintain firm rag grip, (2) Wipe table surface with rag.

Due to the deterministic and costly nature of the replaying experiment, real-world implementations of these experiments are typically limited to a single trial.

### C.3 Real-World Deployment with Video Generation Model

The experimental setup of real-world deployment with a video generation model follows that of the real-world video replay experiment, except that the videos used are different.
In practical implementation, we finetune Vidu 2.0(bao2024viduhighlyconsistentdynamic,) as our video generation model.
AnyPos processes the generated video frames to infer actions, which are executed by the ALOHA robot.
A task is considered successful if the robot accomplishes it as instructed.

## Appendix D Hardware Details

Tab. [9](https://arxiv.org/html/2507.12768v1#A4.T9) and Fig. [12](https://arxiv.org/html/2507.12768v1#A4.F12) show the detailed information of our robot.

Figure: Figure 12: Hardware features.
Refer to caption: x11.png

**Table 9: Hardware.**
| Parameter | Value |
| --- | --- |
| DoF | $(6+1~{}(\text{gripper}))\times 2=14$ |
| Size | $770\times 700\times 1000$ |
| Arm Weight | $3.9\text{kg}$ |
| Arm Payload | $1500g$ (peak), $1000g$ (valid) |
| Arm Reach | $600mm$ |
| Arm repeatability | $1mm$ |
| Arm working radius | $620mm$ |
| Joint motion range | $J1:180°\sim-120°,J2:0°\sim 210°$ |
|  | $J3:-180°\sim 0°,J4:±90°$ |
|  | $J5:±90°,J6:±110°$ |
| Gripper range | $0\sim 80mm$ |
| Gripper max force | $10N$ |
| Cameras | 3 RGB camears: front$\times 1$, wrist$\times 2$ |

## Appendix E Broader Impacts

This work advances robotic manipulation by introducing AnyPos, a framework for IDM learning from scalable, task-agnostic action data (i.e., ATARA).
The application of this framework in various fields may lead to breakthroughs in automation and intelligent systems, benefiting sectors such as household robotics, healthcare assistance, precision manufacturing, and logistics automation.
By reducing reliance on human demonstrations, AnyPos could accelerate the deployment of adaptable robotic solutions in real-world environments.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see weaknesses.
soundness: 3
strengths: S1) The paper is well written, the proposed pipeline is simple and easy to follow.

S2) The automated data collection framework demonstrates a highly effective strategy for generating large-scale robotic datasets.

S3) The design of arm-decoupled estimation and the direction-aware decoder directly addresses the challenges of high-dimensional action spaces and visual ambiguity in task-agnostic data. These components are empirically shown to be critical for achieving the high-precision action prediction required for real-world deployment.
summary: This paper introduces AnyPos, a framework for task-agnostic embodiment modeling in robotic manipulation. It addresses the critical challenges of data scarcity, task-specificity, and poor cross-platform generalization in robot learning. The core innovation lies in decoupling the learning of physically feasible actions from high-level task semantics. The paper achieve this through a two-stage pipeline: First, an automated, safety-aware exploration process collects a dataset of diverse, feasible robot trajectories without human teleoperation. Second, an inverse dynamics model is trained on this data, employing arm-decoupled estimation and a direction-aware decoder to achieve high-precision action prediction robust to distribution shifts. The resulting embodiment model serves as a reusable "motion prior" that can be seamlessly coupled with various high-level policy models. Experiments show a good improvement in action prediction accuracy and real-world task success rates over strong baselines.
weaknesses: W1) I concerns the practical deployment and generalization of the automated data collection framework. My primary question revolves around the necessity and process of training the RL-based projection policy (`f_RL`). To clarify its scope and limitations: when encountering a new physical scene with different objects and layouts, must a new RL policy be trained from scratch in a simulation that explicitly reconstructs that specific bounded workspace volume? Furthermore, was a single, universal RL policy used to collect all the task-agnostic data for the diverse tasks in the paper, or were multiple scene-specific policies required? Ultimately, I seek to understand the inherent limitations of this approach, specifically, how a policy trained on one scene is expected to perform when deployed in a novel, unseen environment without retraining, particularly regarding its ability to avoid collisions and maintain feasibility.

W2) While the paper's core contribution lies in using improved data to train a more robust inverse dynamics model, the proposed "task-gnostic embodiment modeling" framework ultimately relies on a pipeline combining a video generation model (VGM) with this learned model. This architecture appears to have limited practical utility due to its inherent susceptibility to error propagation. The system's performance is heavily contingent on the quality of the generated videos, where any inaccuracies in the predicted future frames are directly translated into action errors by the inverse dynamics model. A shortcoming of the work is the lack of a detailed analysis quantifying this compounding error and visualizing the pipeline's robustness.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: It seems AnyPos depends only on URDF/kinematics and can be “replayed” for new viewpoints. How robust is the IDM to camera shifts?
soundness: 3
strengths: The paper provides a large scale dataset by RL-based exploration, and then recording 610k safe, diverse image-action pairs in 10 hours, which is claimed to be 30× faster than human teleoperation.

The quantitative results for real-world validation looks strong, with high replay success, the demo video also shows performance, though with some failure cases.

The paper presents detailed implementation and hyperparameter reporting.
summary: The paper introduces an embodiment modeling framework that learns a vision-to-action mapping directly from large-scale, automatically collected trajectories. It employs an RL-based explorer to sample EEF targets across the 3D workspace and convert them into feasible joint configurations, creating a large dataset. The inverse dynamics model trained on this data maps visual observations to joint positions with good precision, aided by an arm-decoupled architecture and a Direction-Aware Decoder. The system can then pair this embodiment model with high-level policies such as video-generation or VLA models, achieving real-world bimanual manipulation success while requiring no human teleoperation.
weaknesses: The paper does not provide comparison when replacing learned inverse dynamics model (IDM) with traditional collision-free inverse kinematics (IK). most baselines are vision encoders (ResNet or DINOv2) rather than control pipelines, so the benchmark scope is narrow. Without such an ablation, it is unclear whether the learned model actually outperforms or simply replicates IK performance under noise and multi-arm constraints, since IK is guaranteed to be generalizable but a learned model is not

Variables are introduced abruptly in introduction without clear linkage to the system. The overall abstraction and introduction could better define these terms and explicitly describe the model’s input and output, integrating this explanation with Figure 1 to clarify the pipeline flow.

The cross embodiment experiment is limited given the paper claims embodiment-agnostic modeling

Arms overlap with objects or arms or move out of frame -- conditions that are common in cluttered manipulation scenarios, which the model seems not be able to address.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 1
questions: 1. How's each component contribute to the final performance? How does baselines implemented? Do they similar architecture?
soundness: 2
strengths: 1. The method allows random exploration without direct human supervision, which relax the requirement on human demonstration.
summary: The paper proposed a novel method to collect random exploration data to learn good representation for the bi-manual manipulation tasks. THe method use a biased sampling stragtegy for exploration and learns a good representation via inverse dynamics model. Then the model is used to learn downstream imitation learning policy. Experiments show better performance than learning from scratch with pure teleoperation data and previous approach.
weaknesses: 1. The tasks are not safety critical. The method is hard to generalize to general-purpose tasks, like ones involving safety concerns. Random exploration is not applicable.
2. The writing is poor. The beginning of the paper involves too much distraction of mathematical formulation. It is more straightforward to describe the method in an intuitive way.
3. No ablation on the model design. Why all components are needed?

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Why is reinforcement learning needed for deterministic action mapping, and what concrete advantages does it offer over a straightforward inverse-kinematics or behaviour-cloning approach with safety and collision constraints?
2. Deterministic vs. probabilistic formulation: The paper presents a probabilistic factorization suggesting a distributional treatment of actions, but the actual system predicts a single deterministic joint configuration. How is this formulation relevant if the model never models uncertainty or multiple feasible actions? Since the “world model” is deterministic, how does this help separate “what to do” from “what is physically feasible,” and what additional insight or capability does this probabilistic claim really provide?
3. How does the model handle camera-view and embodiment differences, given that the state representation is trained purely in the 2D image domain—does it rely on multi-view training, 3D understanding, or re-collection for new viewpoints?
4. What evidence supports that separating feasibility from task semantics improves transfer, and can the embodiment model generalize across unseen tasks or robot morphologies without retraining?
5. What is the real contribution of the Direction-Aware Decoder (DAD)? The paper shows marginal experiments but lacks fine-grained ablation; if it is mainly an engineering tweak, why treat it as a core architectural innovation?
6. What concrete metrics and comparisons demonstrate that the RL-generated dataset is superior (in workspace coverage, safety rate, or diversity) to simpler IK-based or rule-based data generation pipelines?
soundness: 3
strengths: The paper’s strengths lie in its clear problem framing and modular design—it separates physical feasibility learning from task semantics through a well-structured pipeline that’s easy to follow. The authors present a fully automated, high-throughput data collection system that efficiently explores the workspace without teleoperation, coupled with a safety-aware exploration scheme that enforces collision avoidance, joint limits, and inter-arm constraints. The precision-oriented inverse dynamics model introduces practical architectural tweaks, such as arm-decoupled estimation and a direction-aware decoder, which demonstrably improve stability and accuracy. Finally, the real-robot replay results show that the learned embodiment prior can be executed reliably, suggesting the overall framework has strong potential as a practical foundation for scalable, modular robot learning.
summary: The paper proposes AnyPos, a framework for learning robot actions in a task-agnostic way. Its core contribution lies in two connected parts: first, an automated exploration pipeline that uses a reinforcement-trained mapper and safety constraints to sample the robot’s workspace and generate large quantities of collision-free image–action pairs without teleoperation or task labels; second, an inverse-dynamics embodiment model that predicts feasible joint positions from visual inputs using an arm-decoupled design and a direction-aware decoder for sub-degree precision. Together, these components aim to provide a reusable “feasibility prior” that can later be coupled with high-level policy or video-generation models to execute diverse manipulation tasks more efficiently than task-specific datasets.
weaknesses: The paper’s main weaknesses lie in its limited empirical validation and unclear justification of core design choices. While the motivation is compelling, the work never actually demonstrates that separating semantics from feasibility improves transfer — all experiments are confined to a single robot, with no cross-task, cross-embodiment, or cross-view evaluations. The decision to use reinforcement learning for deterministic action generation is conceptually weak and lacks comparison against straightforward baselines like inverse kinematics or behavior cloning with physical constraints. The proposed Direction-Aware Decoder adds engineering complexity without clear theoretical grounding or fine-grained ablations to prove necessity. Moreover, the probabilistic decomposition introduced early in the method is never operationalized in training, leaving a gap between the stated formulation and the implemented model. Overall, the paper presents a strong motivation but delivers mostly incremental engineering under a broader conceptual narrative that remains unverified.


Decision: Reject