NoTVLA: Narrowing of Dense Action Trajectories for Generalizable Robot Manipulation

## Abstract

Abstract Vision-Language-Action (VLA) models represent a pivotal advance in embodied intelligence, yet they confront critical barriers to real-world deployment, most notably catastrophic forgetting. This issue stems from their overreliance on continuous action sequences or action chunks, which inadvertently create isolated data silos that disrupt knowledge retention across tasks. To tackle these challenges, we propose the Narrowing of Trajectory VLA (NoTVLA) framework: a novel approach that narrows its focus to sparse trajectories, thereby avoiding the catastrophic forgetting associated with dense trajectory fine-tuning.
A key innovation of NoTVLA lies in its trajectory planning strategy: instead of centering on the target object’s trajectory, it leverages temporal compression and spatial reasoning pruning specifically for the robot end effector’s trajectory. Furthermore, training is conducted using these sparse trajectories rather than dense action trajectories, an optimization that delivers remarkable practical advantages with better performance in zero-shot. In multi-task evaluation scenarios, NoTVLA achieves superior performance and generalization compared to π 0 \pi_{0} while operating under two critical constraints: it uses over an order of magnitude less computing power than π 0 \pi_{0} and requires no wrist-mounted camera.
This design ensures that NoTVLA’s operational accuracy closely approximates that of single-task expert models. Crucially, it also preserves the model’s inherent language capabilities, enabling zero-shot generalization in specific scenarios, supporting unified model deployment across multiple robot platforms , and fostering a degree of generalization even when perceiving tasks from novel perspectives .

## 1 Introduction

Vision-Language-Action (VLA) models have emerged as a transformative force for embodied intelligence, heralding a new era of integrated perception, reasoning, and control (Kim et al., 2024; Brohan et al., 2023; Driess et al., 2023; Alayrac et al., 2022). By coupling large-scale multimodal reasoning engines with action experts, these systems aspire to endow robots with the capacity to operate robustly in open-ended, unstructured environments. Yet, despite rapid progress, several obstacles still impede reliable deployment. Foremost among them is catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017; Zenke et al., 2017), which is aggravated in current pipelines by an overreliance on dense, continuously parameterized action trajectories (or “action chunks”). Such representations foster isolated data silos: when finetuned on a new task, the model may overwrite previously consolidated competencies, degrading prior-task performance.

The root cause of this fragility often lies in the prevailing training paradigm: repeated finetuning on voluminous dense trajectories is computationally expensive and underutilizes the high-level semantic abstraction and reasoning latent in Vision-Language Models (VLMs) (Alayrac et al., 2022; Reed et al., 2022; Hu et al., 2022). Moreover, tightly entangling the VLM and low-level controllers hampers systematic generalization across embodiments and tasks. To address these issues, we introduce the Narrowing of Trajectory VLA (NoTVLA) framework (hereafter, the proposed framework) as shown in Figure [1](https://arxiv.org/html/2510.03895v1#S1.F1), which rethinks representation granularity. Its central premise is to narrow supervision to sparse, semantically aligned segments of behavior, mitigating catastrophic forgetting while lowering adaptation cost.

Figure: Figure 1: Overview of the NoTVLA framework. NoTVLA addresses catastrophic forgetting in Vision-Language-Action (VLA) models by replacing dense action trajectories with sparse, semantically aligned keyframes. Large-scale trajectory data (8.1K trajectories, 100K keyframes) are processed via kinematics-based keyframe and sub-keyframe selection. Instructions and RGB inputs are encoded with Qwen VL 2.5 (7B) to predict anchor points, which are combined with depth queries for anchor-conditioned token generation. A spline-based detokenizer converts these discrete action tokens into smooth, high-frequency trajectories for closed-loop robot control. The framework generalizes across robot embodiments and tasks, supporting zero-shot execution, multi-view robustness, and real-world deployment.
Refer to caption: notvla/Figures/finalimage/NoTvla.png

A key innovation of the proposed framework is a trajectory planning and abstraction strategy that shifts focus from dense object-centric rollouts to a temporally compressed, spatially pruned representation of the robot end-effector path. Through temporal compression and spatial reasoning-based pruning, the method yields leaner yet decision-sufficient action sequences, enabling more sample- and compute-efficient refinement (Li et al., 2023; Liu et al., 2023b). Training on these sparse trajectories—rather than on full-resolution dense streams—improves retention and cross-task transfer. Empirically, the approach attains superior multi-task generalization relative to contemporary VLA baselines (e.g., $\pi_{0}$) while using over an order of magnitude less compute and no wrist-mounted camera, yet approaching the operational accuracy of specialized single-task experts.

Crucially, the framework preserves intrinsic language and reasoning capabilities of the underlying VLM, supporting zero-shot adaptation in selected novel settings (Brohan et al., 2023; Driess et al., 2023). It further enables unified deployment across heterogeneous robot embodiments and sustains competence under viewpoint shifts, indicating robustness to perceptual distributional variation.

In summary, our contributions are as follows:

- •
We decouple the high-level VLM from low-level action experts, achieving higher embodied-task success while substantially reducing finetuning compute.
- •
We show that sparse, semantically pruned trajectory supervision enhances both cross-embodiment and cross-task generalization, alleviating catastrophic forgetting.
- •
Unlike conventional tightly coupled designs, our framework preserves inherent vision-language reasoning, enabling complex instruction following and multi-turn interaction without degradation.

## 2 Related Works

### 2.1 Spatial Reasoning for Generalizable Manipulation

Effective manipulation requires robust 3D spatial reasoning. One line of research embeds this capability by fine-tuning models on 3D-centric datasets (Wu et al., 2025a; Cai et al., 2024; Liu et al., 2025) or by directly processing dense 3D inputs like point clouds (Zhen et al., 2024; Yang et al., 2025b; Song et al., 2025c), which can be computationally intensive and limit modularity. Another direction provides controllers with explicit geometric guidance like keypoints (Manuelli et al., 2019; Yuan et al., 2024) or affordances (Wu et al., 2025b). Our work advances this latter principle through a decoupled anchor-based depth inference mechanism. Instead of requiring the VLA to process full 3D scenes, we use a lightweight head to predict a 2D anchor, inject depth from an external source, and then generate actions. This factorized design significantly reduces the model’s perceptual burden and enhances modularity, allowing for flexible integration across different robot platforms and sensor configurations.

### 2.2 Hierarchical Architectures and Intermediate Representations

To manage the complexity of end-to-end models, hierarchical frameworks that decouple a high-level planner from a low-level action expert have become prominent (Black et al., 2410; Intelligence, 2025; Bjorck et al., 2025). The efficacy of this architecture hinges on the intermediate representation connecting the two modules. While prior works have used abstract sub-goals (Li et al., 2024; de Bakker et al., 2025) or latent features (Li et al., 2025) or explicit trajectory (Yang et al., 2025a), their semantic ambiguity can burden the action expert. Our framework uses explicit 3D waypoints as a clear interface, but more importantly, we introduce a novel spline-based action detokenizer to bridge the frequency gap. This module translates low-frequency waypoints from the VLA into a smooth, high-rate, and jitter-free robot trajectory online, directly addressing a core challenge in deploying hierarchical systems.

### 2.3 From Dense to Sparse Action Representations in VLA Models

Recent Vision-Language-Action (VLA) models have made significant strides by mapping multimodal prompts to dense, high-frequency action sequences (Brohan et al., 2022; 2023; Kim et al., 2024; Team et al., 2024; Song et al., 2025c; Li et al., 2025; Wen et al., 2025; Zhao et al., 2025; Song et al., 2025b; a; Fan et al., 2025; Wang et al., 2025). A lot of datasets (Khazatsky et al., 2025; Mu et al., 2021; Walke et al., 2023; Liu et al., 2023a; James et al., 2019; AgiBot-World-Contributors et al., 2025; Mu et al., 2025; Chen et al., 2025b) are built under this VLA action pattern. However, this end-to-end paradigm’s reliance on dense supervision imposes immense data requirements and can lead to challenges like catastrophic forgetting (French, 1999). This has spurred a shift towards sparse action representations, which offer a more tractable interface between planning and control. While methods like uniform downsampling risk losing critical motion details, and learned segmentation models add complexity, our approach introduces a kinematics-based keyframe selection protocol. By identifying keyframes based on physical properties like acceleration and gripper state changes, we achieve a meaningful and efficient trajectory sparsification that preserves task-critical information without requiring additional learned components.

## 3 Method

As shown in Figure [2](https://arxiv.org/html/2510.03895v1#S3.F2) we present a VLA framework that combines the hardware operability of classical controllers with the generalization ability of VLMs under a unified, cross-robot trajectory annotation.
The system consists of: (1) anchor-based depth inference that equips VLMs with efficient and scalable depth reasoning, (2) a spline-based action detokenizer that turns discrete tokens into smooth, high-rate trajectories, and (3) kinematics-based keyframe selection to preserve task-critical motion.
During training, we employ a kinematics-based keyframe selection method (Section [3.2](https://arxiv.org/html/2510.03895v1#S3.SS2)) to provide sparse supervision for our VLA model, using only the most critical end-effector poses. At inference time, the procedure unfolds in two stages. First, the model generates a planned trajectory via anchor-based depth inference—a two-step, text-guided prediction detailed in Section [3.1](https://arxiv.org/html/2510.03895v1#S3.SS1). Subsequently, this trajectory is processed by a spline-based action detokenizer to produce a sequence of smooth, executable actions.

Figure: Figure 2: Anchor prediction and token generation in NoTVLA. Instructions with RGBD input yield 2D and depth anchors, which condition action tokens. These tokens are converted into trajectories aligned with predicted anchors for precise manipulation.
Refer to caption: notvla/Figures/finalimage/APP_ACTG.png

### 3.1 Anchor-Based Depth Inference

We turn absolute depth prediction into anchor-conditioned reasoning: a 2D anchor localizes the task-relevant region; an external depth source supplies its depth; the VLA then generates a full multi-modal action token stream conditioned on this tri-modal cue (image, language, anchor-with-depth). This decoupling lowers vision model burden and improves cross-view robustness.

Anchor Point Prediction (APP). At *inference time* APP receives only the RGB image and language instruction and outputs a *single* 2D anchor (being the projection of the 3D pose of the gripper end effector in the key frame onto the image plane):

$$ $(u_{a},v_{a})=\mathcal{M}(\mathbf{I},\mathbf{l}),\qquad(u_{a},v_{a})\in[0,W)\times[0,H).$ (1) $$

Training can supervise this point as (i) the object center, (ii) the first contact pixel, or (iii) a task-specific salient location. No depth is produced inside APP. The anchor depth is then obtained from an *external* source:

$$ $d_{a}=\mathcal{S}_{\text{depth}}(\mathbf{I},u_{a},v_{a}),$ (2) $$

where $\mathcal{S}_{\text{depth}}$ can be: (a) an RGB-D or LiDAR sensor ($d_{a}=D(u_{a},v_{a})$) or (b) a monocular estimator. The final depth-augmented anchor is $a=(u_{a},v_{a},d_{a})$.

Anchor-Conditioned Token Generation (ACTG). ACTG is a training method that inputs the depth-augmented anchor $\text{ANCHOR}(u_{a},v_{a},d_{a})$ to enhance the ability of 3D inference of VLM. At inference it consumes $(\mathbf{I},\mathbf{l},a)$ and autoregressively outputs a trajectory token sequence containing depth, image-plane coordinates, gripper, and pose modalities: depth $\mathcal{D}=\{d_{i}\}$; image UV $\mathcal{U}=\{(u_{i},v_{i})\}$; gripper state $\mathcal{G}=\{g_{i}\}$; gripper pose $\mathcal{R}=\{\bm{r}_{i}\}$ with Euler angles $\bm{r}_{i}=[\theta_{xi},\theta_{yi},\theta_{zi}]^{\top}$, which is serialized as

$$ $\mathbf{S}=[\text{CLS},\;\text{IMG},\;\text{TXT},\;\text{ANCHOR}(u_{a},v_{a},d_{a}),\;B_{1},\;\ldots,\;B_{N},\;\text{EOS}].$ (3) $$

$$ $B_{i}=(\mathcal{D}_{i},\;\mathcal{U}_{i},\;\mathcal{G}_{i},\;\mathcal{R}_{i}).$ (4) $$

Reconstructed 3D waypoints use camera intrinsics:

$$ $\bm{p}_{i}=\tilde{d}_{i}\,\bm{K}^{-1}[\tilde{u}_{i},\;\tilde{v}_{i},\;1]^{\top}.$ (5) $$

The sequence $\mathbf{S}$ starts at the current frame and ends at another keyframe (Sec. [3.2](https://arxiv.org/html/2510.03895v1#S3.SS2)). Optionally, depths (and/or UV) can be modeled relative to the anchor: $d_{i}=d_{a}+\Delta d_{i}$ with the same quantization applied to $\Delta d_{i}$ over a symmetric range; we keep absolute depth for simplicity.

Figure: Figure 3: Keyframe selection based on gripper posture. The images show the gripper’s movements over time, with keyframes selected based on the open/close states of the left and right arms. The blue and red lines represent the gripper’s trajectory, with the corresponding open and close states annotated, highlighting the transitions between key poses during object manipulation.
Refer to caption: notvla/Figures/select_keyframes.png

### 3.2 Kinematics-Based Keyframe Selection

We segment raw demonstrations into logical phases (Chen et al., 2025a), as shown in Figure [3](https://arxiv.org/html/2510.03895v1#S3.F3). Let the end-effector pose be
$\bm{T}(t)=[x(t),y(t),z(t),\theta_{x}(t),\theta_{y}(t),\theta_{z}(t)]^{\top}$ and the gripper state $G(t)\in\{0,1\}$. A keyframe at time $t_{k}$ satisfies

$$ $\big\|\ddot{\bm{T}}(t_{k})\big\|_{2}>\alpha\;\lor\;\lim_{\epsilon\to 0^{+}}G(t_{k}{-}\epsilon)\neq\lim_{\epsilon\to 0^{+}}G(t_{k}{+}\epsilon),$ (6) $$

where $\alpha$ is an threshold. In practice, accelerations are computed via simple finite differences, and the threshold (and any component weights) are hand-designed per task; no additional filtering or post-processing is applied.

We discretize the motion between consecutive keyframes and express all poses in the camera frame following the RoboTwin convention (Mu et al., 2024). To reduce information loss, we insert sub-keyframes (Pertsch et al., 2020) and uniformly downsample each segment with constant $N$ and equal temporal spacing:

$$ $\bm{T}_{\text{down}}=\{\bm{T}_{i}\}_{i=0}^{N-1}=\mathrm{UniformSample}(\bm{T}_{k},\bm{T}_{k+1},N).$ (7) $$

This preserves temporal coherence while lowering compute during training.

### 3.3 Spline-Based Action Detokenizer

Robots require smooth, millisecond-level control, whereas VLMs emit tokens at low rates ($\sim$1–2 Hz). We bridge this gap with interpolation and closed-loop updates (Miura et al., 2024; Urain et al., 2022). The token sequence $\{B_{i}\}$ produced by ACTG, consisting of waypoints in the camera frame, is first transformed into the robot’s base (world) frame. Let $\bm{T}_{c}^{w}$ be the transformation from camera to world coordinates. Each waypoint $\bm{p}_{c,i}$ is converted to the world frame:

$$ $\bm{p}_{w,i}=\bm{T}_{c}^{w}\bm{p}_{c,i}.$ (8) $$

The resulting 3D waypoints $\{\bm{p}_{w,i}\}$ are then used for trajectory generation. We employ cubic spline (Farin, 2002) interpolation for the XYZ position to ensure a smooth path:

$$ $\bm{P}(t)=\text{Spline}(\{\bm{p}_{w,i}\},t).$ (9) $$

For orientation, the Euler angles $\{\bm{r}_{i}\}$ are first converted to quaternions $\{\bm{q}_{i}\}$. We then use Spherical Linear Interpolation (SLERP (Shoemake, 1985)) between consecutive quaternions to generate a smooth and unambiguous rotation trajectory:

$$ $\bm{q}(t)=\text{SLERP}(\bm{q}_{i},\bm{q}_{i+1},\frac{t-t_{i}}{t_{i+1}-t_{i}}),\quad t\in[t_{i},t_{i+1}].$ (10) $$

This combined approach ensures a smooth high-frequency trajectory in both position and orientation, suitable for robot execution.

However, this procedure will cause an error on the trajectory. A cubic B-spline reconstruction (Farin, 2002; Piegl & Tiller, 1997; de Boor, 2001)

$$ $\hat{\bm{T}}(t)=\sum_{j=0}^{N}B_{j,3}(s(t))\,\bm{T}_{j}$ (11) $$

exhibits error bounded by

$$ $\Delta\bm{T}_{\text{sub}}(t)\leq K\,\max\|\bm{T}^{(4)}(t)\|_{2}\Big(\tfrac{\Delta t}{M{+}1}\Big)^{4},\;$ (12) $$

indicating sub-keyframes near keyframes sharply reduces error. In practice, we sample at least 10 sub-keyframes between keyframes, which makes the inference close to keyframe very close to gt. The design deploys on Franka Reach, Aloha–AgileX, PiPER, and AGIBOT G1 with closed-loop stability. The details are in the Appendix.

## 4 Experiments

### 4.1 Comparison with Existing Methods

#### 4.1.1 Comparison of Success Rates

In a constrained sensing setup using only head-mounted monocular RGBD and trained across multiple tasks, our model attains accuracy close to single-task experts (e.g., ACT) and classical VLA (e.g., $\pi_{0}$) as shown in Table [1](https://arxiv.org/html/2510.03895v1#S4.T1), shows clear advantages on longer-horizon tasks due to preserved VLM reasoning, but this strength is underrepresented by short-horizon benchmarks and limited when relevant objects fall outside the image.

**Table 1: Performance comparison between expert models and generalist models on various robotic tasks in RoboTwin 2.0 (Mu et al., 2025). Full table is in the Appendix [E.1](https://arxiv.org/html/2510.03895v1#A5.SS1)**
| Task Name | Expert Models | Generalist Models |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ACT | DP | DP3 | $\pi_{0}$ | RDT | Ours |  |
| click alarmclock | 0.32 | 0.61 | 0.77 | 0.63 | 0.58 | 0.78 |
| click bell | 0.58 | 0.54 | 0.90 | 0.44 | 0.18 | 0.94 |
| grab roller | 0.94 | 0.98 | 0.98 | 0.96 | 0.50 | 0.97 |
| handover mic | 0.85 | 0.53 | 1.00 | 0.98 | 0.13 | 0.99 |
| move playingcard away | 0.36 | 0.47 | 0.68 | 0.53 | 0.03 | 0.77 |
| open laptop | 0.56 | 0.49 | 0.82 | 0.85 | 0.15 | 0.57 |
| press stapler | 0.31 | 0.06 | 0.69 | 0.62 | 0.34 | 0.94 |
| Shake bottle | 0.74 | 0.65 | 0.98 | 0.97 | 0.31 | 0.94 |

In the AGIBOT Challenge, our code was evaluated by the official server and demonstrated clear improvements over the UniVLA baseline. Overall, our approach outperformed the baseline across multiple tasks, with particularly pronounced gains on Open drawer and store items and Pickup items from the freezer, and these advantages were confirmed in the official evaluation as shown in Table [2](https://arxiv.org/html/2510.03895v1#S4.T2).

**Table 2: AGIBOT Challenge Official Test Results. Columns 1-10 correspond to the 10 official tasks in the challenge. Details in the Appendix [A.4.1](https://arxiv.org/html/2510.03895v1#A1.SS4.SSS1).**
| Method | Total | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| UniVLA | 2.795 | 0.097 | 0.020 | 0.033 | 0.350 | 0.260 | 0.400 | 1.000 | 0.080 | 0.375 | 0.180 |
| Ours | 3.697 | 0.161 | 0.304 | 0.350 | 0.320 | 0.448 | 0.510 | 0.600 | 0.262 | 0.350 | 0.392 |

On the other hand, as we can see in Figure [4](https://arxiv.org/html/2510.03895v1#S4.F4), we use far fewer training steps than many alternatives and avoid additional single-task fine-tuning, which demonstrates reliable multitask capability. This efficiency also leaves headroom for scaling while maintaining strong language competency. Compared with other VLA models such as $\pi_{0}$, our approach thus offers a more scalable solution with efficient language processing. Although our setting does not require single-task fine-tuning, it remains a technique from which prior VLA paradigms can still benefit. Together, these design choices deliver higher language-processing efficiency and a cleaner scaling path while maintaining strong language competency.

Figure: Figure 4: Training steps and average success rate of different works. NoTVLA uses 7B weight, close to RDT and $\pi_{0}$. The bubble size refer to the single task training step of different works. NoTVLA without single task training still performs better than other models.
Refer to caption: notvla/Figures/test_images/compare.png

Building on this efficiency perspective, we further examine the training cost and fine-tuning requirements. More details of different steps are in the Appendix [C.1](https://arxiv.org/html/2510.03895v1#A3.SS1) No additional single-task fine-tuning was required, as the marginal gains proved limited. This design also brought a substantial reduction in training cost: for instance, training on Robotwin required only 32 GPU hours, and even with the compute budget reduced to roughly 8 GPUs, the model already demonstrated execution capability on most tasks (see ablation study for details). To contextualize efficiency, the left panel of the bubble chart summarizes total training steps, single-task fine-tuning steps, and efficiency by steps: the x-axis denotes overall training steps, bubble size represents single-task fine-tuning steps (zero for NoTVLA), and the y-axis shows success rate under the given budgets. For comparison, ACT, DP, and DP3 are single-task expert models, whose total steps scale linearly with the number of tasks, while $\pi_{0}$ and RDT are general-task models that combine full-task fine-tuning with per-task fine-tuning proportional to the task count.

#### 4.1.2 Zero-shot Comparison

Our method demonstrates strong generalization, performing well in Robotwin’s hard scenarios, including unseen instructions and backgrounds. It responds to color cues absent from training and handles task descriptions that are the inverse of those seen during training. Additionally, it exhibits zero-shot transfer to abstract concepts (e.g., flags) not included in the training set. The results are in Table [3](https://arxiv.org/html/2510.03895v1#S4.T3).

For example, all training instances of stack block two use a green-over-red ordering. At test time, we invert the instruction (stack red on green) and introduce new colors and descriptions. Our model maintains strong performance under these changes, while other models fail to show the same zero-shot robustness.

To fairly compare zero-shot generalization, we collect pre-training and fine-tuning data from Robotwin and train both the open-source $\pi_{0}$ and our model. To avoid overtraining as a cause of weak generalization, we evaluate two $\pi_{0}$ checkpoints with different training budgets, both of which perform well on move pillbottle pad and stack blocks two. We then test these $\pi_{0}$ checkpoints and our model on zero-shot tasks, inverting instruction semantics to probe adversarial language and replacing block colors to test understanding of more complex descriptions. Compared to $\pi_{0}$, our model remains responsive to adversarial instructions and complex language. $\pi_{0}$, while strong on single tasks, shows limited interaction flexibility and suffers from catastrophic forgetting when scaling to multiple tasks. $\pi_{0}$’s subtask decomposition also assumes strictly defined subtasks, risking the loss of crucial linguistic details (e.g., color), which constrains compositionality. For low-level skills, complex instruction combinations are often necessary, and atomic subtasks reduce coverage of diverse scenarios. As for different models, they also have different performances. We have tried Qwen2.5-VL 7B, 3B and InterVL 2.5 1B. Details are shown in the Appendix [A.3](https://arxiv.org/html/2510.03895v1#A1.SS3).

**Table 3: Performance comparison of tasks in the training set and zeroshot tasks. Details of trained tasks are in the Appendix. Details of zero-shot tasks are in the Appendix [A.4.3](https://arxiv.org/html/2510.03895v1#A1.SS4.SSS3)**
| Task Name | $\pi_{0}$-30000 | $\pi_{0}$-5000 | NoTVLA 7B |
| --- | --- | --- | --- |
| Trained Tasks |  |  |  |
| place mouse pad | 23% | 22% | 23% |
| stack blocks two | 90% | 65% | 87% |
| Zero-shot Tasks |  |  |  |
| stack blocks two (inverse) | 0.0% | 0.0% | 70% |
| stack random color blocks | 30% | 18% | 69% |
| move block random color pad | 10% | 9% | 18% |
| stack numberblocks two | 0.0% | 0.0% | 58% |
| stack random blocks two from three | 1.0% | 0.0% | 57% |
| place block flagpad | 32% | 55% | 78% |
| place block flagpad (choice) | 25% | 56% | 62% |

#### 4.1.3 Trajectory Planning Comparison

We validate our framework by comparing its trajectory generation to the planning-based Magma planning trajectory shape across geometric and temporal metrics in the OOD field trajectory in RoboTwin, and it outperforms on all: higher coverage (F1 0.955 vs 0.928), markedly lower DTW (0.456 vs 0.835) and Fréchet distance (0.104 vs 0.230), reduced start/end errors by 46%/59%, consistently smaller orthogonal distances, and favorable Hausdorff results. These gains show our spline-based detokenizer produces more accurate, consistent trajectories, effectively bridging low-frequency model outputs to high-frequency robot control. Full comparison is in the Appendix [C.2](https://arxiv.org/html/2510.03895v1#A3.SS2).

### 4.2 Ablations and Analyses

#### 4.2.1 Depth Reasoning Ablation

In NoTVLA, Anchor Point Prediction (APP) and Anchor-Conditioned Token Generation (ACTG) are introduced to enhance spatial reasoning. APP instructs the model to select a perceptual anchor within the image and query the depth at that specific point, while ACTG provides a single-point depth token to serve as the foundation for subsequent reasoning. A central question arises: do explicit depth inputs, as opposed to monocular depth inference performed by the Vision-Language Model (VLM), improve performance in manipulation tasks?

To address this, we evaluate three distinct variants across a range of representative tasks, including spatiotemporal semantics, object recognition, and long-horizon sequences:
(1) w.APP w.ACTG, which first outputs an anchor point and acquires the associated depth.
(2) w.APP w/o.ACTG, which outputs a anchor point but lacks depth from the sensor.
(3) w/o.APP w/o.ACTG, which performs direct reasoning without explicit anchor point and depth information.

Our results that are shown in Table LABEL:tab:ablation_study demonstrate that the inclusion of explicit depth inputs significantly improves success rates, particularly for long-horizon tasks. These tasks involve broader depth ranges and intraclass scale variation, both of which induce perspective distortion that complicates monocular depth estimation. In addition, such tasks often require multiple rounds of spatial reasoning, where reference points play a critical role in stabilizing repeated inferences. In particular, APP alone further enhances performance in cespatio-temporalemporal tasks, likely due to the proximity of the predicted anchor to the manipulation locus, providing a consistent reference for subsequent reasoning.

**Table 4: Performance comparison of tasks across different methods.**
| Task Name | NoTVLA<br>w. APP w. ACTG | NoTVLA<br>w/o APP w/o ACTG | NoTVLA<br>w. APP w/o ACTG |
| --- | --- | --- | --- |
| Object Recognition |  |  |  |
| place a2b left | 0.38 | 0.28 | 0.30 |
| place a2b right | 0.34 | 0.24 | 0.32 |
| place object scale | 0.43 | 0.39 | 0.36 |
| Long-term Tasks |  |  |  |
| blocks ranking size | 0.53 | 0.33 | 0.28 |
| blocks ranking rgb | 0.76 | 0.66 | 0.59 |
| Stack block three | 0.49 | 0.32 | 0.30 |
| Stack bowl three | 0.83 | 0.78 | 0.68 |

#### 4.2.2 Generalization of Novel Perspectives

Table [5](https://arxiv.org/html/2510.03895v1#S4.T5) compares task performance between In Domain (ID) camera views seen during training and Out of Domain (OOD) novel views. The metric is the success rate for each task under these two conditions. The results demonstrate that our model maintains strong performance even when faced with novel perspectives, with only a minor drop in success rate for most tasks (e.g., Stack block three drops from 0.35 to 0.31; blocks ranking rgb from 0.50 to 0.45). As described in the method section, we decouple 3D reasoning from direct depth map prediction and all the VLM inference are in the camera perspective. This approach avoids forcing the VLM to learn a dense, view-dependent action representation. Instead, it learns to reason about task-relevant 3D geometry relative to a single, robustly localized anchor. This decoupling significantly enhances the model’s ability to generalize across different camera viewpoints, as the core reasoning process is less sensitive to changes in perspective.

#### 4.2.3 Performance of Different Training Steps

The task success rate, evaluated at different training steps (from 500 to 4000), was on mutlitask in the Appendix [C.1](https://arxiv.org/html/2510.03895v1#A3.SS1). The tasks are categorized into short, medium, and long horizons, representing increasing levels of complexity and action sequence length. Our experiments shows that performance does not uniformly increase with more training steps; instead, it often fluctuates. For instance, the grab roller task achieves a perfect success rate at only 500 steps but declines thereafter, suggesting potential overfitting. Similarly, other tasks like place empty cup and place bread basket exhibit non-monotonic performance curves. On the other hands, task horizon significantly impacts performance. Short-horizon tasks like grab roller and place empty cup achieve high success rates early in training, as they involve simpler motion sequences. In contrast, long-horizon tasks such as stack blocks three start with low success rates and require more training to show improvement. This is because our method relies on Kinematics-Based Keyframe Selection to segment demonstrations. Long-horizon tasks has more keyframes, making the sequence modeling problem for the VLM more challenging.

## 5 Conclusion

We present a VLA framework that bridges VLM high-level reasoning with precise low-level control via three components—anchor-based depth inference for simplified 3D perception, kinematics-driven keyframe selection, and a spline-based detokenizer for smooth, high-frequency trajectories. Across diverse manipulation tasks, it matches or surpasses expert and generalist baselines, trains efficiently with fewer steps and no task-specific finetuning, and generalizes zero-shot to novel instructions, objects, and scenarios. Limitations include sensitivity to objects outside the camera’s field of view and benchmarks that underrepresent long-horizon reasoning; future work will enhance sensing and evaluate on more complex, long-horizon tasks toward scalable, efficient generalist robots.

## References

- AgiBot-World-Contributors et al. (2025)
AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang, Shukai Yang, Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu.
Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.
*arXiv preprint arXiv: 2503.06669*, 2025.
- Alayrac et al. (2022)
J.-B. Alayrac, J. Donahue, et al.
Flamingo: A visual language model for few-shot learning.
*Advances in Neural Information Processing Systems*, 2022.
- Bjorck et al. (2025)
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al.
Gr00t n1: An open foundation model for generalist humanoid robots.
*arXiv preprint arXiv:2503.14734*, 2025.
- Black et al. (2410)
K. Black et al.
Pi-0: A dual-system framework for robotics, 2410.
- Brohan et al. (2022)
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.
Rt-1: Robotics transformer for real-world control at scale.
*arXiv preprint arXiv:2212.06817*, 2022.
- Brohan et al. (2023)
Jake Brohan, Denis Driess, Fisher Yu, Yirui Zhu, Shuran Song, Emanuel Todorov, Pieter Abbeel, and Sergey Levine.
Rt-2: Vision-language-action models transfer web knowledge to robotic control.
*arXiv preprint arXiv:2306.14270*, 2023.
- Cai et al. (2024)
Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao.
Spatialbot: Precise spatial understanding with vision language models.
*arXiv preprint arXiv:2406.13642*, 2024.
- Chen et al. (2025a)
Kaiyuan Chen, Shuangyu Xie, Zehan Ma, Pannag R Sanketi, and Ken Goldberg.
Robo2vlm: Visual question answering from large-scale in-the-wild robot manipulation datasets.
*arXiv preprint arXiv:2505.15517*, 2025a.
- Chen et al. (2025b)
Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Qiwei Liang, Zixuan Li, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al.
Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation.
*arXiv preprint arXiv:2506.18088*, 2025b.
- de Bakker et al. (2025)
Vincent de Bakker, Joey Hejna, Tyler Ga Wei Lum, Onur Celik, Aleksandar Taranovic, Denis Blessing, Gerhard Neumann, Jeannette Bohg, and Dorsa Sadigh.
Scaffolding dexterous manipulation with vision-language models.
*arXiv preprint arXiv:2506.19212*, 2025.
- de Boor (2001)
Carl de Boor.
*A Practical Guide to Splines*, volume 27 of *Applied Mathematical Sciences*.
Springer, New York, NY, USA, revised edition, 2001.
ISBN 978-0-387-95366-3.
doi: 10.1007/978-1-4612-6333-3.
- Driess et al. (2023)
D. Driess, F. Xia, et al.
Palm-e: An embodied multimodal language model.
*arXiv preprint arXiv:2303.03378*, 2023.
- Fan et al. (2025)
Y. Fan, P. Ding, S. Bai, X. Tong, Y. Zhu, H. Lu, F. Dai, W. Zhao, Y. Liu, et al.
Long-vla: Unleashing long-horizon capability of vision language action model for robot manipulation.
In *Conference on Robot Learning*, 2025.
- Farin (2002)
Gerald Farin.
*Curves and Surfaces for CAGD: A Practical Guide*.
Morgan Kaufmann, San Francisco, CA, USA, 5 edition, 2002.
ISBN 1-55860-737-4.
- French (1999)
R. M. French.
Catastrophic forgetting in connectionist networks.
*Trends in Cognitive Sciences*, 3(4):128–135, 1999.
- Hu et al. (2022)
E. J. Hu, Y. Shen, et al.
Lora: Low-rank adaptation of large language models.
*arXiv preprint arXiv:2106.09685*, 2022.
- Intelligence (2025)
A. Intelligence.
A dual-system framework for general-purpose agents, 2025.
- James et al. (2019)
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison.
Rlbench: The robot learning benchmark & learning environment.
*arXiv preprint arXiv: 1909.12271*, 2019.
- Khazatsky et al. (2025)
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir
Mirchandani, Daniel Morton, Tony Nguyen, Abigail O’Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn.
Droid: A large-scale in-the-wild robot manipulation dataset, 2025.
URL [https://arxiv.org/abs/2403.12945](https://arxiv.org/abs/2403.12945).
- Kim et al. (2024)
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al.
Openvla: An open-source vision-language-action model.
*arXiv preprint arXiv:2406.09246*, 2024.
- Kirkpatrick et al. (2017)
J. Kirkpatrick, R. Pascanu, et al.
Overcoming catastrophic forgetting in neural networks.
*Proceedings of the National Academy of Sciences*, 114(13):3521–3526, 2017.
- Li et al. (2023)
S. Li, D. Shah, et al.
Llm-planner: Few-shot grounded planning for embodied agents with large language models.
*arXiv preprint arXiv:2212.04088*, 2023.
- Li et al. (2024)
Y. Li et al.
Hamster: A hierarchical agent for multi-step task execution in robotics, 2024.
- Li et al. (2025)
Z. Li et al.
Bridgevla: A vla with a bridge between high and low level, 2025.
- Liu et al. (2023a)
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone.
Libero: Benchmarking knowledge transfer for lifelong robot learning.
*Advances in Neural Information Processing Systems*, 36:44776–44791, 2023a.
- Liu et al. (2023b)
F. Liu, Y. Zhang, et al.
Llm+robot: Large language models for robotics planning and skill learning.
*arXiv preprint arXiv:2304.12244*, 2023b.
- Liu et al. (2025)
Y. Liu, M. Ma, X. Yu, P. Ding, H. Zhao, M. Sun, S. Huang, and D. Wang.
Ssr: Enhancing depth perception in vision-language models via rationale-guided spatial reasoning.
In *Advances in Neural Information Processing Systems*, 2025.
- Manuelli et al. (2019)
Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake.
kpam: Keypoint affordances for category-level robotic manipulation.
In *The International Symposium of Robotics Research*, pp. 132–157. Springer, 2019.
- Miura et al. (2024)
Takato Miura, Naoki Akai, Kohei Honda, and Susumu Hara.
Spline-interpolated model predictive path integral control with stein variational inference for reactive navigation.
In *2024 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 13171–13177. IEEE, 2024.
- Mu et al. (2021)
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su.
Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.
*arXiv preprint arXiv:2107.14483*, 2021.
- Mu et al. (2024)
Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo.
Robotwin: Dual-arm robot benchmark with generative digital twins (early version).
*arXiv preprint arXiv:2409.02920*, 2024.
- Mu et al. (2025)
Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, et al.
Robotwin: Dual-arm robot benchmark with generative digital twins.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 27649–27660, 2025.
- Pertsch et al. (2020)
Karl Pertsch, Oleh Rybkin, Jingyun Yang, Shenghao Zhou, Konstantinos Derpanis, Kostas Daniilidis, Joseph Lim, and Andrew Jaegle.
Keyframing the future: Keyframe discovery for visual prediction and planning.
In *Learning for Dynamics and Control*, pp. 969–979. PMLR, 2020.
- Piegl & Tiller (1997)
Les Piegl and Wayne Tiller.
*The NURBS Book*.
Springer, Berlin, Heidelberg, 2 edition, 1997.
ISBN 978-3-540-61545-3.
doi: 10.1007/978-3-642-59223-2.
- Reed et al. (2022)
S. Reed, K. Zolna, et al.
A generalist agent.
*arXiv preprint arXiv:2205.06175*, 2022.
- Shoemake (1985)
Ken Shoemake.
Animating rotation with quaternion curves.
In *Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques*, SIGGRAPH ’85, pp. 245–254, New York, NY, USA, 1985. Association for Computing Machinery.
doi: 10.1145/325165.325242.
URL [https://doi.org/10.1145/325165.325242](https://doi.org/10.1145/325165.325242).
- Song et al. (2025a)
W. Song, J. Chen, P. Ding, Y. Huang, H. Zhao, D. Wang, and H. Li.
Ceed-vla: Consistency vision-language-action model with early-exit decoding.
*arXiv preprint arXiv:2506.13725*, 2025a.
- Song et al. (2025b)
W. Song, J. Chen, W. Li, X. He, H. Zhao, C. Cui, P. Su, F. Tang, X. Cheng, et al.
Rationalvla: A rational vision-language-action model with dual system.
*arXiv preprint arXiv:2506.10826*, 2025b.
- Song et al. (2025c)
W. Song, Z. Zhou, H. Zhao, J. Chen, P. Ding, H. Yan, Y. Huang, F. Tang, et al.
Reconvla: Reconstructive vision-language-action model as effective robot perceiver.
*arXiv preprint arXiv:2508.10333*, 2025c.
- Team et al. (2024)
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al.
Octo: An open-source generalist robot policy.
*arXiv preprint arXiv:2405.12213*, 2024.
- Urain et al. (2022)
Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki.
Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion.
*arXiv preprint arXiv:2209.03855*, 2022.
- Walke et al. (2023)
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Maximilian Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Q. Vuong, Andre Wang He, Vivek Myers, Kuan Fang, Chelsea Finn, and S. Levine.
Bridgedata v2: A dataset for robot learning at scale.
*Conference on Robot Learning*, 2023.
doi: 10.48550/arXiv.2308.12952.
- Wang et al. (2025)
Y. Wang, P. Ding, L. Li, C. Cui, Z. Ge, X. Tong, W. Song, H. Zhao, W. Zhao, et al.
Vla-adapter: An effective paradigm for tiny-scale vision-language-action model.
*arXiv preprint arXiv:2509.09372*, 2025.
- Wen et al. (2025)
Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al.
Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation.
*IEEE Robotics and Automation Letters*, 2025.
- Wu et al. (2025a)
Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan.
Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence.
*arXiv preprint arXiv:2505.23747*, 2025a.
- Wu et al. (2025b)
Shijie Wu, Yihang Zhu, Yunao Huang, Kaizhen Zhu, Jiayuan Gu, Jingyi Yu, Ye Shi, and Jingya Wang.
Afforddp: Generalizable diffusion policy with transferable affordance.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 6971–6980, 2025b.
- Yang et al. (2025a)
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, and Jianfeng Gao.
Magma: A foundation model for multimodal ai agents, 2025a.
URL [https://arxiv.org/abs/2502.13130](https://arxiv.org/abs/2502.13130).
- Yang et al. (2025b)
Rujia Yang, Geng Chen, Chuan Wen, and Yang Gao.
Fp3: A 3d foundation policy for robotic manipulation.
*arXiv preprint arXiv:2503.08950*, 2025b.
- Yuan et al. (2024)
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox.
Robopoint: A vision-language model for spatial affordance prediction for robotics.
*arXiv preprint arXiv:2406.10721*, 2024.
- Zenke et al. (2017)
F. Zenke, B. Poole, and S. Ganguli.
Continual learning through synaptic intelligence.
In *Proceedings of ICML*, pp. 3987–3995, 2017.
- Zhao et al. (2025)
W. Zhao, G. Li, Z. Gong, P. Ding, H. Zhao, and D. Wang.
Unveiling the potential of vision-language-action models with open-ended multimodal instructions.
*arXiv preprint arXiv:2505.11214*, 2025.
- Zhen et al. (2024)
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan.
3d-vla: A 3d vision-language-action generative world model.
*arXiv preprint arXiv:2403.09631*, 2024.

### USE OF LLMS

We used large language models (LLMs) only for minor assistance in polishing the language and
adjusting the presentation of tables. No LLMs were involved in designing the methodology, conducting experiments, or analyzing results.

## Appendix A Data and Implementation Details

### A.1 Data Sources and Coverage

We integrate three data sources—ManiSkill, RoboTwin 2.0, and AgiBot World—within a unified framework to create a cross-ontology, cross-view, and cross-scenario manipulation corpus.

ManiSkill: Provides high-quality, large-scale single-task trajectory data, including stack cube, push cube, and pull cube tool tasks, with a total of 3000 trajectories.

RoboTwin 2.0: Focuses on dual-arm manipulation and strong domain randomization (texture/lighting/table height/clutter/linguistic perturbation), including multiple robots such as Aloha-AgileX, ARX-X5, Franka, Piper, UR5, and others. We use a high-quality subset of 40 tasks, totaling around 2000 trajectories, divided according to the official evaluation. Extensive data from various perspectives were collected, with precise camera intrinsic and extrinsic parameters. We also created 30 custom tasks based on the official scenarios, collecting a total of 200 trajectories for 10 tasks.

AgiBot World: Contains multi-modal sequences from near-realistic collection processes (AgiBot G1), including RGB-D, fisheye, multi-view calibration, and step-level language. To ensure stable simulation-to-real transitions, we use sub-scenes resembling home/retail/office environments, totaling 10 tasks and 500 trajectories. The evaluation uses official training data from the AgiBot challenge, and the evaluation is performed on their official servers, with code reviewed by the AGIBOT team.

Private Robot Dataset and Open Source Robot Dataset: We collected 1k+ trajectories on Franka and PiPER robot in custom robot setting and followed task configuration of RoboTwin dataset. AGIBOT G1 dataset was from AGIBOT official.

### A.2 Experimental Setup

Usage. We train NoTVLA on a unified simulation mix: 2000 trajectories from RoboTwin (40 tasks), 3000 from ManiSkill (3 tasks), and 1000 trajectories from AGIBOT (10 tasks). Each trajectory is normalized to the camera frame and discretized at gripper state changes; at each keyframe we store image-plane UVs, depth tokens, gripper pose, and a binary gripper flag. Logs are serialized as JSON+NPZ with timestamps and camera intrinsics/extrinsics. The control target is the end effector across embodiments, aligning gripper coordinates to the fingertip midpoint; joint angles are retained for diagnostics. Embodiments include Aloha-AgileX, ARX-X5, Franka Panda, UR5, Piper, and AgiBot-G1; inference uses head cameras, with wrist views used only for augmentation. We sample 10 frames per segment (upsampling to 10 via splines) and allocate 50/0/100 for train/val/test with instruction de-duplication. Language consists of 100 native prompts plus 20 crowd-sourced rewrites per task. All data are versioned with checksums and visually inspected to filter low-quality samples.

Datasets. We use three complementary sources: RoboTwin 2.0, AgiBot World and self collect dataset. RoboTwin2.0 contributes large-scale bimanual demonstrations with domain randomization over textures, lighting, table height, clutter, and language prompts; we adopt its evaluation split and dual-arm embodiments (Aloha-AgileX, ARX-X5, Franka, Piper, UR5). AgiBot World offers real-to-sim–friendly sequences from dual-arm humanoids (AgiBot G1), with RGB-D and fisheye streams, calibration, and step-level language. Across sources, trajectories bundle synchronized visual frames, depth, proprioception, end-effector pose, and gripper state, together with the instruction text. This mixture covers short- and long-horizon skills (e.g., opening/closing, stacking, placement, tool use) in household, retail, and office scenes. In combination, the corpora provide complementary coverage: ManiSkill contributes controllable 3D variety, RoboTwin supplies structured bimanual tasks with broad randomization, and AgiBot contributes high-fidelity, real-world diversity.-

### A.3 Training Weight of Tokens

During inference, token types contribute differently to model accuracy, which is shown in Table [6](https://arxiv.org/html/2510.03895v1#A1.T6). One important consideration is whether increasing the loss weight of trajectory tokens improves execution. Under NoTVLA training, scaling the trajectory-token weight by an order of magnitude does *not* improve manipulation. In fact, it substantially degrades language proficiency and reasoning. After reweighting, we observe declines on standard language benchmarks and lower task accuracy in simulation. A likely explanation is overfitting: the model over-emphasizes numeric trajectory reasoning at the expense of logical interpretation and analysis of instructions.

**Table 6: Performance on MMLU, CMMLU, and CEVAL (STEM, Social Sciences, Humanities, Other, Avg.).**
| Model | STEM | Social | Hum. | Other | Avg. |
| --- | --- | --- | --- | --- | --- |
| MMLU |  |  |  |  |  |
| Qwen7B | 65.64 | 79.79 | 63.27 | 74.89 | 70.08 |
| Ours w/ 10$\times$ | 36.48 | 51.48 | 37.79 | 43.34 | 41.79 |
| Ours | 55.30 | 74.52 | 53.75 | 67.00 | 61.69 |
| CMMLU |  |  |  |  |  |
| Qwen7B | 71.36 | 75.63 | 77.30 | 75.84 | 75.11 |
| Ours w/ 10$\times$ | 35.12 | 44.33 | 43.11 | 42.03 | 41.48 |
| Ours | 57.17 | 66.70 | 67.10 | 63.44 | 63.88 |
| CEVAL |  |  |  |  |  |
| Qwen7B | 73.02 | 84.00 | 76.26 | 73.96 | 76.15 |
| Ours w/ 10$\times$ | 38.14 | 47.27 | 48.64 | 44.01 | 43.68 |
| Ours | 58.14 | 75.27 | 64.59 | 60.16 | 63.45 |

### A.4 Tasks Details Description

#### A.4.1 AGIBOT Challenge

Remove loose trash and food scraps from the countertop, place them into the appropriate waste or recycling container, and wipe the surface clean.

Open the specified drawer, place the target items inside following organizational guidelines, arrange them neatly, and close the drawer securely.

Place the food in a microwave-safe container, set the appropriate power and time, start the microwave, and carefully remove and check the food temperature when finished.

Identify and pick moving objects from the conveyor belt, place them into designated packing boxes or trays while maintaining speed and stability.

Open the freezer, locate and retrieve specified frozen items using proper protection (e.g., gloves), move them to the target location, and close the freezer.

Retrieve products from stock, refill shelves according to merchandising rules (facing front, orderly arrangement), replace expired or mislabeled items as needed.

Pack purchased goods for a customer by grouping items sensibly, protecting fragile goods, and arranging bags or boxes for safe and easy carrying.

Prepare bread and fillings according to the recipe, assemble ingredients in order, cut if required, and present a clean, ready-to-eat sandwich.

Remove dishes, utensils, and leftovers from the table, clear and sanitize the surface, and reset place settings if necessary for the next guest.

Retrieve the stamp, align it with the required location on the document, press firmly to produce a clear impression, and verify alignment and clarity.

#### A.4.2 RoboTwin Official Tasks

Use a hammer to strike the block target repeatedly until the action is completed or the block is secured.

Rank blocks by visual RGB features and sort them according to the specified criterion.

Rank blocks by size and arrange them in the required order.

Locate the alarm clock and press or click the designated button to silence or activate the alarm.

Press the bell actuator at the indicated position to produce a ringing signal.

Open the large bin and empty its contents into the designated disposal area.

Reach for and grasp the roller object securely, then lift or move it to the target location.

Grasp the microphone and transfer it to another agent or specified handover position.

Pick up a mug from a hanging position and place it on the table or target surface.

Grasp the pot handle(s) and lift it carefully to the required height or location.

Move the can and the pot to their respective target positions as specified.

Pick up the pill bottle and place it on the target pad or designated area.

Remove the playing card from its current location and place it at the specified away location.

Pick up the stapler and position it on the given pad or surface.

Open the laptop lid to a functional angle and ensure it is ready for use.

Open the microwave door and prepare the interior for loading or unloading items.

Pick up object A and place it at location B on the left-side target as specified.

Pick up object A and place it at location B on the right-side target as specified.

Place the bread item into the basket, arranging it neatly.

Place the bread onto the skillet or pan and position it correctly for cooking or serving.

Pack or arrange the burger and fries together in the specified container.

Place the can into the basket at the designated spot.

Place the can into the plastic box following placement constraints.

Put the container onto the plate or place the plate into the container as required.

Pick up the pair of shoes and place them together at the target location.

Place an empty cup at the specified position without spilling or tilting.

Position the fan at the target location and orientation.

Place the mouse pad flat at the designated workspace area.

Place the specified object into the basket, ensuring stable placement.

Place the object on the scale platform for weighing.

Position the object on the stand securely and centered.

Place the phone onto the stand in the correct orientation.

Pick up the shoe and place it at the designated spot.

Press down the stapler mechanism to staple the target documents or materials.

Place the object into the cabinet compartment and close the door if required.

Rotate the QR code marker to the specified orientation for scanning.

Position the scanner or object and perform a scan to capture required data.

Grasp the bottle and shake it vertically or as specified to mix contents.

Grasp the bottle and shake it horizontally to achieve the required motion.

Stack three blocks vertically in the specified order and ensure stability.

Stack two blocks as required and confirm alignment.

Stack three bowls together with stable nesting.

Stack two bowls together with proper alignment.

Align the seal with the document and press to produce a clear stamp impression.

Rotate or flip the switch to the target position to change device state.

#### A.4.3 RoboTwin Custom Tasks

Using a previously seen instruction, stack two blocks in the inverse order. The order in instruction is inversed, also.

Following a unseen color-order instruction, stack two random color blocks in the required order.

Pick the colorful block and place it onto the correspondingly colored pad, matching color pairing and ensuring stable placement.

Pick the two specified number blocks and stack them in required numerical order, bottom to top.

Given three candidate blocks, randomly select any two and stack them in required order of color.

Pick the target block and place it onto the flag-marked pad, centered and fully supported.

Pick the colorful block and place it onto the pad with the matching color pattern, aligning edges for stable contact.

From multiple pads, choose a flag-labeled pad and place the block there.

Pick an object from region A and place it at a random valid pose within region B, staying within bounds and avoiding obstacles.

## Appendix B Trajectory Structure and Storage Format

Each trajectory includes the following components:

Visual Stream: RGB-D (has different resolution), optional fisheye/multi-view.

End-Effector Perception: End-effector pose, joint angles, gripper status.

Instruction Text: Original prompt + 20 crowdsourced/rewrite variants.

Calibration Information: Camera intrinsic and extrinsic parameters, timestamp synchronization.

Quality Labels: Success/failure, abnormal frame markings, occlusion/shift indicators.

To align with the NoT-VLA intermediate interface, the data is serialized in JSON + NPZ (or Parquet): meta.json stores instructions, camera parameters, and time references; traj.npz stores image indices, depth, UV keypoints, discretized end-effector pose, gripper flag, and auxiliary quantities for the detokenizer. All entries are hash-checked and versioned; raw video frames are optionally packaged in H.264 with keyframe indexing for fast replay.

## Appendix C Depth Sensing with Anchor

Acquiring accurate, reliable, and task-aligned depth data is crucial for manipulation tasks in 3D space. To obtain such depth data, the following approaches based on depth information can be adopted:

- 1.
Direct sensor depth (applicable to both simulated and real-world scenarios);
- 2.
Output from external monocular depth models;
- 3.
Depth anchor tokens predicted by NoTVLA itself on the image plane, followed by relative depth calculation using prior object dimensions.

This ”anchor-relative depth” paradigm significantly narrows the search space of the Vision-Language Model (VLM), enhancing generalization robustness across different views/scenes while preserving the plug-and-play capability of sensors and models.

### C.1 Training Steps

As shown in Table  [7](https://arxiv.org/html/2510.03895v1#A3.T7), the result presents the performance metrics (presumably success rates, given their range of 0 to 1) of 11 tasks categorized into Short Horizon, Medium Horizon, and Long Horizon groups, evaluated across 8 step horizons (500, 1000, 1500, 2000, 2500, 3000, 3500, and 4000 steps). The performance data across different step horizons reveals critical insights into the interplay between task complexity, horizon length, and system adaptability: First, task inherent difficulty overrides step horizon adjustments—Short Horizon Tasks, though theoretically less complex, show extreme disparities (e.g., ”grab roller” and ”place empty cup” consistently above 0.6, while ”place phone stand” never exceeds 0.25), indicating that task-specific challenges (e.g., object shape, precision requirements) have a more significant impact on success than step count. Second, system stability correlates with task type rather than horizon length: Medium Horizon Tasks like ”stack blocks two” maintain steady performance (0.50–0.75 across all steps) because stacking tasks likely have more predictable action sequences, whereas erratic results in ”place shoe” (Short Horizon) and ”place a2b random” (Medium Horizon) suggest the system struggles with tasks involving variable object positions or unstructured interactions. Third, long-horizon tasks benefit from moderate step increases but hit diminishing returns: ”stack blocks three” and ”blocks ranking rgb” improve from 500 to 3500 steps (peaking at 0.60 and 0.70 respectively) but do not continue to advance at 4000 steps, implying that beyond a certain threshold, additional steps do not compensate for the cumulative complexity of multi-step tasks (e.g., error propagation in block stacking).

**Table 7: Task performance comparison at various horizons (Short, Medium, Long).**
| Task Name | 500 | 1000 | 1500 | 2000 | 2500 | 3000 | 3500 | 4000 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Short Horizon Tasks |  |  |  |  |  |  |  |  |
| grab roller | 0.85 | 0.80 | 0.70 | 0.85 | 0.90 | 1.00 | 0.90 | 0.80 |
| place phone stand | 0.00 | 0.25 | 0.05 | 0.10 | 0.15 | 0.25 | 0.15 | 0.15 |
| place empty cup | 0.75 | 0.85 | 0.75 | 0.70 | 0.65 | 0.60 | 0.70 | 0.80 |
| place shoe | 0.10 | 0.25 | 0.30 | 0.55 | 0.30 | 0.15 | 0.50 | 0.25 |
| Medium Horizon Tasks |  |  |  |  |  |  |  |  |
| place a2b random | 0.05 | 0.20 | 0.40 | 0.25 | 0.35 | 0.50 | 0.25 | 0.20 |
| place bread basket | 0.25 | 0.50 | 0.60 | 0.45 | 0.40 | 0.40 | 0.55 | 0.40 |
| stack blocks two | 0.50 | 0.75 | 0.65 | 0.75 | 0.75 | 0.75 | 0.65 | 0.75 |
| stack colorfulblocks two | 0.55 | 0.65 | 0.50 | 0.65 | 0.55 | 0.65 | 0.40 | 0.70 |
| Long Horizon Tasks |  |  |  |  |  |  |  |  |
| stack blocks three | 0.05 | 0.30 | 0.25 | 0.45 | 0.50 | 0.30 | 0.60 | 0.50 |
| blocks ranking rgb | 0.25 | 0.45 | 0.60 | 0.55 | 0.50 | 0.45 | 0.70 | 0.50 |

### C.2 Trajectory Planning

Table LABEL:tab:planning_performance compares the performance of NoTVLA planning trajectory in 2D and Magma planning trajectory in 2D across 10 key metrics, which evaluates the accuracy, similarity, and spatial alignment of generated or processed shapes. A critical distinction in metric interpretation first emerges: while higher values indicate better performance for cover F1 and cover precision (standard classification/segmentation metrics where 1.0 represents perfect performance), lower values signal superior results for the remaining eight metrics (e.g., DTW, endpoint error, Frechet distance), as these quantify discrepancies or distances between target and predicted shapes—smaller values mean less deviation from the ground truth.
Against this framework, NoTVLA planning trajectory in 2D outperforms Magma planning trajectory in 2D across all metrics, demonstrating consistent superiority in both shape coverage and spatial accuracy. For the coverage-related metrics, NoTVLA planning trajectory in 2D achieves a cover F1 score of 0.9546, which is approximately 2.8% higher than Magma Shape’s 0.9282, and a cover precision of 0.9558 (nearly 0.8% higher than Magma Shape’s 0.9484); these gains indicate that NoTVLA planning trajectory in 2D not only captures more of the target shape (better coverage) but also does so with fewer false positives (higher precision). For DTW (Dynamic Time Warping, a measure of temporal or sequential similarity), NoTVLA planning trajectory in 2D scores 0.4565—less than half of Magma Shape’s 0.8353—suggesting that if the shapes are time-dependent (e.g., motion trajectories), ”Our Shape” aligns far more closely with the target sequence. Similarly, NoTVLA planning trajectory in 2D reduces endpoint error (a measure of how much the final point of the shape deviates from the target) by over 58% (0.0740 vs. 0.1785) and Frechet distance (a metric for comparing continuous shapes) by nearly 55% (0.1035 vs. 0.2297), both of which are critical for tasks where precise shape boundaries or endpoints matter (e.g., robotic path planning or medical image segmentation). Even in finer-grained spatial metrics—such as mean orthogonal distance (average deviation across all points) and median orthogonal distance (middle value of deviations, less sensitive to outliers)—NoTVLA planning trajectory in 2D maintains a clear edge: its mean orthogonal distance (0.0322) is roughly 32% lower than Magma Shape’s (0.0476), and its median (0.0286) is 35% lower, confirming that this superiority is consistent across most points of the shape.

### C.3 Zero Shot of Different Models

The Table LABEL:tab:different_models_performance evaluates the zero-shot performance of three NoTVLA models (differentiated by their parameter sizes: 7B, 3B, and 1B) across three distinct task categories—Color Recognition, Target Selection, and Spatial Semantics—offering key insights into how model scale influences generalization ability in zero-shot settings, where models are tested on tasks without explicit prior training. A foundational observation from the data is the clear correlation between model parameter size and overall zero-shot performance: larger models consistently outperform smaller ones across nearly all tasks, while the smallest 1B parameter model shows minimal to no success in most cases, highlighting a critical threshold in parameter scale for enabling effective zero-shot generalization.

In the Color Recognition category, which focuses on tasks requiring identification and manipulation of colored blocks, the performance gap between model sizes is particularly striking. The 7B model maintains strong effectiveness here, while the 3B model exhibits a modest drop in performance relative to the 7B variant—suggesting that color-related zero-shot tasks demand a certain level of model capacity to encode and apply visual-semantic associations, but that even the 3B model retains some ability to generalize. In contrast, the 1B model fails entirely at these tasks, indicating that its capacity is insufficient to capture the basic visual cues or task logic needed for recognizing and stacking colored blocks without prior training.
The Target Selection category, which involves tasks centered on manipulating specific objects (pillbottle pads) and includes both ”easy” and ”hard” difficulty levels, reinforces the size-performance trend while also revealing how task complexity interacts with model capacity. Across all Target Selection tasks, the 7B model achieves the highest success rates, with the 3B model showing noticeable underperformance—especially in the harder variants of the task. This suggests that as task difficulty increases (e.g., when choices or precision requirements are added), the gap between larger and smaller models widens, as larger models have more capacity to handle the additional complexity of decision-making or fine-grained object interaction in zero-shot scenarios. The 1B model’s complete failure here further underscores that even relatively straightforward target-focused tasks require more than 1B parameters to generalize to without training.

In the Spatial Semantics category, which tests the ability to understand and act on spatial relationships (e.g., placing objects to the left or right, or in random positions), the same pattern holds, but with a subtle distinction: the 1B model shows minimal success in a small subset of tasks (e.g., ”place a2b left (easy)”) rather than complete failure. This tiny improvement, while negligible in practical terms, hints that spatial semantics tasks may have some simpler sub-components that the 1B model can partially grasp—though its overall inability to perform consistently across even easy spatial tasks confirms its limited capacity. For the 7B and 3B models, spatial tasks follow the familiar trend: the 7B model outperforms the 3B model, with both showing better performance on easier variants of the tasks, emphasizing that spatial reasoning in zero-shot settings, like color recognition and target selection, relies heavily on model scale to encode and apply abstract spatial concepts.

Collectively, these results point to two core conclusions: first, model parameter size is a primary determinant of zero-shot performance for the NoTVLA series, with a sharp drop-off in effectiveness when scaling down from 3B to 1B parameters—indicating that 3B parameters may represent a minimum threshold for meaningful zero-shot generalization across the tested task types. Second, task complexity (as reflected in ”easy” vs. ”hard” labels) interacts with model size: harder tasks amplify the performance gap between larger and smaller models, as they require more capacity to handle uncertainty, decision-making, or fine-grained reasoning. This suggests that for zero-shot applications involving visual manipulation or spatial reasoning, investing in larger model sizes (e.g., 7B parameters) is critical for achieving reliable performance, while smaller models (especially 1B) are unlikely to be viable unless tasks are extremely simple or include explicit training data.

**Table 8: ZEROSHOT task performance comparison across different models.**
| Task Name | NoTVLA 7B | NoTVLA 3B | NoTVLA 1B |
| --- | --- | --- | --- |
| Color Recognition |  |  |  |
| stack block two(easy) | 0.86 | 0.84 | 0.00 |
| stack random color blocks two(easy) | 0.69 | 0.55 | 0.00 |
| Target Selection |  |  |  |
| move pillbottle pad(hard) | 0.38 | 0.21 | 0.00 |
| move pillbottle pad(easy) | 0.38 | 0.27 | 0.00 |
| move pillbottle pad choice(hard) | 0.26 | 0.14 | 0.00 |
| move pillbottle pad choice(easy) | 0.28 | 0.20 | 0.00 |
| Spatial Semantics |  |  |  |
| place a2b left(easy) | 0.39 | 0.21 | 0.04 |
| place a2b right(easy) | 0.28 | 0.17 | 0.00 |
| place a2b randomly(hard) | 0.36 | 0.19 | 0.02 |
| place a2b randomly(easy) | 0.34 | 0.24 | 0.02 |

## Appendix D Cross Embodiment

A RealSense D455 depth camera is used, and camera calibration is performed via automatic calibration supplemented with manual fine-tuning to ensure the highest possible accuracy of camera extrinsic parameters.

Considering the incomparability of joint spaces across different robots, normalized EE poses are adopted as the unified control interface; joint angles are only used for diagnosis and visualization. Unified gripper coordinate system mapping and workspace constraints are established for six robotic platforms: Aloha-AgileX, ARX-X5, Franka Panda, UR5, Piper, and AgiBot G1. For cross-platform deployment, only the Action Expert (e.g., Inverse Kinematics (IK), Diffusion Policy, ACT, etc.) needs to be replaced to generate low-level control signals from the unified EE trajectory.

### D.1 Close-loop

Achieving closed-loop control is a foundational goal for both Vision-Action (VA) and VLA models, as it ensures control stability in the presence of external disturbances (e.g., slight shifts in object position, minor robotic drift) or unmodeled dynamics (e.g., variations in friction between the end effector and a manipulated object). To realize this, our spline-based action detokenizer integrates a mechanism for real-time secondary inference, designed to enable trajectory correction without interrupting ongoing operation. The workflow operates as follows: upon initiating execution of a trajectory generated via the initial model inference, the VLA system simultaneously triggers a secondary inference process. This secondary inference leverages the latest visual feedback (e.g., a camera frame captured mid-execution) to re-estimate the target trajectory, accounting for any deviations from the original plan that may have occurred since execution began. During the interval when secondary inference is in progress, the robot continues to execute the original trajectory—this avoids motion halts and maintains task continuity, which is particularly critical for time-sensitive operations.

Once the secondary inference is completed, the system performs trajectory merging to integrate the updated plan into the ongoing execution. Rather than abruptly switching to the new trajectory (which would introduce discontinuities in velocity or acceleration), the detokenizer generates a short, spline-interpolated transition segment that connects the end effector’s current position and velocity to the starting point of the newly inferred trajectory. This transition segment ensures smooth motion throughout the handoff, while the remaining portion of the original trajectory is discarded to prevent further accumulation of errors. By iterating this process at fixed intervals (e.g., every 0.5 seconds), the system achieves continuous closed-loop control: each secondary inference cycle corrects for recent deviations, and the spline-based merging ensures that these corrections do not compromise trajectory smoothness. For example, if a target object shifts 2 cm horizontally during the “pre-grasp” phase of a task, the secondary inference will adjust the “post-grasp” trajectory to align with the object’s new position, and the transition segment will ensure the end effector adapts to this shift without jerk or delay.

Take the real situation as an example. At $t_{0}$, waypoints $\mathcal{P}_{0}=\{(u_{i},v_{i},d_{i})\}_{i=1}^{n}$ are projected to 3D using intrinsics $\bm{K}$ via $\bm{p}_{i}=d_{i}\,\bm{K}^{-1}[u_{i},\,v_{i},\,1]^{\top}$, then fit with a cubic B-spline (centripetal parameterization) to produce $\bm{T}_{0}(t)$ (C^2 continuous inside segments). At $t_{1}$, new waypoints $\mathcal{P}_{1}$ arrive. At $t_{2}{=}t_{1}{+}\Delta t$, we update from the current pose $\bm{T}_{0}(t_{2})$, the executed subset of $\mathcal{P}_{0}$, and $\mathcal{P}_{1}$. We re-align the remaining (unexecuted) waypoint list by locating the closest still-pending waypoint to the current pose instead of using a fixed distance culling radius. Let the ordered (future) waypoint set be $\mathcal{P}^{\prime}=\{(u_{i}^{\prime},v_{i}^{\prime},d_{i}^{\prime})\}_{i=1}^{n^{\prime}}$ with 3D back-projections $\bm{p}_{i}^{\prime}=d_{i}^{\prime}\,\bm{K}^{-1}[u_{i}^{\prime},v_{i}^{\prime},1]^{\top}$. Define

$$ $k^{*}=\arg\min_{1\leq i\leq n^{\prime}}\big\|\bm{p}_{i}^{\prime}-\bm{T}_{0}(t_{2})\big\|_{2}.$ (13) $$

We compute a local forward direction (preferred motion direction) as

$$ $\hat{\bm{v}}_{\text{path}}=\begin{cases}\dfrac{\bm{p}_{k^{*}+1}^{\prime}-\bm{p}_{k^{*}}^{\prime}}{\|\bm{p}_{k^{*}+1}^{\prime}-\bm{p}_{k^{*}}^{\prime}\|_{2}},&k^{*}<n^{\prime},\[6.0pt] \dfrac{\bm{p}_{k^{*}}^{\prime}-\bm{p}_{k^{*}-1}^{\prime}}{\|\bm{p}_{k^{*}}^{\prime}-\bm{p}_{k^{*}-1}^{\prime}\|_{2}},&k^{*}=n^{\prime}.\end{cases}$ (14) $$

We decide whether to keep the closest waypoint itself by a directional consistency test:

$$ $\gamma=(\bm{p}_{k^{*}}^{\prime}-\bm{T}_{0}(t_{2}))^{\top}\hat{\bm{v}}_{\text{path}},\qquad\text{keep }k^{*}\text{ iff }\gamma>0.$ (15) $$

All earlier indices are discarded. Let the drop indicator be $\delta_{k^{*}}=\mathbb{1}[\gamma\leq 0]$. The refreshed pending waypoint sequence is then

$$ $\mathcal{P}_{\text{pending}}=\{(u_{j}^{\prime},v_{j}^{\prime},d_{j}^{\prime})\mid j\geq k^{*}+\delta_{k^{*}}\}.$ (16) $$

## Appendix E More Results and Reality Performance

### E.1 RoboTwin 2.0 Full Results

The performance comparison of ACT, DP, DP3 $\pi_{0}$, RDT and NoTVLA is shown in Table [10](https://arxiv.org/html/2510.03895v1#A5.T10).

**Table 10: Performance comparison between expert models and generalist models on various robotic tasks with old loss data.**
| Task Name | Expert Models | Generalist Models |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ACT | DP | DP3 | Pi0 | RDT | NoTVLA |  |
| click alarmclock | 0.32 | 0.61 | 0.77 | 0.63 | 0.55 | 0.78 |
| click bell | 0.58 | 0.54 | 0.90 | 0.44 | 0.92 | 0.94 |
| grab roller | 0.94 | 0.98 | 0.98 | 0.96 | 0.94 | 0.97 |
| handover mic | 0.85 | 0.53 | 1.00 | 0.98 | 0.97 | 0.99 |
| move playingcard away | 0.36 | 0.47 | 0.68 | 0.53 | 0.60 | 0.77 |
| open laptop | 0.56 | 0.49 | 0.82 | 0.85 | 0.62 | 0.57 |
| press stapler | 0.31 | 0.06 | 0.69 | 0.62 | 0.88 | 0.94 |
| shake bottle | 0.74 | 0.65 | 0.98 | 0.97 | 0.87 | 0.94 |
| Beat Block Hammer | 0.56 | 0.42 | 0.72 | 0.43 | 0.77 | 0.98 |
| Blocks Ranking RGB | 0.01 | 0.00 | 0.03 | 0.19 | 0.03 | 0.76 |
| Blocks Ranking Size | 0.00 | 0.01 | 0.02 | 0.07 | 0.00 | 0.53 |
| Click Alarmclock | 0.32 | 0.61 | 0.77 | 0.63 | 0.61 | 0.78 |
| Click Bell | 0.58 | 0.54 | 0.90 | 0.44 | 0.80 | 0.94 |
| Grab Roller | 0.94 | 0.98 | 0.98 | 0.96 | 0.74 | 0.97 |
| Handover Mic | 0.85 | 0.53 | 1.00 | 0.98 | 0.90 | 0.99 |
| Lift Pot | 0.88 | 0.39 | 0.97 | 0.84 | 0.72 | 0.90 |
| Move Can Pot | 0.22 | 0.39 | 0.70 | 0.58 | 0.25 | 0.76 |
| Move Pillbottle Pad | 0.00 | 0.01 | 0.41 | 0.21 | 0.08 | 0.12 |
| Move Playingcard Away | 0.36 | 0.47 | 0.68 | 0.53 | 0.43 | 0.77 |
| Move Stapler Pad | 0.00 | 0.01 | 0.12 | 0.00 | 0.02 | 0.28 |
| Open Laptop | 0.56 | 0.49 | 0.82 | 0.85 | 0.59 | 0.57 |
| Place A2B Left | 0.01 | 0.02 | 0.46 | 0.31 | 0.03 | 0.38 |
| Place A2B Right | 0.00 | 0.13 | 0.49 | 0.27 | 0.01 | 0.34 |
| Place Bread Basket | 0.06 | 0.14 | 0.26 | 0.17 | 0.10 | 0.63 |
| Place Burger Fries | 0.49 | 0.72 | 0.72 | 0.80 | 0.50 | 0.69 |
| Place Can Basket | 0.01 | 0.18 | 0.67 | 0.41 | 0.19 | 0.77 |
| Place Cans Plasticbox | 0.16 | 0.40 | 0.48 | 0.34 | 0.06 | 0.51 |
| Place Container Plate | 0.72 | 0.41 | 0.86 | 0.88 | 0.78 | 0.92 |
| Place Dual Shoes | 0.09 | 0.08 | 0.13 | 0.15 | 0.04 | 0.21 |
| Place Empty Cup | 0.61 | 0.37 | 0.65 | 0.37 | 0.56 | 0.90 |
| Place Fan | 0.01 | 0.03 | 0.36 | 0.20 | 0.12 | 0.30 |
| Place Mouse Pad | 0.00 | 0.00 | 0.04 | 0.07 | 0.01 | 0.23 |
| Place Object Scale | 0.00 | 0.01 | 0.15 | 0.10 | 0.01 | 0.43 |
| Place Object Stand | 0.01 | 0.22 | 0.60 | 0.36 | 0.15 | 0.75 |
| Place Phone Stand | 0.02 | 0.13 | 0.44 | 0.35 | 0.15 | 0.25 |
| Place Shoe | 0.05 | 0.23 | 0.58 | 0.28 | 0.35 | 0.59 |
| Press Stapler | 0.31 | 0.06 | 0.69 | 0.62 | 0.41 | 0.94 |
| Put Object Cabinet | 0.15 | 0.42 | 0.72 | 0.68 | 0.33 | 0.81 |
| Rotate QRcode | 0.01 | 0.13 | 0.74 | 0.68 | 0.50 | 0.73 |
| Scan Object | 0.02 | 0.09 | 0.31 | 0.18 | 0.04 | 0.24 |
| Shake Bottle | 0.74 | 0.65 | 0.98 | 0.97 | 0.74 | 0.94 |
| Shake Bottle Horizontally | 0.63 | 0.59 | 1.00 | 0.99 | 0.84 | 0.96 |
| Stack Blocks Three | 0.00 | 0.00 | 0.01 | 0.17 | 0.02 | 0.49 |
| Stack Blocks Two | 0.25 | 0.07 | 0.24 | 0.42 | 0.21 | 0.88 |
| Stack Bowls Three | 0.48 | 0.63 | 0.57 | 0.66 | 0.51 | 0.83 |
| Stack Bowls Two | 0.82 | 0.61 | 0.83 | 0.91 | 0.76 | 0.96 |
| Stamp Seal | 0.02 | 0.02 | 0.18 | 0.03 | 0.01 | 0.36 |

Figure: Figure 5: Franka Operation in the RoboTwin Simulation
Refer to caption: notvla/Figures/finalimage/Multiview_small.png

Figure: Figure 6: Zero Shot Franka Operation in the RoboTwin Simulation
Refer to caption: notvla/Figures/finalimage/command_generalization_small.png

Figure: Figure 7: AGIBOT G1 Operation in the Simulation
Refer to caption: notvla/Figures/finalimage/Agibot.png

Figure: Figure 8: PiPER Operation in the Reality
Refer to caption: notvla/Figures/finalimage/Realworld.png

Figure: Figure 9: Franka Operation in the Reality
Refer to caption: notvla/Figures/finalimage/Realword1.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The language capabilities degradation, while remedies, still occurs as suggests by Tab. 6. Do you have any insight into which specific language reasoning skills (e.g., spatial, logical, mathematical) are most affected by this sparse fine-tuning?
soundness: 4
strengths: 1. The paper is generally well-written and clear. 
2. The paper provides very extensive experiments in both simulation and reality, that show promising results on NoTVLA’s manipulation capabilities, generalizability, and the preserved language reasoning capabilities. The system is also training-efficient.3
3. The analysis provides rich insights on the language understanding degradation, zero-shot generalization, training efficiency, *e.t.c*.
summary: This paper proposed Narrowing of Trajectory VLA (NoTVLA), which tries to solve the catastrophic forgetting issues of fine-tuning VLA models. The paper attributes this issue to the dense trajectory representation, and addresses the issue by applying Keyframe Selection on training data, and performing Anchor Point prediction (APP) with Anchor-conditioned token generation (ACTG) during inference. The actions are decoded with spline-based detokenizer for smooth and high-frequency trajectories. Experiments demonstrate its effectiveness.
weaknesses: ## 

1. The Kinematics-Based Keyframe Selection relies on an acceleration threshold $\alpha$ that is "hand-designed per task." This introduces a manual, expert-in-the-loop step that could hinder scalability.
2. The APP and ACTG modules requires the depth information of the scene. While the VLA system has a good performance, a sensitivity analysis on the depth noise or bias is necessary to show how the system would work with different depth sensors and environments.
3. The 2D anchor representation could limit some tasks. Additional 2D or 3D representation (*e.g.*, via visual visual prompting, robot-object constraints, *e.t.c.*) could be discussed for comprehensiveness.
4. Some editorial and writing issues
    1. While the experiments validate the hypothesis of “dense fine-tuning is the cause of catastrophic forgetting”, the reason or the mechanism is not intuitively explained, especially at the beginning of the paper. This causes confusion and doubts.
    2. Figure and table captions are not self-contained.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: (1) In Figure 4 (left), is the x-axis scale correct? The number of training steps for pi0 seems an order of magnitude too large.

(2) There are also a number of questions in the weaknesses section that I would appreciate be addressed.
soundness: 2
strengths: (1) While the use of intermediary action spaces such as waypoints has been studied extensively and is thus not new, I like the authors’ motivation of this design decision in the context of VLAs (see summary for the motivation), though admittedly said motivations were a bit difficult to parse from the writing. In particular, the fact that VLA training tends to require a significant departure from the base VLM is not a widely discussed phenomenon (as far as I’m aware), so it is a strength that this paper has identified this problem and proposed a solution for it.

(2) The technical approach is sound and is justified by ablations (section 4.2.1), and experiments are reasonably thorough (though there seems to be no real-world experiments, which is a weakness).

(3) The empirical results agree with the motivations. Their VLA needs much less compute to train because it is less of a departure from the base VLM, and it also generalizes better than baselines like pi0 to slightly OOD task variations (table 3).
summary: This paper presents an approach (NoTVLA) that trains a vision-language-action (VLA) model on an intermediary waypoint-based action space (as opposed to the raw EEF position action space). The full process of predicting actions from the VLA looks like the following: (1) given the image and language instruction, the VLA predicts the pixel XY coordinates of the next waypoint (e.g., the pixel coordinates of the object to pick up). (2) A depth sensor (e.g., an RGBD camera) is queried to obtain depth at this pixel coordinate. (3) The VLA is queried again with the image, language instruction, and now pixel XY location and depth to produce N action tokens. These tokens describe the path to take to reach from the current robot position to the waypoint predicted in the first step, with each token encoding an intermediary XY pixel location, depth prediction, gripper state, and EEF orientation. (4) Finally a motion planner is used to generate a smooth trajectory that connects these waypoints and the full trajectory segment is played open-loop. The authors argue that the reason to choose this intermediary action space instead of the original action space as most VLAs do is that (1) action prediction across robot embodiments or very diverse robot states looks more similar, which encourages feature sharing, and (2) this waypoint prediction problem is less of a departure from the original base VLM pre-training than traditional VLA training, allowing for less VLA training compute to be spent and better generalization since the model retains more of its pre-training. On a few simulated benchmarks the authors compare their approach against imitation learning (e.g., diffusion policy) and VLA (e.g., pi0) baselines and find that their approach generalizes better and gets better success rates.
weaknesses: (1) Seemingly no real-robot experiments to validate their approach

(2) The authors claim that a strength of their approach is that their policies do not require a wrist mounted camera. However this statement can only be made if the authors have trained policies both with and without wrist mounted cameras, and empirically found that they are not needed. Also I don’t buy that wrist cameras can truly be avoided — there are many tasks for which the arms will occlude the shoulder camera, making wrist cameras necessary. 

(3) In the abstract and introduction the authors frequently use the term "catastrophic forgetting”, but do not sufficiently explain in what context catastrophic forgetting is happening. Is the “forgetting” happening when the base VLM is fine-tuned into a VLA? Or is it happening when the VLA is then further fine-tuned on specific downstream tasks? My assessment after reading the paper is that the authors are referring to the former, but this could be made more clear. 

(4) Several statements (concentrated especially in the introduction) are not clear, and it’s very difficult to parse what they mean. Further, many claims are simply unsubstantiated. Listing a few:

- Lines 41 to 45: “catastrophic forgetting […] is aggravated […] by an over-reliance on dense […] action trajectories […] when fine-tuned on a new task the model may overwrite previously consolidated competencies, degrading prior task performance” —> it is unclear to me what this statement means. My approximate takeaway is that the authors are trying to say that action prediction is a significant departure from the base VLM capabilities.

- Similarly lines 46-50: “The root cause of this fragility often lies in the prevailing training paradigm […]”. The authors should avoid making statements like “the root cause of X is Y” without some sort of evidence. This evidence can take the form of prior work citations, or of an easy to follow step-by-step logical argument.

- Line 53: “mitigating catastrophic forgetting” —-> this statement doesn’t explain *how* the approach mitigates catastrophic forgetting. My interpretation is that it makes the VLA action prediction task look a lot more like the base VLM pre-training tasks, but this needs to be explicitly mentioned.

- Figure 1 caption and rest of the introduction: same thing. In what context is catastrophic forgetting happening? Why do the authors think it is happening? Why does their approach solve this?

(5) The use of waypoints diminishes the applicability of the method to only pick-and-place tasks. I.e., can non-pick-and-place tasks, like T-shirt folding, stirring, or tool use be solved by the same method? How will waypoints be selected in such cases?

(6) It seems that with the proposed approach, entire trajectory segments, i.e., everything connecting two key-frames (like a gripper opening and the subsequent gripper closing), are executed completely open-loop. By open-loop I mean the VLA is not inferenced again until the motion planner executes the full trajectory segment. This is both a strength and a weakness; it is a strength in that the motion segment will likely be smooth, but it is a weakness if something unexpected happens while the motion segment is being played, like the object falling from the robot’s grasp, that would require reactivity and more closed-loop control.

(7) The acceleration threshold used to pick keyframes needs to be selected by the data-collector manually per task (line 240), which increases the burden on the data collector.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * Is the Pi-0 generalist policy fine-tuned on your training dataset from the RoboTwin tasks?
* What were the failure modes of your method? Does, for example, the target object being occluded cause your method to fail?
soundness: 3
strengths: This paper's method is novel and offers a clear interface between high-level keypoint reasoning and low-level control. The empirical results are strong and comprehensive: the method outperforms both per-task expert policies (e.g., ACT, DP, DP3) and generalist models (π0, RDT) across the RoboTwin2 and AGIBot benchmarks. Moreover, the paper demonstrates gains in data and computational efficiency (Figure 4).
summary: This paper presents a framework (NoTVLA)  for a multi-step VLA system that first predicts two image-based keypoints, and then, conditioning on those two keypoints with included depth information, predicts a trajectory key points on the image plane. These key points are projected to the world frame using depth and fit with a spline curve to generate high-frequency controls. They evaluate on Robotwin2 and on AGIBot environments, outperforming baselines across tasks.
weaknesses: The main reason for my score is the missing details for their baselines. More clearly:

* The paper does not clearly describe the training setups for its baselines. In particular, it is unclear what data the single-task experts are trained on, and whether pi0 was fine-tuned on the same trajectories used to train NoTVLA. Without these details, it is difficult to assess these comparisons. 
* Including a comparison to works such as Hamster [1] and LLARVA [2] would help show the benefits of the two-stage key point and then spline fitting method over other similar VLA approaches. 
* Some abbreviations (e.g., DP) are introduced without being defined
* I found the main framework figure distracting (Fig. 1), and it did not seem to have a natural direction for me to parse. Some of the terms in it, such as (u, v) and Bn, were not defined until later in the methods section. 


[1] Yi Li, Yuquan Deng, Jesse Zhang, et al. HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation.

[2] Dantong Niu, Yuvan Sharma, Giscard Biamby, et al.  LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1、How does NoTVLA’s performance on tasks requiring complex 3D spatial reasoning compare against VLA models that were pre-trained or fine-tuned on 3D-centric datasets? This would help quantify the trade-offs of your decoupled approach.
2、Could you provide a comparative analysis between your kinematics-based keyframe selection and other sparsification strategies to demonstrate its specific advantages in preserving task-critical information?
3、Could you discuss potential failure modes or task types where the focused perception of the anchor-based approach might be limiting due to a lack of broader scene context?
soundness: 2
strengths: 1、The paper clearly identifies catastrophic forgetting in VLAs as a key problem and proposes a structured, well-motivated solution.
2、Provides thorough multi-task evaluation on several robotic platforms and includes informative ablation studies validating the core design choices.
3、 Demonstrates significant practical advantages, including reduced computational cost and the ability to operate effectively without a wrist-mounted camera.
4、The anchor-based depth inference module presents a clear and effective decoupling of semantic perception from geometric reasoning enhancing system modularity.
summary: This paper introduces the Narrowing of Trajectory VLA (NoTVLA) framework, aiming to address catastrophic forgetting in Vision-Language-Action models for robotic manipulation. The core contribution is a shift from dense action trajectory supervision to sparse, semantically salient trajectory supervision. The method features: (1) an anchor-based depth inference module that decouples 2D semantic anchoring from 3D geometric reasoning by leveraging an external depth source, and (2) a kinematics-based keyframe selection strategy to identify critical moments in demonstrations. Evaluations on multi-task benchmarks demonstrate that NoTVLA achieves competitive performance compared to several VLA baselines, with reduced computational cost and improved zero-shot generalization in certain scenarios, while operating under a constrained sensing setup.
weaknesses: 1、As noted in Section 2.1, a major direction for spatial reasoning involves fine-tuning on 3D-centric datasets. The paper does not compare NoTVLA against such methods, leaving the claimed advantages of its decoupled, anchor-based approach unvalidated against models with more integrated 3D understanding.
2、The paper fails to demonstrate the advantage of its proposed kinematics-based keyframe selection over established action segmentation or keyframe extraction methods. Without this comparison, the claimed benefits regarding the preservation of task-critical information remain unsubstantiated.
3、The core hierarchical framework, which decouples a high-level planner (VLM) from a low-level controller, is a well-established paradigm in robot learning. While the specific implementation is competent, the paper does not establish a significant architectural innovation over prior hierarchical systems.
4、The anchor-based approach, while focused and efficient, inherently narrows the model’s perceptual field. The potential trade-off, specifically the risk of losing broader scene understanding compared to methods that process denser 3D representations, is not sufficiently discussed.