Trace Anything: Representing Any Video in 4D via Trajectory Fields
Authors: Xinhang Liu
Yuxi Xiao
Donny Y. Chen
Jiashi Feng
Yu-Wing Tai
Chi-Keung Tang
Bingyi Kang
Venue: ICLR 2026 Poster (2026)
Keywords: Video Representation
4D Scene Representation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Building 4D video representations to model underlying spacetime constitutes a crucial step toward understanding dynamic scenes, yet there is no consensus on the paradigm: current approaches resort to additional estimators such as depth, flow, or tracking, or to heavy per-scene optimization, making them brittle and hard to generalize. In a video, its atomic unit, the pixel, follows a continuous 3D trajectory that unfolds over time, acting as the atomic primitive of dynamics. Recognizing this, we propose to represent any video as a Trajectory Field: a dense mapping that assigns each pixel in each frame to a parametric 3D trajectory. To this end, we introduce Trace Anything, a neural network that predicts the trajectory field in a feed-forward manner. Specifically, for each video frame, the model outputs a series of control point maps, defining parametric trajectories for each pixel. Together, our representation and model directly construct a 4D video representation in a single forward pass, without additional estimators or global alignment. We develop a synthetic data platform to construct a training dataset and a benchmark for trajectory field estimation. Experiments show that Trace Anything surpasses existing methods or performs competitively on the new benchmark and established point tracking benchmarks, with significant efficiency gains. Moreover, it facilitates downstream applications such as goal-conditioned manipulation, simple motion extrapolation, and spatio-temporal fusion. We will release the code, the model weights, and the data platform.

Published as a conference paper at ICLR 2026
TRACE ANYTHING: REPRESENTING ANY VIDEO IN 4D VIA TRAJECTORY FIELDS
Xinhang Liu ByteDance Seed HKUST
Yuxi Xiao ByteDance Seed Zhejiang University
Donny Y. Chen
Jiashi Feng ByteDance Seed
ByteDance Seed
Yu- Yu-Wing Tai Dartmouth College
Chi-Keung Tang HKUST
Bingyi Kang ByteDance Seed
Trace Anything
T:
(i, b
Xi
frume index & coordinate
3D esjectory
10.11-175
Input Frames
Trajectory Field
Any video* can represented 4D with Trajectory Field, dense mapping assigning each pixel in each frame to parametric 3D trajectory. We propose Trace Anything, neural network that predicts the trajectory field with a single forward pass.
Figure I:
ABSTRACT
Building 4D video representations to model underlying spacetime constitutes a crucial step toward understanding dynamic scenes, yet there is no consensus on the paradigm: current approaches resort to additional estimators such depth, flow, or tracking, or to heavy per-scene optimization, making them brittle and hard to generalize. In video, its atomic unit, the pixel, follows continuous 3D trajectory that unfolds over time, acting as the atomic primitive dynamics. Trajectory Field:
Recognizing this, we propose to represent any video* dense mapping that assigns each pixel in each frame to a parametric 3D trajec- tory. To this end, introduce Trace Anything, neural network that predicts the trajectory field in a feed-forward manner. Specifically, for each video frame, the model outputs series of control point maps, defining parametric trajecto- ries for each pixel. Together, our representation and model directly construct a 4D video representation in single forward pass, without additional estimators or global alignment. We develop a synthetic data platform to construct a training dataset and benchmark for trajectory field estimation. Experiments show that Trace Anything surpasses existing methods or performs competitively on the new benchmark and established point tracking benchmarks, with significant efficiency gains. Moreover, it facilitates downstream applications such as goal-conditioned manipulation, simple motion extrapolation, and spatio-tempont fusion. We will release the code and the model weights.
'Here, "any video" extends beyond monocular videos to include image pairs unordered unstructured image collections that capture dynamic

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Understanding dynamic scenes requires more than disjoint reconstruction of 3D space at each time step; it demands modeling how the scene evolves in both space and time. A central challenge toward spatial intelligence Yang
(2025clb); Xiao et al. (2025a) to de- velop 4D video representation that captures the underlying spacetime dynamics in way that is geometrically grounded and scalable. Rather than relying additional estimators such as depth, flow, or tracking, or on heavy per-scene optimization,
observe that the atomic elements of video, its pixels, naturally trace out 3D trajectories in the world, which acts as the atomic primitive of dynamics.
Recognizing this, we propose Trajectory Fields, a versatile 4D representation for any video that associates each pixel in each frame with a parametric 3D trajectory. as illustrated in Figure 2 Unlike prior 4D reconstruction methods that produce disjoint per-frame point clouds (Zhang et al., 2025a Li et al. 2025 Sucar et al. 2025) and rely on estimated optical flow or 2D tracks to build cross-frame correspondences, Trajectory Fields offer more direct and compact way to model scene dynamics.
Input Frames
Per-Pi 3D Parametric Trajectories
Figure 2: Given the input frames (left), trajec- tory field represents the video at the atomic level, mapping each pixel in each frame to 3D trajec- tory, expressed as parametric curve (right).
Building this representation, we propose Trace Anything, feed feed-forward neural network that estimates trajectory fields directly from video frames. As shown in Figure []] with a single forward pass over all input frames, predicts stack of control point maps for each frame, defining spline- based parametric trajectories for every pixel. This design brings three key advantages: (i) its one- pass scheme eliminates intermediate estimators and iterative global alignment, (ii) it predicts all trajectories (per pixel per frame) jointly in a shared world coordinate system, and (iii) it generalizes across diverse inputs, including monocular videos, image pairs, and unordered photo sets.
To support training and evaluation at scale, we develop Blender- platform featuring diverse environments, moving characters, and camera trajectories. It produces photo-realistic renderings with dense annotations, including 2D/3D trajectories, depth, semanties, flow, and camera poses. videos (120 frames each)
From this platform, we release (i) the Trace Anything dataxe/-10,00 for training trajectory field estimation models, and (ii) the Trace Anything benchmark- 200 curated videos for evaluating models' ability to capture motion jointly across all frames.
Trained with our new dataset, Trace Anything achieves state-of-the-art results field benchmark and performs competitively on established point tracking benchmarks, while offer- ing significant efficiency gains. Moreover, our paradigm facilitates downstream spatial reasoning applications, including simple motion extrapolation, matiootemmpoal fusion, and goal-conditioned manipulation.
trajectory
In summary, our contributions are:
We propose Trajectory Fields as atomic-level and versatile 4D video representation, grounded principled formulation.
We present Trace Anything, a feed-forward network that predicts trajectory fields without requiring extra estimators or per-scene optimization.
We develop synthetic data platform for large-scale training and benchmarking of trajectory field estimation.
Extensive experiments on existing and new benchmarks demonstrate competitive accuracy, faster inference, and new capabilities.
RELATED WORK
(Dynamic) 3D scene reconstruction. Reconstructing 3D structure from multi-vies view images is a
long-standing problem in computer vision. Classical Stracture-from-Mooiie (SfM) pipelines (Hart-
2

---

Published as a conference paper at ICLR 2026
ley & Zisserman 2003 Agarwal et al. 2011 Schonberger & Frahm 2016)
proceed in sequential
stages: feature extraction, image matching, triangulation, relative pose estimation, and global bundle adjustment. Deep learning has improved individual components (DeTTone et al. 2018 [Sarlin et al. 2020) yet stage-wise pipelines remain prone to error accumulation. DUSL3R (Wang et al.
2024a)
addressed this by directly predicting 3D pointmaps from image pairs. Fast3R (Yang et al.
2025a).
VGGT (Wang et al. al.] 2025a), x² (Wang et al. 2025b) and MapAnything (Keetha et al. 20255 further
relaxed the pairwise assumption with all-to-all attention, enabling joint reasoning over all frames
and avoiding O(N²) pairwise inference. However, both traditional and learning- based methods generally assume static scenes and sufficient camera baselines, leading to degraded performance in
dynamic settings. To handle monocular videos with dynamics, MegaSAM (Li et al. 2025) integrates optimization-bane SLAM, while Monst3R (Zhang et al. 2025a). POMATO (Zhang et al. 2025b), Easi3R Chen et al. (2025), St4RTrack (Feng et al. 2025). and Dynamic Point Maps (Sucar et al.) 2025) extend D-St networks to dynamic scenes. These methods typically generate dis- joint per-frame point clouds, relying on optical flow or 2D tracks for fross-frame correspondences, and their pairwise inference often requires costly per-scene optimization for global alignment. In contrast, Trace Anything directly estimates trajectory fields that produce dynamic point clouds with cross-frame correspondences, sharing the feed-forwart spirit of Yang et al. (2025a) and Wang et al. (2025a) and performing one-pass inference over all frames.
Point tracking. Particle Video (Sand & Teller 2008) first introduced long-range particle trajec-
tories in videos. Early deep learning methods (Harley al. 2022] Doersch al.) 2022 2023) ap- proached this with global matching and local refinement. Co'Tracker (Karaev et al. 2024b) leveraged
transformer-basee architecture to enable tracking through occlusions, followed by works (Li al.,
2024a Cho et al. 2024) that improved efficiency with 4D correlation volumes. CoTracker3 (Karaev et al., 2024a) further leveraged unlabeled data to boost performance. 3D point tracking remains comparatively new. OmniMotion (Wang et al. (2023) addressed the task via test-time optimization,
while SpatialTracker (Xiao et al. 2024) introduced the first feed-forward 3D tracker by combining 2D tracking with monocular depth priors. DELTA (Ngo al. 2025) achieved dense 3D track- ing using a transformer with upsampling for high-resolution outputs. Concurrently, SpatialTrack- erV2 (Xiao et al. 2025b) scaled training across real and synthetic data, and St4RTrack (Feng al.) 2025) and POMATO/Zhag et af. (2025b) extended 3D reconstruction models for tracking via joint optimization. Unlike prior approaches, our method bypasses monocular depth estimation and 2D trackers and directly predicts dense 3D trajectories in feed-forward manner.
4D representations for NVS. A large class of 4D representations has been developed for novel view synthesis (NVS) in dynamic scenes, aiming to deliver immersive effects such as "bullet time." " Since Neural Radiance Fields (NeRF) (Mildenhall et al.. 2020) introduced implicit volumetric represen- tations, many extensions have incorporated temporal dynamics. One class of methods (Gao et al.)
2021
Li et al. 2021a b) Xian et al. 202T) directly conditions the radiance field on time, treating density and color as continuous functions of space and time. Another class (Pumarola et al.) 2021
Zhang et al.] 202 Park et al., 202 b) Tretschk et al., (02I)) maps observations to a canonical
space and models dynamics via deformation fields. Grid-based approaches (Cao & Johnson] 2023
Fridovich-Keil et al. 2023 Wang et al. 2022; Attal et al. 2023] Liu et al.] (024) discretize the 4D
volume into compact planar factors for efficiency. Also in this line of work, [Wang et al. (2021)
proposed 'neural trajectory fields', with a different formulation and purpose than 'trajectory fields in this study. More recently. 3D Gaussian Splatting (3DGS) (Kerbl al. (023) has been extended to dynamics (Wu et al.) 2024, Yang et af.] 2023; Luiten et al. 2023] Yang et al., 2024 Li et al.) 2024b), improving rendering quality and speed. These efforts focus on photorealistic appearance and typically assume precomputed camera poses or point clouds. Our work is orthogonal: we pro- pose a reometry-centre paradigm that directly infers trajectory fields from raw videos, emphasizing accurate 3D motion modeling. Integrating NVS with our paradigm, e.g., using trajectory fields to initialize dynamic 3DGS models, is a promising future direction.
METHOD
The atomic elements of video, its pixels, naturally trace out 3D trajectories in the world, forming the primitive units of dynamics. Recognizing this, we aim to model dynamic scenes through trajectory fields, a 4D representation that encodes each pixel in each frame as a continuous 3D trajectory over time. In the following, first formalize trajectory fields in Section 3.1 then present Trace Anything
3

---

Published as a conference paper at ICLR 2026
Toput Frames
Contral Polnt
Per-Pisel Trajerterles
Aggregated 1D Reprorstatiin
Skond
Starad
Staad
Slaadd
Head!
Figure 3:
Trace Anything pipeline. Input frames are processed by a geometric backbone consisting of image encoder and fusion transformer. The control point head outputs dense control point maps P, E Dx XwX.xxx x3 where ((a) is the k-th control point for pixel (u, v) in frame 4,- These i,,,
define continuous 3D trajectories (t) (t) via cubic B-splines, yielding a 4D reconstruction.
Section 3.2, a feed-forward network designed to estimate them, and finally describe the overall training scheme in Section 3.3 In this section, we define field as a mapping from domain M to a codomain V:F:M. + V. where M may be a discrete or continuous space, and V may represent scalars, vectors, functions. We provide preliminaries fields in Section A and on parametric curves in Section
3.1 PROBLEM FORMULATION
We formalize trajectory fields, a 4D representation of dynamic 3D scenes in a video. Let { be collection of N RGB frames, where each I; E 3xx captures the scene at different time steps or viewpoints. A trajectory field is defined as
T: [N] [H] [W] + C([O, I),"),
+ (-)
(I)
where [N]. [H]] and [W] denote the discrete sets of frame indices and pixel coordinates, respectively,
and x; 1,,0 [0, 1] - R33 is continuous 3D trajectory for pixel (u, u) in frame 7,. The domain
M [N] x [H] [nl-un classical physical fields, our trajectory field is defined on a discrete pixel grid and imposes no spatial continuity The codomain is V C([0, 1], R²,, the space of continuous functions from j0, 1] to R Here, fo. 1] is a normalized continuous time parameter spanning the full temporal extent. For video input, t 0 corresponds to the first frame and t = 1 to the last frame.
Each trajectory (I) is parameterized as a basine-based curve with 1) control points, defined as
P. E DxDxxx x3
(2)
where (kk s.a.v E 3 is the k-th control point for pixel (u, v) in frame I, with E {0, 1,... D 1}. Each of the N input frames contributes its stack of D control-point maps, so the video produces N x D control-point maps in total.
Given basis functions for (m) the trajectory
D-1
Xi, = Σ
P
i, d& (t).
The form of the basis functions (che( k=0 depends on the type of parametric curve. In our imple- mentation, use cubic B-splines with elamped knots as detailed in Section B
Figure 2 illustrates this formulation of trajectory fields. For any pixel from any frame, its 3D co- ordinate at any time f E [0, 1] can be obtained with Equation (3) This fundamentally differs from existing 4D reconstruction methods that predict per-frame disjoint point clouds and establish cross- frame correspondences via estimated optical flow or 2D tracks. Ideally, two conditions should hold: (C1) pixels in static regions collapse to degenerate trajectories; (C2) corresponding pixels from dif- ferent frames map to the same 3D trajectory.
4

---

Published as a conference paper at ICLR 2026
3.2 NETWORK ARCHITECTURE
Building on the formulation in Section 3.1 we propose a feed-forward network, Trace Anything (Figure 3), which predicts trajectory fields directly from video or unstructured image sets. For each frame, outputs control point maps defining parametric curves over time, enabling trajectory field estimation in single pass. This design eliminates reliance on depth or optical flow and avoids per-scene iterative optimization, providing a compact, efficient approach to modeling 4D scenes.
Geometric backhone. We build Trace Anything upon feed-forward geometric backbone, similar in spirit to recent models (Wang et al. 2025a] Yang al., 2025a). Each frame is first tokenized by an image encoder, followed by fusion transformer that integrates spatio-temporal context across views through interleaved Trame-wise and global attention layers. For sequential video input, we additionally inject temporal index embeddings, while the architecture remains compatible with un- ordered or unstructured image collections.
Control Point Head. Built the backbone features, the control point head outputs dense control point maps P. E IR Dx xxxxxxxxx x3 for each input frame I,. Each pixel (u, u) has D control points (P i, 1 compactly parameterizing its 3D trajectory. Predictions are in a shared world coordi- nate system, with an optional local CP head estimating control points in each frame's local camera
(k) system. The head also predicts per-control-pote confidence scores Li,,, for confidence--ecighted training and filtering uncertain estimates at inference.
Curve evaluation. Given the predicted control points and basis functions (d)(()) continuous trajectories xi,,,o(() are obtained via Equation at any E [0, 1]. In particular,
At evaluation time, the trajectory can be queried
D-1
D-1
Xi, (0) Σ
P())
suu.v (0)
(K)
(0)
X = P
Luu dx(
= P P(D-")
(4)
k=O
where (+) holds for cubic B-splines with clamped knots or for Bézier bases.
To obtain the 3D coordinates of pixel from frame i evaluated at the acquisition time of another frame j. we substitute the corresponding temporal parameter tú into its trajectory:
X (i, v) Xi, ((z).
(5)
In most cases, tú is obtained from metadata frame order. Otherwise, auxiliary timestamp head predicts normalized timestamps I [0, 1). As special case, evaluating each trajectory at frame i's own acquisition time 1; recovers the 3D point map for frame T₁:
X, v) (ta)
(6)
Trace Anything outputs the trajectory field with single network inference for all frames, avoiding pairwise inference and subsequent global alignment, while being self-contained and independent of external estimators for monocular depth, optical flow, 2D tracks.
3.3 TRAINING SCHEME
To train Trace Anything, we directly supervise the accuracy of predicted trajectories. Intuitively, trajectory predicted from frame should, when evaluated at the timestamp of another frame j. land exactly at its ground-truth 3D location at frame j's acquisition time.
Trajectory loss. For pixel (u, v) in frame i, the predicted 3D position evaluated at X, (u, w) (Equation 5)) while the corresponding ground truth is _ i- (u, v). We define the loss as
( v) ||,,,(u,,)) v)
Confidence adjustment. To account for the varying reliability of predicted trajectories across pix- els and control points, we incorporate confidence adjustment. For each control point, the network
(i(R) predicts scalar confidence 2;,,, 0 alongside its 3D coordinates. The confidence associated with X X,
(u, v) is then aggregated using the same basis functions as in Equation 3
D-1
(u = Σ
(K)) iun & (t,))
(8)
k=0
5

---

Published as a conference paper at ICLR 2026
The final confidence-adjusted loss then becomes
Σ Σ
v) (4)s(())) + a log (u,
Laaj-oooff
(9)
(4J) (u.D)E
where Q denotes valid pixels with ground-truth supervision. This adjustment downweights uncertain predictions while discouraging overconfident ones.
Timestamp supervision. When ground-truth timestamps available, we directly supervise Times- tamp Head with an L1 regression loss:
Lime
t
(10)
N
Static regularization. To encourage condition (CI), pixels in static regions should map over- lapped 3D control points. We enforce this by minimizing the variance of their control points:
Estatic
Var (P (Ip()) (.
(11)
Astatic
Rigidity regularization. For pixels segmented as belonging to the same rigid region, their trajec- tories should preserve internal distances across control points. Equivalently, the pairwise distance between any two pixels p. q within a rigid segment should remain constant across control points. We enforce this by minimizing the variance of their distances:
Crgid
Σ
Var (1IP)
P( 112)112)20))
(12)
Rrigial
(Pan)
Correspondence regularization. encourage condition (C2), pixels with known cross-frame correspondences should share identical control points. Let Roor be the set of matched pixels uli,u,v), (','',)) v')). We penalize discrepancies between their control-point sequences:
Lawr
Σ
-P(k) J,u'
(13)
D
Lu
((.,u,)) (d.a DE))
Final objective. The overall loss combines the core trajectory supervision with the above regular- ization terms:
L = Limj--conf + Xeime + Astatic Cstatic + Arigia + Xcom
(14)
4 TRACE ANYTHING DATA PLATFORM
Data-driven modeling of dynamic scenes is limited by the lack of large-scale datasets with dense, reliable annotations. Existing synthetic datasets and generators (Dosovitskiy et al.] 2015 Mayer et al. 2016 Harley et al. 2022, Greff et al.) 2022; Zheng et al.] (023) are typically small and biased toward rigid motion, with sparse short-term annotations, which are insufficient for realistic scene understanding and diverse dynamics. To address this, we develop a scalable 4D Scene Data Platform Blender that synthesizes photo-realistic dynamic scenes with dense ground truth annotations.
Trace Anything dataset. Using our platform, we build dataset whose primary purpose is to train the Trace Anything model trajectory field estimation. The current release contains about 10K unique scenes, each with 120 annotated frames. The collection spans wide range of settings and motions, with examples shown Figure B The dataset exhibits diversity across multiple aspects: (i) Environment diverse indoor and outdoor backgrounds from public asset libraries and procedural generation (Raistrick et al. 2023 2024); (ū) Dynamics- - articulated human characters and movable objects with both rigid and non-rigid motion; (iii) Camera motion smooth trajectories sampled around active regions to mimic natural filming. Rendered RGB videos are paired with per-pixel 2D/3D trajectories, depth maps, camera poses, semantic masks, which facilitate the training scheme introduced in Section 3.3] Since our platform is fully programmable, it can be easily extended with new assets, domains, annotation modalities support future research.
Trace Anything benchmark. To evaluate the task of trajectory field estimation, we construct a benchmark consisting of 200 videos, each with 120 frames. A key difference from established point

---

Published as a conference paper at ICLR 2026
Input Frames
Predicted Cloud Trajoctories
Input Frames
Predicted Point Cloud & Trajectories
Figure 4: Video-based trajectory field estimation DAVIS Perrazzi al. 2016). Trace Any- thing predicts trajectory fields that can yield dynamic point cloud sequences and dense 3D trajecto- ries, while remaining robust complex non-rigid motion and occlusions.
tracking datasets (Koppula et al., 2024) lies the evaluation protocol: point tracking benchmarks evaluate estimated trajectories only for pixels sampled from the first frame (first-to-all), whereas our benchmark evaluates trajectories for pixels sampled from all frames (all-to-all). This requires models not only to follow motion from single starting frame, but also jointly capture dynamics across the entire sequence. In addition, our benchmark provides denser trajectory annotations, cov- ering more pixels per framel, and evaluates in world coordinates, requiring models to reason about both global geometry and motion.
5 EXPERIMENTS
In this section, we evaluate our method across series challenging settings, demonstrating its competitive accuracy, faster inference, and novel capabilities. Please refer to the appendix for im- plementation details and additional experimental results, and the supplementary materials for videos and interactive demos.
5.1 TRAJECTORY FIELD ESTIMATION
We present qualitative results of trajectory field estimation videos and image pairs. Further results on unstructured, unordered image sets are included in Section and qualitative comparisons provided in Section E.4
Video trajectory field. For computational efficiency, we uniformly subsample long sequences to fewer than 60 frames. Figure shows qualitative results DAVIS (Perazzi et al., 2016), covering diverse dynamic scenes. Our predictions faithfully reconstruct both dynamic and static components of the scene, yielding dense, pixel-level 3D trajectories. These trajectories capture motions ranging from near-rigid transformations. such as toy train moving along a track, to highly non-rigid defor- mations, such as humans or animals in motion, while also handling severe occlusions and preserving global scene structure.
Image pair to trajectory field. Our approach can also infer trajectory fields directly from image pairs, effectively reconstructing the implied spatio-temporal dynamics and interpolating intermedi- ate motion. For this experiment, we use BridgeData V2 (Walke et al., 2023), large and diverse dataset of robotic manipulation behaviors. Image pairs are sampled from video sequences with temporal gap of 10-20 frames. As illustrated in Figure 5 given an initial image goal image specifying the desired outcome, our model predicts trajectory field that captures plau- sible 3D trajectories both objects and agents involved. These trajectories can also re- projeecce with estimated camera poses to yield 2D trajectories (see Section E.3 and Figure ffor details). In the context of robot learning. this naturally aligns with goal-conditioned manipulation. where predicted trajectories be interpreted feasible robot end-effector motions (Bharadhwaj al. 2024).
scene and

---

Published as a conference paper at ICLR 2026
Table Quantitative results on video-based trajectory field estimation. CA is reported in 10-2 and SDD in 10-3, Best in bold, second-best underlined.
Method
EP
EPE
CA
SDD,
APDsn
AJ
Runtime (4)
0.518
0.555
DELTA
0.404
0.384
0.425
6.24
1.75
24.7
16.1
231.6
SpaTrackerV2
0.296
0.291
0.366
1.24
1.51
25.5
16.8
178.4
SpuTrackerV22
0.288
0.282
0.323
7.04
1.59
31.3
21.3
174.7
MonsT
0.316
0.258
0.330
McnST3R*
0.295
0.240
0.313
6.49
L.78
20.3
13.4
102.5
St4TTaakk
0.278
0.247
0.370
1.76
27.2
14.6
22.5
SelRTrack
0.232
POMATO
0.272
0.254
0.308
6.78
1.44
26.0
15.4
81.8
POMATO*
0,270
0.2388
0.303
5.71
1.29
28.8
17.2
8088
5.15
Trice Anything
0.234
0.318
0.298
5.09
1.06
34.5
20.6
2.3
Initial Image
Goal Image
Predicted 3D Trajecteries
Projected Trajectories
Figure 5: imagge-air-basee trajectory field estimation (goal-conditioned manipulation) Bridge (Walke et al.] 2003). Given an initial and a goal image, Trace Anything predicts a tra- jectory field that interpolates the 3D motion of both the robot and manipulated objects. We further show the projected 2D trajectories (see Section E.3 and Figure for details).
5.2 QUANTITATIVE EVALUATION
We quantitatively evaluate trajectory field estimation on the Trace Anything benchmark, introduced in Section 4 In contrast to established point tracking benchmarks, which evaluate trajectories only from the first frame, our protocol requires all-to-all predictions: every pixel in every frame must be associated with complete trajectory spanning the entire sequence. Evaluation conducted in two settings: (i) rideo-based inference, where models process 30-frame video clips, and (ii) image- pair-based inference, where trajectories are estimated from two frames sampled frames apart. We present other quantitative results in Section E,5 and ablation study in Section E.6
Metrics. We evaluate reconstruction accuracy using end-point error (EPE). Specifically, EPEmix EPEsta and EPEdya measure the average 3D end-point eπror over all points, static points, and dy- namic points, respectively. To further verify whether the conditions C1 and C2 introduced in Sec- tion 3 are satisfied, we introduce two complementary metrics. Static Degenerucy Deviation (SDD) quantifies the temporal jitter of trajectories in static regions, where smaller values indicate better compliance with CI. Correspondence Agreement ( (CA) measures how consistently dynamic trajecto- ries are predicted from corresponding pixels in different source frames, with lower values indicating better compliance with C2. For video input, also report 3D tracking metries: APDaD (Aver- age Percentage of Points within a threshold), which measures spatial accuracy, and AJ (Average Jaccard), which captures both spatial alignment and occlusion correctness.
Baselines. For video-basee inference, we compare against state-of-the-a dynamic scene recon- struction and point tracking approaches, including CoTracker3 (Karaev al. 2024a) (lifted to
8

---

Published as a conference paper at ICLR 2026
Table Quantitative results image-pair- trajectory field estimation. CA reported in 10-2 and SDD in 10-3. Best in bold, second-bes best underlined.
Method
EPE
EPE.
EPEaya
CA
SDDJ
Runtime (8)
SEA-RAFT VGGT
0.226
0.193
0.427
18.22
0.77
1.91
RAFT-3D
0.281
0.219
0.324
17.50
0.98
0.37
MAS(R
0.220
0.181
0,328
33.99
1.70
2.39
MonST3R
MonST3R*
0.198
0.155
0,321
18,10
1.27
2.44
St4RTrack
0.211
0,202
0,325
15,33
0.63
1.39
St4RTrack*
0.203
0.177
0.318
13.49
0.64
1.41
POMATO
0,181
0.137
0,320
19.58
0.84
4.75
POMATO
0.175
17.72
0.66
4.20
Easi3R
0.284
0.269
0.323
20.41
0.91
5.08
Trace Anything
0.135
0.106
0.304
12.41
0.54
0.20
Table 3: Ablation study on loss terms. CA is reported 10-2 and SDD in 10-3 Best bold.
Method
EPE
EPE, 1
EP
CA
SDD
w/o Lunti
0.305
0.273
0.334
8.52
1.65
w/o Ligid
0.247
0.236
0.321
6.22
1.13
w/o Leor
0.241
0,220
0.303
6.17
1.10
Full loss
0.234
0.218
0.295
5.09
1.06
DELTA Ngo et al., 2025), SpaTrackerV2 (Xiao et al. 20256), MonsT3R (Zhang et al. 2025a), St4RTrack Feng et al., 2025). POMATO (Zhang et al,
3D using VGGT Wang et al. (2025a))
compare against the opti- cal flow method SEA-RAFT (Wang et al. (024b) (lifted to 3D with VGGT Wang et al. (2025a) ) the scene flow method RAFT-3D (Teed & Deng (021). and several 3D reconstruction approaches, including MAS03R (Leroy et al.) 2024), MonST 3R (Zhang et al.) 2025a), St4RTrack (Feng et al.) 2025), POMATO (Zhang al. 2025b) and Easi3R (Chen et al. 2025). We fine-tune SpaTrackerV2, MonST3R, St4RTrack, and POMATO using the same training scheme as our method Section D on Kubric and Trace Anything datasets. These fine-tuned variants are denoted with an asterisk (*).
2025b) and Easi3R (Chen al. 2025). For image- amage-pair pai inference,
Results. Quantitative results are shown in Tables I and 2 Trace Anything achieves the best perfor- mance across all metrics. substantially reducing and-point errors both static and dynamic regions while also attaining the lowest SDD and CA, indicating stronger compliance with consistency con- ditions. Moreover, it runs over an order of magnitude faster than optimization- underscoring the advantage of our one-pass design.
approaches,
Runtime breakdown. As shown in Figure 6 our approach exhibits total runtime that scales ap- proximately linearly with the number of frames. The fusion transformer is the most time-consuming stage, followed by image encoding and curve evaluation. With single-pass inference and no per- scene optimization external estimators, it exhibits clear efficiency advantage, as shown in Ta- bles and
5.3 LOss ABLATION SDUDY
As shown Table 3 all three loss terms contribute to the final performance. Removing the static regularizer Lstatie leads the largest degradation, especially on static regions, confirming its impor- tance for enforcing no motion in static areas. Dropping the rigidity loss Liigdd or the correspondence loss Leor also degrades performance across all metrics. The full loss configuration achieves the best overall results, indicating that these regularizers are jointly necessary for stable and accurate trajectory estimation.
5.4 FACILITATED CAPARILITIES
The Trajectory Field representation and the Trace Anything model facilitate a number additional capabilities beyond basic tracking.

---

Published as a conference paper at ICLR 2026
Analysis
laput Image Pair
Estrpelattdd Tajesterics
lipui Image Pair
Extmgelmnd Trieeterics
encoding
fusian
[N]
Figure 7: selocity-based forecasting. Per-pixel trajectories extrapolated by tangent continuation, with reconstructed trajec- tories red and extrapolated ones in blue.
Figure 6: Stage-wise runtime vs. number of input frames.
Trmmes
Casonical
Trajatoriss
Figure 8: Spatio-temporal fusion. The trajectory field can be leveraged to fuse observations of the dynamic entity across different frames into a canonical frame.
Motion forecasting. The trajectory field inherently encodes 3D point velocities, enabling velocity- based forecasting. As shown in Figure 7 tinuation, allowing dense motion forecasting without additional predictors. We additionally present instruction-bass based forecasting results in Section
per-pixel trajectories be extrapolated by tangent
E.7
Spatio-temporal fusion. In Figure 8 predicted trajectory fields enable dynamic entities observed across multiple frames to be consistently fused back into common canonical frame. This provides mechanism for aggregating partial observations over time, overcoming occlusions and view changes by aligning pixels common reference.
CONCLUSION
We introduced Trajectory Fields, 4D representation that encodes each pixel of each frame 3D trajectory, and Trace Anything, feed-forward model that predicts trajectory fields from input frames, eliminating auxiliary estimators and per-scene optimization. and evaluation, we developed synthetic data platform. Experiments show that Trace Anything delivers competitive accuracy and inference efficiency, while enabling additional useful capabilities.
support large-scale learning
ACKNOWLEDGEMENTS
This work done during the first author's ByteDance Seed internship, and supported in part by Research Grant Council of the Hong Kong SAR under Theme-base Research Scheme, grant no. T22-666/23R..
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, lan Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome day. Communications of the ACM, 14510):105-112, 2011.
Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, and Changil Kim. HyperReel: High-fidelity 6-DoF video with ray-conditioned sam- pling. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Pre- dicting point tracks from internet videos enables generalizable robot manipulation. In European Conference on Computer Vision (ECCV), pp. 306-324. Springer, 2024.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 130-141 141, 2023.
Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen. Easi3r: Estimating disen- tangled motion from dust3r without training. IEEE/CVF International Conference on Computer Vision (ICCV), 2025.
Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon- all all-pair correspondence for point tracking. In European Conference Computer Vision (ECCV), pp. 306-325. Springer, 2024.
Joon-Young Young Lee. Local
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.00 00.699111.99,,... 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems (NeurIPS), 35:16344-1635, 2022.
Carl de Boor. A Practical Guide to Splines. Springer, 1978.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervieed inter- est point detection and description. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 224-236,7 2018.
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point a video. Advances in Neural Information Processing Systems (NeurIPS), 35:13610113666, 2022.
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In IEEE/CVF International Conference on Computer Vision (ICCV) pp. 10061- 10072, 2023.
Alexey Dosovitskiy, Philipp Fischer, Eddy llg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In IEEE/CVF International Conference Computer Vision (ICCV), pp. 2758-2766, 2015.
Gerald Farin. Curves and Surfaces for CAGD: A Practical Guide. Morgan Kaufmann, 2002.
Haiwen Feng, Junyi Zhang, Qianqian Wang, Yufei Ye, Pengcheng Yu, Michael Black, Trevor Darrell, and Angjoo Kanazawa. St4rtrack: Simultaneous 4d reconstruction and tracking in the world. IEEE/CVF International Conference on Computer Vision (ICCV), 2025.
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields space, time, and appearance. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12479-12488, 2023.
Chen Gao, Ayush Saraf. Johannes Kopf, and fia-Bin Huang. Dynamic view synthesis from dynamic monocular video. JEEE/CVF International Conference on Computer Vision (ICCV), pp. 5712-
2021.
11

---

Published as a conference paper at ICLR 2026
Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundaries video generation models. arXiv preprint wxxx:2506.001110 2025.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsuch-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), 2022.
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Track- ing through occlusions using point trajectories. In European Conference Computer Vision (ECCV), pp. 59-75. Springer, 2022.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.11833 2024a.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It better track together. In European Conference on Computer Vision (ECCV), pp. 18-35. Springer, 2024b.
Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fis- cher, Arno Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, Jonathon Luiten, Manuel Lopez-Antequera. Samuel Rota Bulò, Christian Richardt, Deva Ramanan, Sebastian Scherer, and Peter Kontschieder. MapAnything: Universal feed-forward metric 3D reconstruction, 2025, arXiv preprint urXiv:2509.13414
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Gruphics (TOG), 42(4))1-14, 2023.
Skanda Koppula, Ignacio Rocco, Yi Yang. Joe Heyward, Joao Carreira, Andrew Zisserman, Gabriel Brostow, and Carl Doersch. Tapvid-3d: benchmark for tracking any point in 3d. Advances in Neural Information Processing Systems (NeurIPS), 37:82149-82166. 2024.
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision (ECCV), pp. 71-91. Springer, 2024.
Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. Taptr: Tracking any point with transformers as detection. In European Conference on Computer Vision (ECCV), pp. 57-75. Springer, 2024a.
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8508-8520, 2024b.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6498-6508, 2021a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4498-6508. 2021b.
12

---

Published as a conference paper at ICLR 2026
Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang. Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust struc- ture and motion from casual dynamic videos. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025.
Xinhang Liu, Yu- Wing Tai, Chi-Keun Tang, Pedro Miraldo, Suhas Lohit, and Moitreya Chatterjee. Gear-nerf: free- vieppittn rendering and tracking with motion-aware spatio-temporal sampling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19667-19679, 2024.
Ilya Loshchilov, Frank Hutter, al. Fixing weight decay regularization in adam. arXiv preprint 1rXiv:1711.05101, 5(5):5, 2017.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint wxXi:22308.09713. 2023.
Nikolaus Mayer, Eddy lg., Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4040-4048, 2016.
Moritz Menze, Christian Heipke, and Andreas Geiger. Joint 3d estimation of vehicles and scene flow. In ISPRS Workshop on Imuge Sequence Analysis (ISA), 2015.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
Tuan Duc Ngo, Peiye Zhuang, Chuang Gan, Evangelos Kalogerakis, Sergey Tulyakov, Hsin-Ying Lee, and Chaoyang Wang. Delta: Dense efficient long-range 3d tracking for any video. Interna- tional Conference on Learning Representations (ICLR), 2025.
Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss. Refu- sion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7855-7866 IEEE, 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo Martin-Bi Brutala-ruuall.. Nerfies: Deformable neural radiance fields. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5865-5874, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensionall representation for topologically varying neural radiance fields. ACM Transactions on Graphies (TOG), 2021b.
Federico Perazzi, Jordi Pont-Tuset, Brian MeWilliams, Luc Van Gool, Markus Gross, and Alexan- der Sorkine-Homunng A benchmark dataset and evaluation methodology for video object seg- mentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 724-732, 2016.
Les Piegl and Wayne Tiller. The NURBS Book. Springer, 1997.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10318-10327 2021.
Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei. Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang, and Jia Deng. Infinite photorealistic worlds using procedural generation. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12630-12641, 2023.
13

---

Published as a conference paper at ICLR 2026
Alexander Raistrick, Lingjie Mei. Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, Zeyu Ma, and Jia Deng. Infinigen indoors: Photorealistic indoor scenes using procedural generation. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 21783-21794, June 2024.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysís, pp. 1-16. IEFE, 2020.
Peter Sand and Seth Teller. Particle video: International Journal of Computer Vision (IJCV), 80(1):22-91, 2008
Long-range motion estimation using point trajectories.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1938-4947) 2020.
Johannes L Schonberger and Jan- Jan-Michael Frahm. Structure Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4104-4113, 2016.
menuctre-frrmm-tee from-motion revisited. In IEEE/CVF
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A bench- mark for the evaluation of rgb-d slam systems. 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 573-580. IEEE, 2012.
Edgar Sucar, Zihang Lai, Eldar Insafutdinov, and Andrea Vedaldi. Dynamic point maps: A versatile representation for dynamic 3d reconstruction. arXiv preprint arXiv:2503. 2025.
Zachary Teed and Jia Deng. Raft-3d: Scene flow using figidmmotion embeddings. In IEEE/CVF
Conference Computer Vision and Pattern Recognition (CVPR), pp. 3375-8384, 2021.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In JEEE/CVF International Conference on Computer Vision (ICCV), pp. 12959-12970, 2021.
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen- Quan Vuong, Andre Hc, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: dataset for robot learning at scale. In Conference on Robot Learming (CoRL), 2023.
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2/05.05594 2021.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2025a.
Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real- time. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13524- 13534, 2022.
Qianqian Wang, Yen-Yu Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19795-19806, 2023.
Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In IEEE/CVF Conference Computer Vision and Pattern Recog- nition (CVPR), pp. 20697-20709, 2024a.
Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizum Li, Junyi Chen, Jiang- miao Pang, Chunhua Shen, and Tong He. 33 Scalable coamuutation-qqivariaann visual geometry learning, 2025b. URL htt.ttttss//aaatt wwwaaaiww..oggabbbaoo http:oogg./abb//...3333333
14

---

Published as a conference paper at ICLR 2026
Yihan Wang, Lahav Lipson, and Jia Deng. Sea-raft: Simple, efficient, accurate raft for optical flow. In European Conference on Computer Vision (ECCV), pp. 36 54. Springer, 2024b.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for tree-viewpoint video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9421-9431, 2021.
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels 3d space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20406 20417, 2024.
Yuxi Xiao, Longfei Li, Shen Yan, Xinhang Liu, Sida Peng, Yunchao Wei, Xiaowei Zhou, and Bingyi Kang. Spatialtree: How spatial abilities branch out in milms. arXiv preprint arXiv:2512.20617, 2025a.
Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackrr2: 3d point tracking made easy. IEEE/CVF International Conference on Computer Vision (ICCV), 2025b.
Jianing Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025a.
Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10632-10643, 2025b.
Shusheng Yang, Jihan Yang. Pinzhi Huang, Ellis Brown, Zihao Yang. Yue Yu, Shengbang Tong, Zihan Zheng, Yifan Xu, Muhan Wang, al. Cambrian-s: Towards spatial supersensing in video. arXiv preprint 1 I. (04670, 2025c.
Zeyu Yang, Hongye Yang. Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene repre- sentation and rendering with 4d gaussian splatting. International Conference on Learning Repre- sentations (ICIR), 2023.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for figelficyitty monocular dynamic scene reconstruction. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
2024.
Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoint video using layered neural representation. ACM Transactions on Graphics (TOG), 40(4):1-18. 2021.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, De- qing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. International Conference on Learning Representations (ICLR), 2025a.
Songyan Zhang, Yongtao Ge, Jinyuan Tian, Guangkai Xu, Hao Chen, Chen Lv, and Chunhua Shen. Pomato: Marrying pointmap matching with temporal motion for dynamic 3d reconstruction. IEEE/CVF International Conference on Computer Vision (ICCV), 2025b.
Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-seale synthetic dataset for long-term point tracking. In IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pp. 018855-18855 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
1
Introduction
2
Related Work
2
3
Method
Problem Formulation
3.2
Network Architecture
3.3
Training Scheme
4
Trace Anything Data Platform
5
Experiments
5.1
Trajectory Field Estimation
5.2
Quantitative Evaluation
8
5.3
Loss Ablation Sdudy
5.4
5.4
Facilitated Capabilities
Conclusion
10
Fields
17
Parametric Curves
17
Data Platform Samples
18
D Experimental Details
19
Additional Experimental Results
19
E.1
Comparison on Real- World Benchmarks
19
Additional Qualitative Results
22
E.3
2D Trajectories, Dynamic Masks, Scene Flow, and Camera Poses
23
Qualitative Comparison
23
E.5
Additional Quantitative Comparison
24
E,6
Ablation Study
26
Facilitated Capability: Instruction based Forecasting
27
E.8
Additional Analyses
28
iiiitations
30
LLM Usage Declarations
30
16

---

Published as a conference paper at ICLR 2026
A FIELDS
field is mapping defined on a domain M, either continuous space (e.g., IR") or a discrete space (e.g., 2,), to a codomain V. which may be scalar space (e.g., IR), vector space (e.g., R3), or function space (e.g., C (N)). Formally, the field is given by
F M V.
(15)
For instance, the radiance field introduced in Millenhalll and a viewing direction d E S (the unit sphere) to a density o E R and a color E IR.. This is expressed as
2020 maps 3D coordinate R
R 5 R R
(16)
(x,d) - (a,c).
As discussed in Section
3.1 the trajectory field introduced in this work is defined as
T [ x
H| x [W] C( (),
(17)
(i, (i,u,)) v) (-).
where [N], [H]. and [W] denote the discrete sets of frame indices and pixel coordinates, respectively,
and Xi,u, [0, 1] -> RR3 is continuous 3D) trajectory for pixel (u, v) in frame 7;. The domain is
M [N] x [4] x [W], and the codomain is V C[[0, 1], R²), the space of continuous functions
from 1] to R
B PARAMETRIC CURVES
Our trajectory field representation assigns each pixel to a 3D trajectory, expressed as a parametric curve. In computer graphics, parametric curves essential for modeling smooth trajectories and surfaces in applications like geometric design and animation (Farin 2002 Piegl & Tiller 1997). A
spline-based parametric curve x(t) [0, 1] + RR maps parameter
[0, to 3D space, defined
as
x(1) Σ (R)x(1), k=
(18)
where Pa E R3 are control points and (t) are basis functions.
As widely used class, Bézier curves use Bernstein polynomials as basis functions. A Bézier curve of degree d with d + control points Po. Pi...., Pa R is defined as
d
x(t)= - P.B.((()))
(t,a(t)
e²(1 t)d-3
(19)
where Bi,a(() are Bernstein polynomials (Farin 2002). Bézier curves interpolate the first and last control points but lack local control, as adjusting one control point affects the entire curve.
B-spline curves, in contrast, provide local control through a knot vector that defines the parame- ter intervals where basis functions are active. A B-spline curve of degree p with control points Pn-1 E IR is defined
n- x(t) E (t).
(20)
1=0
where (t) are B-spline basis functions determined by knot vector via the Cox-de Boor recur- sion formula (de Boor 1978).
In our implementation of Trace Anything, employ cubic B-splines 3) with clamped, non- uniform knot vectors to parameterize trajectories (t). Each segment is defined by four control points, corresponding to the cubic degree (p 3). A trajectory is defined
n-1
X; (t = Σ
((t)
(k)
Nk.s(t,,
(21)
km
where ((A) fuu.v 3 are control points indexed by u. and (t. are cubic B-spline basis func- tions determined by knot vector (lo, 11,..... 1m). The basis functions computed via the
17

---

Published as a conference paper at ICLR 2026
D = Bezier Curve with N Frame Evaluations
Porameticc Duree Xii
Contill Polgoon
Owd Contrgl ro
P-11
Trame Cowcations
10-
CCordinatt
25
15
1.0
PM
Coordnnae
Figure A: 2D Example illustrating D) control points and N frame evaluations. parametric curve (blue) defined by D = 4 control points (black squares) is evaluated at N -6 timestamps corresponding to video frames (red dots).
Cox-de Boor recursion formula:
if S t < tu+1 for k < + P.
N1,o(1)
if Sts tk+1 for k + p.
(22)
otherwise,
t 44 -N Napp- (1) + 1+ppt1 - 1.+1---11)) 1ktp - lk
(1)
(23)
1ktp11 - 1x+1
for p 1,2,3, with non-zero denominators assumed. For 4,7, 10 control points, we define knot vectors with multiplicity / = and = ensure interpolation the first and last control
points (x (0) = i,a, (1) = P (a-11) 1)) The knot vectors t, are defined as:
(0)
i,0,0
00,0,0,01,,11,
if n 4,
t 0..0,,,,,1/..1/3.1/8. 2/3, 2/ 1,
n 7,
(24)
n 10.
Internal knots have multiplicity up to 3, ensuring C -onctiiiuyy between segments. Knot differences are precomputed for efficient evaluation. Confidence values are interpolated alongside 3D coordi- nates !kk Luu. using the same basis functions (t) Nk,3(t), enabling uncertainty-awer trajectory modeling.
Interpreting and We provide simple 2D illustration in Figure Here, denotes the num- ber of control points that define the underlying smooth trajectory curve (e.g., Bézier or B-spline curve), while N denotes the number of video frames. The variable e [0, 1] is a continuous time parameter of the curve, and each of the N frames corresponds to specific normalized timestamp
C DATA PLATFORM SAMPLES
We illustrate the diversity of our 4D Scene Data Platform in Figure The collection spans wide range of settings and motions, covering diverse indoor and outdoor environments from public asset libraries and procedural generation, articulated human characters and movable objects with both rigid and nin-rigid motion, and smooth camera trajectories sampled around active regions to mimic natural filming.
18

---

Published as a conference paper at ICLR 2026
Figure B: Sample renderings from our 4D Scene Data Platform.
D EXPERIMENTAL DETAILS
We generate training data using Kubric (Greff et al. 2022) and proposed 4D data engine. Specifically, we render 20K videos with 24 frames each using Kubric, with half containing continu- ous camera motion and the other half discrete camera motion, and over 10K videos with 120 frames each from our engine. While Kubric equips models with preliminary ability to capture rigid object motion, it is largely limited to rigid dynamics and textured backgrounds. Our engine complements this with diverse non-rigid object motions and more complex, varied environments.
Our released model uses an image encoder and fusion transformer initialized with Fast3R (Yang et al. (2025a) pretrained weights, while the CP heads are randomly initialized. For the choice of parametric curves, our released model adopts B-splines, as detailed in Section B All models are trained on images at resolution of 5 512 pixels on the longest side, using AdamW (Loshchllov al. 2017) with a learning rate of 0.0001 and a cosine annealing schedule.
In the first stage, we train 20K Kubric videos; in the second stage, we use a mixture of 20K Kubric videos and 10K from our engine. We adopt a batch size 1, with each batch sampling up to 30 frames. Training takes 7.22 days on 32 NVIDIA A100 80GB GPUs. To enable efficient large-scale training, we leverage Flash Attention (Dao et al. 2022 Dao 2023) for improved time and memory efficiency, and apply DeepSpeed ZeRO Stage 2 (Rajbhandari et al.) 2020), which partitions optimizer states, moment estimates, and gradients across machines.
Baseline training datasets. We present a comparison of the training data used by different methods in Table A We group the datasets by label type into Depth or Point Maps and Trajectories, where the latter encompasses supervision from both 3D tracks and scene flows. On Trace Anything bench- mark, we also report fine-tuned variants of competitive baselines (MonsT3R, POMATO, St4RTrack, and SpaTrackerVa on the Kubric and Trace Anything datasets using the same training recipe as our method, designated with in Table and Table 2
E ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results. Please also refer to the supplementary materials for video results, including the presented features, interactive visualization demos, and qualitative comparisons.
E. COMPARISON ON REAL- WORLD BENCHMARKS
KITTI scene flow, We evaluate scene flow on the KITTI Scene Flow 2015 dataset (Menze et al. 2015), comparing our method with VGGT (Wang et al., 2025a), MASt3R (Leroy et al. 2024), and MonST3R (Zhang et al.) 2025a). Because KITTI images have a very wide aspect ratio, which differs from the training distributions of our method and all competing baselines, we fine-tune each approach on the Trace Anything dataset with an aspect ratio of 0.3 to 0.4 for epoch. Despite this brief adaptation, KITTI should still be regarded As shown in the qualitative results in Figure C our method accurately identifies moving objects and recovers their motion direction and velocity, in scenes containing multiple independently
benchmark for our method.
19

---

Published as a conference paper at ICLR 2026
Method
Depth or Point Map Datasets
Trajectory
Datasets
RAFT-3D
FlyingThings3D,
KITTI
MAS(3
Habitat, ARKitScenes, BlendedMVS, MegaDepth, StaticScenes3D, ScanNet+t, CO3D-v2. Waymo, Mapfree, WildRGB, VirtualKITTI, Unreal4K, TartanAir, internal
dataset
VGGT
Co3Dv2, BlendMVS, DL3DV, MegaDepth, Kubrie, WildRGB, ScanNet, HyperSim, Mapillary, Habitat, Replica, MVS-Synlh PointOdyssey, VIrTualKITTI, Aria Synthetic Environments, Aria Digital Twin, Objaverse-like synthetic assets
Easi3R
Training-free (built upon DUS:3R)
MonST3R
PointOdyssey, TartanAir, Spring, Waymo
POMATO
Tartanair, ParallelDomain4D, Carla
PointOdyssey, Dynamic Replica
SpaTrackerV2
VKITTI, TartanAir, Spring, DL3DV, BEDLAM, MVS-Synth, CO3Dv2, COP3D, WildRGBD, ScanNet++, OmniObject3D
PointOdyssey,
Dynamic Replica, Kubric
DELTA
Kubric
St4RTrack
PointOdyssey,
Dynamic Replica,
Kubric
Ours
Kubrie, Trace
Anything
Table A: Training datasets used by different methods.
moving vehicles. We evaluate both short-range scene flow (between consecutive image pairs) and long-range scene flow (across 15 consecutive frames). The quantitative metrics are: 1) End-Point Error (EPE), defined as the L2 norm between the predicted scene flow and the ground truth, and 2) Accuracy (Acc.), defined the percentage of points whose relative error is below 10% The quantitative results in show that our method achieves the best performance across all metrics, outperforming prior approaches in both short- and long-range evaluations.
Input Image
Fstimated Scene Flow
Figure C: Qualitative results scene flow estimation on KITTI. The and flow components color-coded for visualization.
20

---

Published as a conference paper at ICLR 2026
Table B: Quantitative comparison of scene flow estimation on KITTI. The best results are bolded and the second-best are underlined.
Short- Short-Range
Long-Range
Method
EPE 4
Acc.
EPE +
Acc. 1
VGGT
0,415
0.510
0.407
0.536
MonST3R
0.401
0.559
0.409
0,515
MASt3
0.425
0.487
0.438
0.481
Ours
0.375
0.574
0.372
0.589
3D point map estimation. While our approach designed and trained for trajectory field estima- tion. it naturally produces dense 3D point maps as intermediate representation. We provide both qualitative and quantitative comparisons of these point maps against VGGT (Wang al. 2025a),
MASI3R (Leroy et al., 2024), and MonST3R (Zhang al. 2025a) on the Bonn dataset (Palazzolo
et al. 2019) and TUM Dynamics (Sturm et al., 2012). As shown in Figures D and E baseline methods frequently exhibit artifacts around moving non rigid subjects (this is particularly evident with VGGT and MASt3R). In contrast, our approach consistently preserves object geometry more accurately and maintains sharper motion boundaries, resulting in herent scene reconstruction. The quantitative results in Table C show that our method achieves the best performance across all metrics the Bonn dataset and performs par with VGGT on TUM Dynamics.
accurate and visually co-
Iepui
VGGT
MInSTRR
Figure D: Qualitative comparison of 3D pointmap estimation the TUM Dynamics dataset.
VOGT
MonSTIR
Figure E: Qualitative comparison of 3D pointmap estimation the Bonn dataset.
21

---

Published as a conference paper at ICLR 2026
Table C: Quantitative comparison of point map estimation on TUM Dynamics and Bonn. The best results are bolded and the second-best are underlined.
TUM Dynamics
Bonn
Method
Accuracy
Completeness |
Overall |
Accuracy
Completeness
Overall |
VGGT
0.351
0.227
0.289
0.481
0.403
0.442
MonST3R
0.410
0.300
0.355
0.530
0,522
0.526
MASBR
0.553
0.390
0,471
0.577
0,416
0,497
Ours
0.381
0.247
0.314
0.363
0.270
0.317
E.2
ADDITIONAL QUALITATIVE RESULTS
Videos as input. Figure presents the qualitative results of our point map estimation, demonstrating effective handling of complex multi-object motion scenarios compared to baselines. Furthermore, Figure G showcases the superior fidelity of our Trajectory Field estimation. These challenging video inputs highlight the ability of representation model the fine structural motion of objects, accurately capturing dynamics in complex geometries such as the articulated limbs of a robot dog or the caribou horns.
Input Video Franes
VOGT
XASER
MaOSTIR
Ous
Figure F: Qualitative comparison of 3D pointmap estimation on Internet video clips.
Inpor Video Frames
Teajetory Firkd
Inper Video Peames
Trijonory Firld
Figure G: Qualitative results of trajectory field estimation Internet video clips.
Image pairs as input. Figure H presents additional qualitative examples using image pairs as input, highlighting the ability of our trajectory field to capture fine-grained, pixel-level 3D motion.
Unstructured image sets as input. Beyond videos or image pairs, our method also handles unstruc- tured, unordered image sets, a setting not addressed by prior work. The inputs lack both temporal ordering and continuous camera motion, yet our framework by design can also cope with such chal- lenging As shown in Figure our method predicts plausible trajectory fields and camera poses under these conditions. For clarity, we present the input images in chronological order, although no sequence information is provided to the model.
22

---

Published as a conference paper at ICLR 2026
Slegersehs. ewi.ch/yool Siegersell, wwz.dhyaa
auf der
owz.c
Figure H: Additional qualitative results with image pairs as input.
E.3 2D TRAJECTORIES, DYNAMIC MASKS, SCENE FLOW, AND CAMERA POSES
The outputs of Trace Anything can naturally yield 2D trajectories, dynamic masks, scene flow, and camera poses.
2D trajectories. Given the predicted per-pixel 3D trajectories, and with known or estimated camera parameters, can project them into the image plane obtain 2D trajectories. Figure ] overlay the projected 2D trajectories on the first input frame. We also demonstrate this feature in Figure
Dynamic masks. Our method effectively disentangles static and dynamic components. After Trace Anything predicts control points, we compute the variance over the control-point set associated with each pixel; thresholding this per-pixel variance yields dynamic mask that cleanly separates static from dynamic regions, as illustrated in Figure K
Scene flow. Given an input image pair, the scene flow can be obtained as the difference between the two endpoints the predicted trajectories. In Figure we present 4D reconstruction together with the estimated scene flow from image pair in the Spring dataset. To highlight robustness under long-range motion, the two images are chosen from non-consecutive frames.
Camera poses. Since Equation (6) provides world-coordinate point map for each image. follow Yang et al. (2025a) (Sec. 4.2) to estimate focal length, rotation, and translation. Our method handles both continuous camera motion in videos and discrete poses from unordered image sets. As shown in Figure M it correctly recovers camera motion even in dynamic scenes- for example, forward camera movement with perpendicular object motion, objects in free fall captured by unordered image set. In the second example, we present the input images in chronological order, although no temporal information is provided to the model.
E.4 QUALITATIVE COMPARISON
We provide qualitative comparisons of reconstructed point clouds on DAVIS (Perazzi et al. 2016). As shown in Figure our method better preserves fine object details (e.g., the elephant's 's tail and the flamingo's neck), correctly handles complex motion, and disentangles static and dynamic objects. Please refer the supplementary videos for clearer visual comparisons.
23

---

Published as a conference paper at ICLR 2026
8
Figure I: Trajectory fields and camera poses estimated from an unstructured, unordered image set. No sequence information is provided to the model.
E.5 ADDITIONAL QUANTITATIVE COMPARISON
duttr--distribution input. To evaluate our model under out-of-distribution conditions, we con- struct an additional benchmark from PointOdyssey (Zheng et al. 2023), consisting of 50 videos of 30 frames each. Our model has never been trained or fine-tuned on PointOdyssey. As shown in Table D our method maintains advantages across all metrics well inference efficiency.
3D tracking. Although our primary task is trajectory field estimation, our method achieves strong results 3D tracking without task- dataset- fine-tuning. We quantitatively compare against other approaches on the TAPVid-3D (Koppula et al.] (024) benchmark. For each subset of TAPVid-3D (ADT, DrikeTrack, and PStudio), we sample 50 videos of 60 frames each, using
24

---

Published as a conference paper at ICLR 2026
Input Frames
Trajectory Field
Projected 2D Trajectories
Figure Projected 2D trajectories overlaid on the first input frame.
Table D: Quantitative results on out-of-distribution data. CA reported in 10-2 and SDD in 10-3 Best in bold, second-best underlined.
Method
EPF
EPR 4
EPP 4
CA
SDDJ
Runtime (s)
St4RTrack
0.269
0.243
0.325
9.82
1.70
19,9
POMATO
0.344
0.319
0.397
6.24
1.72
84.1
Easi3R
0.368
0.307
0.376
7.10
1.99
125.1
Trace Anything
0.256
0.212
0.319
6.19
1.37
2.3
cry other frame as input, and report APDap (average percent of points within a threshold, measuring spatial accuracy) and AJ (average Jaccard, capturing both spatial and occlusion correctness).
In T we present these quantitative results. Notably, SpaTracker (Xiao et al. 2024) is designed and trained for 3D tracking. Our approach remains competitive, surpassing it some metrics and running orders of magnitude faster. as SpaTracker limited to a fixed number of query points per run, whereas model performs pixel tracking in single forward pass.
25

---

Published as a conference paper at ICLR 2026
Figure Dynamic mask estimation.
Pair
Scene Flow
Recorstrotton
Scone Hlo
Figure L: 4D reconstruction and scene flow from a single image pair. From a non-consecutive image pair in the Spring dataset, our method recovers both the 4D reconstruction and the scene flow, with I and z components color-coded for visualization.
E.6 ABLATION STUDY
Table presents ablation studies on our Trace Anything benchmark, evaluating both the choice of geometric backbone and the type of parametric curve. For the geometric backbone, we compare the effect of initializing the image encoder and fusion transformer with different pretrained models, including Fast3R (Yang et al.) 2025a), VGGT (Wang et al.] (2025)), and "None" (following the Fast3R architecture but with random initialization). For the parametric curve types, we evaluate polynomial curve as well Bézier and B-spline curves with varying numbers of control points.
shown Table polynomial curves underperform because their parameters lack the clear ge- ometric and physical interpretability. In contrast, B-spline curves with ten control points achieve the best overall performance, and accuracy generally improves as the number of control points in- creases. For the backbone, training without pretrained initialization struggles pared with Fast3R, VGGT yields modest gains on certain metrics but incurs substantially higher runtime. Nonetheless, detail or involve large-baseline scenarios. Based on these results, we adopt Fast3R with B-spline curves (10 control points) the default configuration in Trace Anything.
converge. Com-
observe VGGT can be beneficial in settings that demand fine structural
Although method restricts parametric curves control-point-busee ones such as Bézier and B-spline, experimented with polynomial curves during the early development phase.
26

---

Published as a conference paper at ICLR 2026
Figure M: Estimated camera poses the 4D reconstruction.
Input Frames
MonST3R
St4RTrack
POMATO
Easi3R
Ours
Figure N: Qualitative comparison on DAVIS (Perazzi al. 2016). Our method better recovers fine details and handles complex motion while disentangling static and dynamic objects.
E.7 FACILITATED CAPABILITY: ISTRUCTION-BASEES FORECASTING
As an extension of Section 5.4 our method also enables instruction based motion forecasting. With natural language instructions input, leverage image video generation models to produce
27

---

Published as a conference paper at ICLR 2026
Table E: Quantitative results on 3D tracking. Best bold, second-best underlined.
ADT
DriveTrack
PStudio
Method
APDgn f
AT
APD
AJ
APD t
AJ
Runtime (8) +
+ CoTracker
8,9
9.7
5.4
172.4
S14KTrack
15.2
13,4
8.5
7.4
7.2
6.9
18.9
POMATO
18.2
13.6
11.3
7.8
12.2
8.3
69.2
SpaTracker
18.3
17.4
16.0
10.1
16.2
10.3
191.1
Trace Anything
20.5
15.6
15.5
9.6
16.3
10.8
Table F: Ablation study on Trace Anything benchmark. CA is reported in 10-2 and SDD in 10-3, Best in bold, second-best underlined.
Backbone
Curve Type
EPE
EPE, 1
EPE
SDD
Runtime (s)
None
B S-pliine (10 control points)
0,472
0.416
0.505
8.17
1.08
2.3
Fast3R
Polynomial (degree 3)
0.619
0.582
0.673
9.19
1.10
1.8
Fast3R
Bezier (4 control points)
0.299
0.271
0.312
5.08
1.11
1.7
Fast3R
Bezier (10 control points)
0.238
0.224
0.319
6.13
1.08
2.5
Fast3R
B-Spline (4 control points)
0.281
0.264
0.330
6.01
1.08
1.7
Fast3R
Splline control points)
0.237
0.229
0.317
5.81
1.11
2.1
Fast3R
S-Sliine (10 control points)
0.234
0.218
0.295
5.09
1.06
2.3
VGGT
B-Spline (10 control points)
0.236
0.221
0.276
6.11
1.07
7.2
future states conditioned on the instructions, and then apply Trace Anything to lift these forecasts into 2D trajectory fields.
In Figure O we forecast hand-object interactions under different instructions. We use Gemini 2.5 Flash Image (Nano Banana) to generate images of future states corresponding to each instruction, and then apply Trace Anything on the generated image pairs to estimate the trajectory field.
In Figure P forecast robot actions conditioned different instructions. We Seedance 1.0 (Gao et al.] 2025) to generate videos of future states for different instructions, and then apply Trace Anything predict the trajectory fields from the generated videos.
"Gentl place the lime upple." "Lift up
Initial Image
Figure O: Instruction based forecasting. Future states are generated using Nano Banana.
E.8 ADDITIONAL ANALYSES
Temporal interpolation. To evaluate the model's ability to interpolate motion over continu- ous time, we conduct an experiment where each input consists 6 frames at time steps = (0, 0.2, 0.4, 0.6, 0.8, 1.0]. We estimate the trajectory field and then evaluate the predicted trajecto- ries both at these given frames and at the unseen interpolated time steps = f0.1, 0.5, 0.7, 0.9). This evaluation is performed over 100 sets of input frames. As shown in Figure the EPE inter-
28

---

Published as a conference paper at ICLR 2026
"Move forward to the desk."
'Hovve forward to the sofa."
Initial Image
'Move farward meter, then return to the orginaal position.
"Move buchwarrd diagonally to the right meters.
Figure P: Instruction-bart based forecasting. Future states are generated using Seedance 1.0.
polated time steps is slightly higher than the EPE at the given frames. This mild increase is expected and indicates that the model tends to perform marginally better at the original given frames.
EPE at Given and Interpolated Timesteps
0.470
0.465
0.460
0.455
X Given
Interpolated
Mean (Given) 0. 0.468
Mean (interpo sted) 0.474
0.6
Time
Figure Q: EPE at given and interpolated timesteps. EPE at interpolated time steps is slightly higher than at the given frames.
Curve capacity analysis. To validate the choice of parametric curve, compare different spline- based models--ineea siiccewsselinnann Bézier, and B-spline with varying numbers of control points, and additionally compare against the network prediction of our approach and the state-of-
29

---

Published as a conference paper at ICLR 2026
the-art 3D tracker SpaTrackerV2 (iiao et al. 2025b). As shown in Table G cubic B-spline with D=10 control points is sufficiently expressive to model long and complex 3D trajectories. When optimally fit to all trajectories from 20 challenging real-world scenes in TAPVid-3D (Koppula al. 2024) (each spanning 120 frames), the D=10 B-spline achieves lower fitting emoor than Spaaraac-- erV2 (Xiao al. 2025b). This confirms that the spline representation itself is not the bottleneck, and its capacity is sufficient for the vast majority of real-world motion.
Table G: Curve-fitting accuracy. Optimal fitting of various parametric curves (linear, Bézier, B- spline) complex real-world 3D trajectories from TAPVid-3D (Koppula et al. 2024). A cubic B-spline with D=10 control points achieves the lowest error, outperforming state-of-the-ae 3D tracking methods, indicating that curve capacity is not the limiting factor.
Methodology
Curve Type
Control Points
RMSE
Max Error
Linear
2
1,530
2.180
Linear
4
1,251
2.053
Linear
7
0,481
0.838
Linear
10
0.311
0.732
Optimization-Bann Based
Bezier
4
1.029
1.724
Fitting
Bezier
7
0,482
1.208
Bezier
10
0,225
0,474
B-spline
4
1.029
1.724
B-spline
7
0.294
0.639
B-spline
10
0.200
0.444
Spa V2
0.592
1,270
Ours
B-spline
10
0.520
1.157
F LIMITATIONS
Since Trace Anything is trained for trajectory field estimation, we rely on synthetic data to obtain dense annotations. This inevitably introduces domain gap with real-world scenarios. Incorporating partial annotations from real data may help bridge this gap and represents a promising direction for future work.
Our parametric curve representation, with a limited number of control points, has restricted expres- sive power for highly complex motions. In such cases, we mitigate the issue by clipping trajectories into fixed window sizes or downsampling frames. However, these strategies may fail in scenarios such as repeated back-and-fore motion, and performance also degrades as the number of frames increases. A more fundamental solution likely requires training with larger-scale datasets with high quality.
As the first attempt dense per-pixel trajectory field estimation, approach offers efficiency advantages but may be less precise than sparse 3D tracking methods (Xiao et al., 2024 2025b). Incorporating fine-grained point-level estimation from such methods into our framework could be an interesting direction for future research.
G LLM USAGE DECLARATIONS
We declare that Large Language Models (LLMs) were used in a limited capacity during the prepa- ration of this manuscript. Specifically, LLMS were employed for grammar checking, word choice refinement, and typo correction. All core technical contributions, experimental design, analysis, and conclusions are entirely our own. The use of LLMs did not influence the scientific methodology, result interpretation, or theoretical contributions of this research.
30


Reviews:
[{"review_id": "bUXaDcCMzw", "reviewer": "Reviewer_TmW6", "summary": "The paper introduces a new representation called, Trajectory Fields, for 4D geometry and temporal understanding. Given a 2D video, it reconstructs the dynamic scene in 3D using a set of trajectories. Each trajectory is considered as a small point (or atomic primitive) is moving in 3D space. Trained on a proposed dataset, the method outperforms all existing methods on benchmark datasets\n\n---\n\nThe introduced representation is novel and working. However, there are some remaining concerns on missing comparison and unclear details. My initial rating is **4: marginally below the acceptance threshold**, but would like to raise the rating based on the response with a much better understanding of the paper.", "questions": "* $L_{corr}$ in Eq. (13) seems important for ensuring the 3D consistency of trajectory. What's the loss value look like when the model is converged?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "* **Good, novel representation**\n\n  Existing dynamic 3D reconstruction approaches have a disentangled representation for geometry and motion. To overcome it, the paper proposes a single unified representation, called Trajectory field. In a nutshell, this is similar to dynamic 3DGS but without 3D Gaussian attributes. However, new component proposed in the paper is a controlled point concept that represent the trajectory with a set of controlled points and basis function (eg., B-spline)\n\n* **Good accuracy**\n\n  The proposed method outperforms existing methods by a large margin with the fastest runtime in the proposed benchmarks.\n\n* **Dataset**\n\n  If the proposed dataset is released, it will be a good contribution to the community.", "weaknesses": "* **Accuracy on other public benchmark datasets** is missing\n\n  The paper evaluates accuracy on the proposed benchmark only (Table 1 and Table 2), which can be close to the domain that the model is trained on. At least, Table A and B in the supplementary provides accuracy on PointOdyssey and TAP-Vid3D dataset. On PointOdysey, the method performs very competitive as it's on the synthetic domain whereas it has mixed results on the real-world based dataset in Table B. It would be curious to see how the method works on other real-world benchmark datasets (eg., KITTI SceneFlow 2025).\n\n* **Evaluation on 3D pointmap** is missing\n\n  As a related (or downstream) task, the method can naturally output 3D pointmaps. It would be curious to see how the method performs on this 3D pointmap prediction task, comparing to the state-of-the-art reconstruction method, eg., MASt3R, VGGT, and its variant family, also on standard **real-world** dataset such as Bonn, TUM-Dynamics, etc. \n\n* **Confusing notations**\n\n  To be honest, it may need more details to fully understand the equations related to the representation. Here are some questions on the notation for clarification. Assuming we have N input frames (indexed as 0, 1, 2, ..., N-1), \n\n  * Time interval for D? D is the number of control points. Does it mean that there are D control points between each frame pair (eg., 0th and 1st), resulting in N * D total control points in the output sequence? or is it like N frames share the D control points?\n  * Instead of using the basis function, what if we assume a linear motion? Does it break the formulation? (not so sure if I understood the concept correctly). Does the accuracy drop significantly?\n  * Why is $t$ in Eq. (1) defined [0, 1] not [0, N-1]? Does $t=1$ mean the timestep that corresponds to the last frame (ie, N)? or does it mean any other thing?\n\n* **Missing loss ablation study**\n\n  The paper proposes multiple loss functions from Eq. (10) to (13), but ablation study is missing. It would be good if the paper can provide ablation study and validate the design choice on the loss functions.\n \n* **Overlaying all trajectories from all frames?**\n\n  If the method aggregates all trajectories from all frames $I_i$, this will be quite computationally expensive to represent the scene and can be redundant (eg., it is not like that a very minimal set of atomic primitives move over time and represent the dense 4D scene). How is the method actually implemented? Are there any merging mechanism? (or is this question from my misunderstanding of the paper?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KLGAaUj2TK", "reviewer": "Reviewer_YYx7", "summary": "This paper proposes a feed-forward network to estimate the 3D trajectories for every pixels in every timestamp, called trajectory fields. To be specific, they use b-splines to model the 3d trajectories, and the control points of the b-spline for each pixel are predicted by a fusion transformer. Thus the each pixel is traced along time. Furthermore, they develop a synthetic data platform, which could benefit training and benchmarking. Several promising results are shown in both video and paired images datasets.", "questions": "1. How the temporal index embeddings are injected?\n2. How the rigid region in equation 12 is determined?\n3. Why the correspondence regularization in equation 13 is not enough to regularize static consistency (C1)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed model achieves to reconstruct and represent the dynamic monocular videos in a feed-forward manner, making the whole process more efficient.\n2. The idea is complete, and shows promising results on two datasets. \n3. The model can work on both sequential videos and unsequential paired images. \n4. The model proposes an novel framework to fuse the information across different timestamps in one feed-forward inference step.", "weaknesses": "1. The presentation is confusing. 1) The authors try to elaborate trajectory field in a formal ‘field’ definition. However, this definition is kind of counter-intuitive. The trajectory field is defined as a function over a discrete sequential image space, but not a trajectory function over space and time, making it hard to understand for many readers, since the mathematical definition always bothers. This would lead to misunderstandings, like someone may assume the trajectory is continuously distributed in the space, but it’s not. Therefore, I kindly request the authors to further elaborate the definition in the next version for easier understanding. 2) The relationship between control points number D and time t is not well discussed. If I understand right, the control points are located uniformly on the spatial temporal line, with k=0 the first frame (t=0) and k=D-1 the last frame (t=1). But the presentation about this part is extremely unclear.\n2. Following weakness 1, the definition over discrete space limits the spatial interpolation. In other words, it does not support tracing a new point in the space by the learned trajectories. \n3. The experiment is not complete. First of all, since the training requires masks to segment static parts, which is not used for other end-to-end models. So it’s unclear how important this design contributes to the performance. Ablation study is missed here. Second, there is no experiment about motion interpolation along time. For example, given 0, 0.2, 0.4, ..., 1 to predict, eval the model on intermediate time 0.2, 0.3, ... Based on the ablations in Table C, it seems the number of control points do inflence the performance. This shows the temporal interpolation is sensitive to the control point density. Thus this experiment is necessary. \n4. The control point number is fixed, which constrains the input video length. Since the motion is smoothly interpolated by b-spline basises, the motion complexity is constrained by the number of control points. \n5. The motion forecasting in section 5.3 is not persuasive. The tangent continuation only works in linear constant straight motions, no rotations and accelerations are supported. As for instruction-based forecasting, that ability comes from the video generation model but not trajectory field, thus this should not be claimed as an emergent capability.\n\nI’m open to modify my rating to a large extent based on the authors’ response and other reviewers’ opinions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pvvzfNyyUP", "reviewer": "Reviewer_kaqe", "summary": "This paper proposes Trajectory Fields which map each pixel in each frame to a parametric 3D trajectory. To learn the Trajectory Field from videos, the paper proposes Trace Anything module, a feed-forward network to predict control points of B-splines for each pixel in each frame. \n\nTo facilitate the training of the proposed model, this paper builds a synthetic video dataset with dense labels, covering 2D/3D trajectories, depth maps, camera poses, and semantic masks.", "questions": "1. Given a video of a dynamic scene, is the information sufficient to predict 3D trajectories for every pixel, considering that there are occlusions? If not sufficient, then the motivation of the proposed method is problematic. \n\n2. Since the Trace Anything benchmark evaluates trajectories for pixels sampled from all frames instead of only the first frame, how are the current point tracking approaches adapted to this evaluation protocol?\n\n3. For quantitative comparisons in Table 1, 2, A and B, have other baselines been trained on Trace Anything benchmark dataset? and how?\n\n4. What is the fundamental difference between “all-to-all” evaluation and “first-to-all” evaluation?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It is new to formulate 4D video understanding as pixel-level Trajectory Fields.\n\n2. Collecting a densely labeled dataset for dynamic scene understanding is a very good contribution.\n\n3. The overall presentation is clear and easy to follow.", "weaknesses": "1. Modeling 3D trajectories with cubic B-splines over-simplifies the actual movement of 3D points. Complex trajectories cannot be well described with cubic B-splines (p = 3) used in this paper. This means that the proposed method is hard to model complex scenarios. \n\n2. The training of Trace Anything requires dense annotations, which are rarely available in most real-world datasets. When trained solely on synthetic data, the Trace Anything model inevitably suffers from a significant domain gap in real-world applications. This gap encompasses not only differences in geometry and appearance, but also in motion distribution, thereby limiting the scalability of the proposed method.\n\n3. The training objective comprises six terms, yet the paper lacks ablation studies to assess the effectiveness and necessity of each component. Furthermore, these loss functions require extensive additional data annotations, including labels for static and rigid pixels.\n\n4. Experiment settings and results need to be more comprehensive. \n\n4.1. For clarity and reproducibility, it is recommended to explicitly list the training datasets and labels used for all baseline methods.\n\n4.2. In the quantitative comparison on the newly collected synthetic dataset (Trace Anything benchmark), 3D tracking evaluation metrics should be included in addition to trajectory field estimation.\n\n4.3. Quantitative results on real-world datasets should be provided, as this is a crucial evaluation of Trace Anything’s ability to generalize beyond synthetic data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5KcJFEwFI9", "reviewer": "Reviewer_o94T", "summary": "This paper introduces a feed-forward method to directly represent the dynamic video in a continuous trajectory field, which can predict the 2d trajectories (2d in a way it predicts for the image coordinate) for the image sequence at any queried time frame. Specifically, this method uses B-splines as basis functions and uses the predicted feature-based “points” to get the final trajectory. The “control points” are predicted using the large-model image encoder. The authors have shown good performance on video/image pairs to the trajectory dataset DAVIS and BridgeData V2, along with other applications including motion forecasting, spatio-temporal fusion, etc, showing the great potential of this proposed trajectory field representation.", "questions": "- Can the model handle large camera motions? I found that in the \n- Could the authors provide an intuitive explanation for the condition 2? \n- I wondered if the authors have trained their image encoder end-to-end for the experiments they showed in the paper or used a pretrained model here.\n- What about the generalizability of the proposed method? I have seen one experiment in the appendix, how about more different OOD data? Also, does this generalizability come from the large-model image decoder or the continuous trajectory representation itself?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- To directly represent the image sequence/video as a continuous trajectory field is reasonable and has many advantages, such as no need for depth/geometry/camera priors, can directly query the motions, etc. \n- This continuous trajectory field representation has many practical applications, as shown in the paper. For example, the goal-conditioned trajectory generation for robotic manipulation. This could be a great future work.\n- As a direct feed-forward network, the computation is efficient and the model is fast during inference.\n- The authors have also released (will release) a benchmark dataset for the trajectory estimation, which is needed from the research community.\n- The paper is well-written, and all the figures and included videos are of high quality.", "weaknesses": "- The authors could provide more experimental results on video to trajectory datasets to further highlight its robustness and generalizability.\n- I am a bit confused by the definition in the paper referring to 3d points. It seemed that the predicted “3d points” in the paper referred to the (u,v) coordinate at a specific time frame t. It would be better to make this notation clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)