Track3R: Joint Point Map and Trajectory Prior for Spatiotemporal 3D Understanding
Authors: Seong Hyeon Park
Jinwoo Shin
Venue: NeurIPS 2025 poster ()
Keywords: computer vision
Primary area: applications

Abstract: Understanding the 3D world from 2D monocular videos is a crucial ability for AI.
Recently, to tackle this underdetermined task, end-to-end 3D geometry priors have been sought after, such as pre-trained point map models at scale.
These models enable robust 3D understanding from casually taken videos, providing accurate object shapes disentangled from uncertain camera parameters.
However, they still struggle when affected by object deformation and dynamics, failing to establish consistent correspondence over the frames.
Furthermore, their architectures are typically limited to pairwise frame processing, which is insufficient for capturing complex motion dynamics over extended sequences.
To address these limitations, we introduce Track3R, a novel framework that integrates a new architecture and task to jointly predict point map and motion trajectories across multiple frames from video input.
Specifically, our key idea is modeling two disentangled trajectories for each point: one representing object motion and the other camera poses.
This design not only can enable understanding of the 3D object dynamics, but also facilitates the learning of more robust priors for 3D shapes in dynamic scenes.
In our experiments, Track3R demonstrates significant improvements in a joint point mapping and 3D motion estimation task for dynamic scenes, such as 25.8% improvements in the motion estimation, and 15.7% in the point mapping accuracy.

Track3R: Joint Point Map and Trajectory Prior for
Spatiotemporal 3D Understanding
Seong Hyeon Park KAIST
Jinwoo Shin KAIST and RLWRLD
seonghyp@kaist.aco ac.
j jinoosskaist. ac
Abstract
Understanding the 3DD world from 2D monocular videos is crucial ability for AI Recently, to tackle this underdetermined task, end-to-end 3D geometry priors have been sought after, such pre-trained point map models at scale. These models enable robust 3D understanding from casually taken videos, providing accurate object shapes disentangled from uncertain camera parameters. However, they still struggle when affected by object deformation and dynamics, failing to establish consistent correspondence over the frames. Furthermore, their architectures are typically limited to pairwise frame processing, which is insufficient for capturing complex motion dynamics over extended sequences. To address these limitations, we introduce Track3R, novel framework that integrates a new architecture and task to jointly predict point map and motion trajectories across multiple frames from video input. Specifically, our key idea is modeling two disentangled trajectories for each point: one representing object motion and the other camera poses. This design not only can enable understanding of the 3D object dynamics, but also facilitates the learning of more robust priors for 3D shapes in dynamic scenes. In our experiments, Track3R demonstrates significant improvements in a joint point mapping and 3D motion estimation task for dynamic scenes, such as 25.8% improvements in the motion estimation, and 15.7% in the point mapping accuracy.
Introduction
The recent advance of 3D prior models by the point mapping frameworks has enabled robust and accurate 3D understanding in casually taken video frames. Unlike the 3D reconstruction methods that rely on external matching, depth, and pose estimation priors [1], these models directly learn 3D shape priors via end-to-end designs predicting dense 3D points directly from 2D) frames. Notably, DUSt3R [2] has introduced the pair-wise point mapping, which, given a pair of two image frames with unknown camera parameters, maps every pixel one frame to 3D points in the other frame's view. This design allows robust 3D representation disentangling the 3D shapes and the camera motion, providing a strong prior trained at scale.
However, current methods are often challenged when given videos capturing dynamic scenes, affected by moving and deforming objects. For example, the point map task accounts for only variable camera poses, but cannot establish a consistent motion trajectory of the 3D points over the frames. Furthermore, the typical model architectures suffer from the constrained temporal window size of frames, which hinders modeling complex dynamics spanning have been approaches to mitigate the problem, such as injecting motion estimation priors [3], memory bank architecture [4], they cannot generalize well to various tasks in dynamic scenes and fail to learn holistic priors that can disentangle the object dynamics.
wider windows. Although there
To tackle this problem, propose new framework which can jointly predict the point map and their motion trajectory from a multiple number frames in the end-to-end manner, coined Track3R.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(b)
Video Frames
Pair-wise point mapping Joint point mapping and trajectory
Figure 1: The joint point mapping and trajectory prediction. The outputs of a pair- wise point mapping [3] and our method are compared, processing distant frames of a video [5] capturing dynamic scenes. With the first frame as the reference, the 3D points predicted for other frames.
Specifically, our key idea is modeling two disentangled trajectories for each point: one representing the 3D motion and the other representing camera views. We enable this task via generalizing well-adopted point mapping framework, the Siamese transformer architecture with two parallel decoders [6, 7] for the pair-wise point mapping, where the first and the second decoders process distinct frames.
To be specific, assuming W input frames, we introduce a factorized temporal attention over the tokens in the same spatial locations of the frames [8], which enables predicting point maps beyond the frame constraint. For instance, while the vanilla architecture would perform Permutatin on annttattinWW,.... 2) pair- wise iterations, our enhanced architecture would produce an equivalent set of point map sequences in single forward pass,' as depicted in Figure 1. Then, we modify the semantics represented by these sequences: the output by the first decoder represents the 3D motion trajectory in a fixed camera coordinate system, and the other by the second decoder to be 3D points in the variable camera views.
Our model can readily utilize 3D shape priors pre-trained on static scenes scale, which enables training focused on fine-tuning the model in dynamic scenes for learning motion priors. For example, we employ 3D trajectory annotations from human motion datasets [9] the ground truth for training, and synthesize the input frames via dedicated 3D rendering framework for the human bodies [10]. While the dynamic video data in prior art, such as Kubric [11] and Point Odyssey [12], are with sparse set of trajectory annotations, human motion datasets provide a denser set of point trajectories, which can facilitate learning stronger priors for joint 3D geometry and motion in dynamic scenes.
Track3R not only can provide more accurate outputs for the tasks involving 3D motion estimation, but also enables the learning of more robust priors for 3D geometry in dynamic scenes. For example, it achieves significantly improved experimental results compared to strong baselines, e.g., relative 25.5 25.8% on 3D motion estimation and 15.7% on point mapping tasks.
We note that the vanilla architecture and the enhanced architecture essentially identical if W 2.

---

2 Related Work
2.1 3D prior learning
The traditional approaches have focused on learning priors for matching and depth features, which only partially describe the 3D configuration images and prone to the error accumulation in the independent prior models. Since the introduction of end-to-end architectures, such as the image-based neural fields [13] and the feed-forward Gaussian splatting [14], the direct 3D prior learning from image inputs has gained research interest. However, their main focus is typically on reconstructing photometric colors of the image pixels, assuming the camera parameters should be known a priori, which hinders learning robust 3D representation from casually-taken videos [15].
Recently, the point mapping task [2] has been introduced, which choose directly regress the 3D point coordinates without requiring known camera configurations. The point map can effectively disentangle the influence of camera motion in the 3D geometry, which has been shown to enable learning robust 3D shape priors trained at scale. Our goal is to further generalizing the point mapping 3D prior learning for dynamic scenes, which we further discuss as follows.
2.2 3D tracking models
Recently, the conventional 2D video point tracking tasks [16] have evolved towards understanding motion in 3D space [17]. Since the goal of the 3D tracking is predicting the motion trajectory and detecting the occlusions of a query point in video frames, assuming that the camera poses are known priori, the state-of-the-- models, depth estimation, and camera pose estimation methods, e.g., DELTA [18], TAPIP3D [19], and SpatialTracker [20]. We note that these models focus on motion estimation in videos rather than learning holistic prior for 3D shapes, and further discuss the 3D point mapping methods and their extension to dynamic scenes in the follwing subsection.
methods in this field often propose unifying the 2D video point tracking
2.3
3D point mapping models
Since the seminal work in DUSt3R [2], which introduced the Siamese decoding architecture for the pair-wise point mapping, numerous contributions have been made to enhance the generalization performance to various 3D modeling scenarios, namely the application dynamic scenes, and the extended temporal window size for processing multiple frames.
For example, the approaches for fine-tuning from synthetic data is found effective for enhancing the 3D reconstruction in dynamic scenes [3, 21], although they require an external prior models metric depth estimation. Later, the extended task definitions have been
for motion estimation introduced to directly learn priors for the temporal correspondence, or tracking between a fair of input frames [22, 23]. However, these methods are still constrained the pair-wise architecture and often rely on heavy test-time optimization for processing multiple frames. Although the technique for expediting the optimization is concurrent area of research [24], the time cost persists to be high.
In order to achieve the multi-frame modeling without relying on the optimization, new architectures have been proposed, such as the memory bank [4, 25], the multi-view cross attention [26], non- Siamese decoders [27], and employing DINO [28] features as additional inputs [29]. However, the task considered by these models cannot explicitly handle motion in video frames, which hinders generalization to dynamic videos. While our framework also considers an enhanced architecture with the modified attention, we additionally generalize the task for joint point mapping and trajectory prediction, which has significant effect for understanding dynamic scenes.
It is also worth noting concurrent works such as GFlow [30], POMATO [31], St4RTrack [32], which have been devoted to tackle handling dynamic videos in point mapping architectures. For example,
GFlow [30] employs the Gaussian Splatting [33] for better optimization, POMATO [31] proposes a new temporal prediction head for motion estimation, and St4RTrack [32] employs wasr tracking task and architecture
3

---

(a) The joint point mupping and trujectory prediction
Tokenization
Decoder
x
Cross Attention
Point Trajectory
Decoder 2
y
Input Frames
Point maps
view off
(b) Pair-wise point mapping architecture
(c) Track3R architecture (ours)
MLPP
MLP
MLP
MLPP
MI.P
MI.P
Permutation
Trajectory Encoder
Trajectory Encoder
CA
CA
CA
CA
CA
CA
CA
SA
SA
SA
SA
SA
SA
SA
SA
Wil
x
X
X
X
1-1
(-1
ydll
V-1
JIW
Figure Illustration the prediction pipeline in Track3R. The top figure (a) depeicts the overall pipeline, with the first decoder (Decoder 1) and the second decoder (Decoder 2), shared by both the point mapping frameworks. The bottom-left figure (b) illustrates the design of the decoder blocks in the pair-wise point mapping, using the self-attention (SA) and the cross-attention (CA) mechanisms. The bottom-right figure (c) illustrates our architecture for joint point mapping and trajectory, equipped with the proposed trajectory encoder.
Method
In this section, we provide the details of our architecture design for point mapping and trajectory prediction given a video sequence. We denote scalars using normal letters, and tensors using bold letters with superscript denoting frame indices. For example, an input RGB video frame is I E R U/X V33 where U x V is the resolution, and frame tokenization is F E R. NxD, where N U V. with the patch size P and the embedding dimension D. Tensors can be indexed, such as (n) E R., where F = [1),1,
P
(((N). Finally, to highlight that a feature or data for frame i conditioned on the frame j. we use the superscript iljj such as the point map X RUXVx3
3.1 Pair-wise point mapping
Given pair of frames (T,P), P)), the point mapping model aims to predict 2 different 3D points: the first decoder predicting i which represents the 3D coordinate of I' and the second decoder predicting Y which represents the 3D coordinate of I' in the camera view of I Specifically, the vanilla Siamese architecture [2] employs transformer blocks with ross-attention. To describe the operations within a block, we denote the tokens in the first decoder as x;l and the second decoder as Y where E (O...., L} is the block index, with the initialization xili Fs and Y?! = Fj.
In each transformer block (Figure 2b), the cross-attention CA(::-), placed next to the self-attention SA(-), conveys information between the two decoders, followed by the MLP(-) layer, producing the parallel outputs,
x CA(SAAX Y/hh) and Y CA(SA(Y
x-1 MLP (X and Y!! MLP(P MY
(2)
assuming the skip-connections [34, 35] existing in the layers. To produce the final outputs, the DPT head layer [36] employed, which takes these parallel block-wise tokens as the input,
x bi Head (x x ;... x(11
and YJli Head ii Y!! : Y
(3)

---

Although we abuse the same notations SA, CA. MLP, and Head for the two decoders and for all block indices / [I,...L]; we note that their weight parameters are all different.
This pair-wise point mapping model is typically executed twice, for the permutation the in- put frames, which enables downstream tasks, such 2-view geometry, estimating camera poses, etc. When processing larger number of frames, e.g., W > 2, the inference is performed over Permutation(W;22 for all E {l...., W and j { W}.
3.2 Generalized point trajectory prediction
As discussed in the previous section, the point mapping W frames would essentially produce two sets of point map sequences per frame XW E R [[W-I)xUxVs3 and Y E (((W-))UXXXX For example, assuming W 3 input frames, {I II, I.. I would be associated with the point maps from the first and second decoders, respectively:
x 222 and y²: [ Y
Given these sequences, represent the same semantics: the 3D coordinate of in the camera view of IS. Our key idea is to resolve this duplication, by considering a modified definition, X3)I and X representing the 3D coordinate of Is in the frame index of I' and 1 respectively, in the fixed camera view of I That is, we predict the 3D motion trajectory of the frame 1
quickly notice that the outputs from the first decoder, X and X
In order to learn this new task, we employ the confidence-awee regression loss from the original point mapping literature [2], yet provide the 3D trajectory annotations the true targets for X's, which have been relatively sparse in datasets considered by prior art. Therefore, we further facilitate the learning by utilizing human motion dataset, which provide a denser motion trajectory and human body models [9]. where the corresponding video frames can be readily rendered via a dedicated framework [10]. We note that the task definition of Y's remain unchanged, representing the 3D coordinate of a frame pixels mapped the views of other frames.
3.3 Trajectory encoder
Despite the generalized task, the pair-wise architecture is still constrained to process the complex dynamics spanning W > 2. To tackle this problem, we describe our method jointly process multiple frames (i.e., W >> 2). Specifically, we enable it with the trajectory encoder module. This module is based on the factorized temporal attention, which collects the tokens in the same spatial index over the frames to encode the inter-frame dynamics back to each token.
Let consider the frame I paired with others {I' ,..., IW-1} and their corresponding tokens within the intermediate cross-attention stage of the decoder blocks in Equation (1),
X" (k<W) = W WW and Y WWI(k<W) = (Y .... Y WWIW-1
wp
(5)
Intuitively, gathering from a same spatial index, e.g., a stack of tokens [Y (n), Y W (n)] (n)] by indexing each element in Equation (5), can represent the spatio-temporal dynamics of the patch region represented by FW (n). Therefore, projecting this feature onto each token can encode the dynamics. Specifically, apply factorized attention' with causal masks implement the function, coined trajectory attention :)
x WD TA(X!" Wb, ; x(I(ke)) and YWW:= :- TA(YY! &
(6)
where
W/s (n); [X (m), (n),... (n)),, A (m) CA (Y WD (n): (I) (n), ....
x X (n) cA cA(X
(7)
(8)
We adjust the relative position embedding to encode spatial index with the size D/2, and time index with the size D/2.
5

---

However, naively inserting this layer to each decoder block of pre-trained Siamese model results in sub-optimal performance after training on dynamic scenes. fact, prior art finds that retaining strong 3D prior learned from static datasets is crucial for dynamic scenes [3]. Since the trajectory attention deviates the computation graph of a pair-wise model, the model can lose the pre-trained 3D prior. We note that it is also non-trivial to pre-train multi-frame model from scratch, since the training data for 3D geometry is often a pair of images [2], rather than a video stream data.
To address the problem, we aim to minimize the effect of modification in the initial state of the model. Specifically, inspired by model inflation techniques in video transformers [8, 371, which maintain image prior by attenuating the activation of the temporal attentions, we introduce the layerscale LS(-) initialized to a very small scalar [38] to the module, referring to the whole layer as the trajectory encoder TE(::-).
x = TE(X) W X )
(9)
= X + (TA(XX), & {WHikww)))
Y W J (Y! w1s Y (Wilkeey
(10)
E + LS (TA wi, ywllkww))
This design ensures that the model is equivalent to the pair-wise architecture in the initial state, retaining the pre-trained 3D prior.
To summarize, we enhance the model architecture for multi-frame, and generalize the task definition (in Section 3.2) of the corresponding outputs. Throughout the training dynamic scenes, our model initialize from a strong 3D shape prior, and gradually learn to model complex muuli--frame dynamics and predict 3D motion estimation, achieving the joint point mapping and trajectory prior.
Experiment
In this section, we present the experimental details and compare Track3R to state-of-the-art base- lines. In Section 4.1, we provide the training details, such as the training dataset, schedules, and hyperparameters. In Section 4.2, we discuss the experimental details, such as the baselines, check- points, and the inference configurations. Next, we provide the results on the joint point mapping and trajectory prediction in Section 4.3, which aims to evaluate the quality of the motion and shape prior learned by Track3R in dynamic scenes. Then, we further study the downstream task, e.g., the feed-forward camera pose estimation in Section 4.4, which aims compare the ability to disentangle the camera motion in dynamic scenes. Finally, we present the ablation study of the proposed method in Section 4.5.
4.1 Training details
We initialize the Track3R with the pre-trained weight published by MonST3R [3], a pair-wise point mapping architecture trained on dynamic scenes covered by synthetic datasets [12, 39, 40], starting from the pre-trained DUSt3R [2], which is trained on 8M images that capture the real-world static scenes at scale, such as ScanNet [41] and StaticTThing33D [42].
For the fine-tuning, we employ the 3D trajectories and video frames, the combination of SMPL-X human motion trajectory [9] and the associated video frames rendered with BEDLAM [10]. along with the publicly available Waymo [43] dataset. Specifically, we supervise the output X from the first decoder with the 3D trajectory annotation, and supervise the output Y from the second decoder with the ground truth depth maps, registered to the world coordinate system based on the annotated camera poses. We employ the confidence-avee aware and manle-ivvariant regression loss, following the baselines [2. 3].
We train Track3R for 25 epochs using the AdamW optimizer [44] with 25k clips of length W- 6 per epoch, the mini-batch size 16, and the learning rate 10-4, The training takes approximately 36 hours on the system equipped with 8 Vidia A100 GPUs.

---

Table 1: Joint 3D motion and shape performance. The quality of 3D motion estimation (EPEap) and the point mapping (Abs-rel) are compared among Track3R (ours) and the baselines. The baselines that perform additional test-time optimizations are grouped the top block, and the baselines for feed-forward prediction are grouped in the bottom block. The best scores are highlighted with bold numbers, and the best scores within a model category are highlighted with underlines.
iPhone
Sintel
Point Odyssey
Method
FPS
3DE3D
Abs-rel
EDE3D
Abs-rel
EDEaD Abs-rel
DUSt3R (2024)
0.62
0.973
1.212
0.565
0.422
0,933
0.184
MonST3R (2025)
0,41
0.619
0.310
0,401
0.335
0,481
0.090
MegaSaM (2025)
0.97
0.598
0.211
0.372
0.231
0,455
0.091
Align3R (2025)
0.17
0.581
0.290
0.487
0.263
0.670
0.075
(205))(0225)
91.81
0.692
0.419
0.667
0.517
0.771
0.214
Spann3R(2025)
20.77
0.658
0,431
0.523
0.622
0.591
0.231
CUT3R(2025)
27.43
0.701
0.408
0.681
0.428
0.609
0.177
Track3R (ours)
23.81
0.431
0.344
0.312
0.374
0.399
0.165
Table 2: Feed-forward camera pose estimation. The quality of estimating camera translation (ATE and RPE-t) and rotation (RPE-r) are comapred among the feed-forward prediction baselines. The best scores are highlighted with bold numbers, and the second-best scores are highlighted with underlines.
iPhone
Sintel
TUM-dynamic
Method
ATE
RPE-t
RPE-r
ATE
RPE-t
RPE-r
ATE
RPE-t
RPE-r
Fast3R (2025)
0.413
0.294
1.561
0.377
0.150
3.233
0.129
0.111
2.794
Spann3R (2025)
0.552
0.310
2.301
0.329
0.110
4.471
0.056
0.021
0.591
CUT3R (2025)
0.291
0.184
0.848
0.213
0.064
0.596
0.046
0.015
0.473
Track3R (ours)
0.233
0.150
0.694
0.201
0.066
0.621
0.051
0.020
0.517
4.2
Experimental details
We consider 7 different baseline point mapping models, DUSt3R [2]. MonST3R [3], MegaSaM [24], Align3R [21]. Fast3R [27]. Spann3R [25], and CUT3R [4]. We experiment with the checkpoint provided in the official open-source repository hosted by their authors, following the default image processing in each model, e.g., the input dimensions are, the longer side length of 512 in DUSt3R (2],
MonST3R [3], Align3R [21], and Fast3R [27], the longer side length of 672, in MegaSaM [24], and, the square 256 x 256 in Spann3R [25].
Unless otherwise specified, we always choose the temporal window size of the inference W 6 for evaluating our method and the baselines. We note that the pair-wise processing baselines are iteratively executed to match the required window size.
To evaluate the feed-forve forwadd camera pose estimation in Section 4.4, we employ the weighted Pro- crustes solver to derive the relative rotation and translation between the frames, and the weighted least squares solver to estimate the camera intrinsic parameters, similar to the experimental configuration in CUT3R [4].
4.3 Joint point mapping and 3D motion estimation.
In this section, we evaluate the joint point mapping and 3D motion estimation quality. We employ 3 different test datasets covering dynamic scenes: iPhone dataset [45], Point Odyssey [12], and Sintel [46]. iPhone dataset covers the real-world scene, which originally provides monocular video frames, depth, and camera poses collected with the Lidar and IMU sensors [45]. To obtain the 3D motion trajectory annotation, we utilize the track optimization part of the sophisticated stereo video optimization framework [22]. Point Odyssey and Sintel are synthetic datasets rendered with the 3D engine, which provides the ground truth depth and motion trajectories. We note that the validation sets are utilized for these datasets.
7

---

Table 3: Ablation study. The effect of trajectory encoder, the joint training objective, and the training with human motion dataset are studied in terms of the quality of 3D motion estimation (EPEan) and the point mapping (Abs-rel).
iPhone
Sintel
Point Odyssey
Modules
EPE3p
Abs-rel
EPE3p
Abs-rel
EPE3p
Abs-rel
Base Model
0.803
0.879
0.695
0.737
0.980
0.281
+ Trajectory Encoder
0.682
0.593
0.576
0.531
0.704
0.239
+ Joint Objective
0.445
0.357
0.429
0.396
0.463
0.181
+ Human Motion
0.431
0.344
0.312
0.374
0.399
0.165
To provide holistic view over both 3D motion and shape qualities, we borrow metrics from literature for each task: the 3D end- and-point-error (EPEap; a regression quality of 3D motion trajectory) [47] and the absolute relative error (Abs-rel; the accuracy of point mapping to be within 1.25-factor of the ground truth) [4]. We compare Track3R and the baselines: DUSt3R [2), MonST3R [3],
MegaSaM [24], Align3R [21], Spann3R [25], Fast3R [27], CUT3R [4] in Table 1.
To begin with, Track3R demonstrates the strongest results in terms of the quality of 3D motion estimation (ECEgp), e.g., relative 25.8% improvement compared to Align3R [21] in iPhone dataset (0.581 - 0.431). We observe the baselines employing test-time optimizations (upper block in Table can demonstrate overall improved motion estimation performance than the feed-forward prediction methods (lower block in Table 1), except Track3R (ours).
Although the test-time optimization can reinforce overall consistency in point mapping along the frames, and inject external motion prior (e.g., MonST3R [3]), it significantly deteriorates inference speed (FPS). Since our Track3R can directly learn the 3D motion prior during training, instead of the test-time, it can achieve a better motion estimation performance and provides reasonable inference speed as well, e.g., 23.81 (FFS).
In terms of the point mapping accuracy (Abs-rel), Track3R consistently demonstrates the strongest results among the feed-forward prediction methods, e.g., relative 15.7% improvement compared to CUT3R [4] in iPhone dataset (0.408 + 0.344). Although there is gap between the test-time optimizations and ours, the significant improvement compared to the strongest feed-forward baseline (CUT3R (4]) supports the effect of the joint prior learning by our method.
4.4 Downstream tasks
In this section, as the downstream application, we evaluate the herd-forward camera pose estimation in dynamic scenes, and provide visualization of the point maps in the world coordinate system.
First, in the feed-forward camera pose estimation, we consider the benchmark with respect to Sintel [46] and TUM-dynamics [48], following the configuration in CUT3R [4] for dynamic scenes, and also the customized iPhone dataset [45] evaluated with the sensory camera poses. In Table 2, we compare Track3R (ours) and the feed- feed-forward prediction baselines: Fast3R [27], Spann3R [25], and CUT3R [4]. Our method achieves the strongest results in iPhone dataset, and a performance comparable to CUT3R [4] on in Sintel and TUM- TUM-dynamics datasets. For example, Track3R can demonstrate significant improvements in iPhone dataset,, which supports that the 3D prior learn by our method can provide a robust prior that better generalize to the real-world dynamic scenes. We observe that CUT3R [4] particularly performs well on synthetic datasets, which is possible by the pose prediction head trained with the ground truth poses in synthetic datasets. In the extension of our goal to learn joint 3D priors, employing such an idea into Track3R can be interesting future work.
Next, for a qualitative demonstration, we provide the visualization of the 3D points in Figure 3, executed on DAVIS video frames [5]. We find our method tends to demonstrate more consistent point maps over the frames, e.g., the background objects and the scene are consistently depicted, comparing the pair-wise point mapping baseline [3] (with red boxes) and our method performing joint point mapping and trajectory prediction (with blue boxes). This supports the significance of our method, which facilitates learning useful representation for predicting accurate point maps as well, even if affected by complex dynamic scenes.
8

---

Video Frames
Pair-wise point mapping
Track3R (ours)
Figure 3: Qualitative comparison points in dynamic scenes. The 3D points predicted the wise point mapping baseline [3] and Track3R (ours) compared, using DAVIS video samples [5]. Dynamic regions are highlited with the red and blue boxes.
4.5 Ablation study
In this section, we conduct ablation study of the proposed techniques, namely the trajectory encoder, the joint point mapping and trajectory prediction, and the training with human motion data, comparing EPExp (3D motion estimation) and Abs-rel (point mapping) in Table 3. Overall, we find employing each module is significant to the performance of Track3R, where flexibility to process multi-frand frame dynamics is enabled by the trajectory encoder, robust prior learning is achieved via the joint point mapping and motion estimation objective, and additional performance boosts are observed when trained on human motion.

---

Although the pair-wise architecture [2, 3] can produce pointmaps for more than frames by executing multiple pair-wise inferences, its design inevitably enforces the assumption that the distributions of consecutive pointmaps are independent. For example, given {T', 1 I² J. pair-wise model assumes that joint density Pr(Yils Y Y is proportional to Pr (Y Pr(Y Pr((Y)))) However, in practice, including the scenarios represented by our evaluation, there exists an extreme case where I' and I$ are completely non-overlapping, so that the pair-wise model assigns an erroneous estimate of Pr(Yik), which can induce significant failure modes of estimating the joint density. Since Track3R can relax this constraint for multiple frames, it can learn the joint point mapping and trajectory prior that is more close to the true nature of the dynamic scenes.
Discussion
In this section, we discuss the limitation of Track3R and the future research directions in Section 5.1. We also discuss the potential negative societal impact in Section 5.2.
5.1 Limitation
Despite the promising results demonstrated by Track3R, the scarcity of dynamic scenes for training can hinder the generalization performance. To mitigate the problem, we employ the human motion dataset for training. However, the synthesized video imputs can make distribution shift in the visual texture learned in pre-trained model. Therefore, designing new training datasets, self-supervised learning with unlabeled data, or an objective functions robust to the distribution shift can be interesting future directions. It is also worth noting that focus the realistic scenarios where the observation is captured by a monocular video camera, rather than multiple synchronized cameras capturing one scene. Although it would be straightforward to apply Track3R for the synchronized cameras, we believe that there is a room to exploit useful properties, such as epipolar geometry [49] of the synchronized cameras, which is another interesting future direction.
5.2 Potential negative societal impact
While the joint point mapping and motion estimation by Track3R can be beneficial for various video understanding applications, such as novel-view synthesis, depth estimation, and action recognition, the emergence of unexpected behavior within Track3R can lead to misrepresentations the real video data. For those applications that require extremely accurate models for safety-related judgments, such as depth estimation for autonomous driving, the unexpected behaviors must be carefully managed. To ensure the reliability of systems using point tracking predictions, we recommend to conduct thorough investigations and implement robust mitigation strategies to minimize potential risks, thereby increasing the overall safety and effectiveness of these applications.
6 Conclusion
In this paper, we propose Track3R, joint point mapping and motion estimation framework for learning holistic 3D priors dynamic scenes. We tackle the limitations existing point mapping baselines, sub-optimal under complex dynamic scenes. For example, we propose to encode the dynamics of the 3D points over multiple frames beyond the pairs, and generalize the task definition to predict the 3D motion trajectories, as well as the static point maps. Our method significantly improves the expressiveness of the model architecture for dynamic scenes, and enables predicting the disentangled representation of the 3D motion and shapes. In the experiments, we find method can outperform the baselines, for both the 3D motion estimation task and the point mapping task. Overall, our work highlights the effectiveness of jointly solving 3D geometry and motion tasks, and we believe our work could inspire researchers to further leverage in the future.
Acknowledgements. This work was supported by Institute for Information & communications Technology (ITM) grant funded by the Korea MSST)) NN..RS-201--II1 Artificial Intelligence Graduate School "Pogram(KAIST) No. RS2024-005509779, Global AI Frontier Lab), the National Supereomputing Center with supercomputing resources including technical support 200-22255-EE-0 and RLWRLD, Inc.
10

---

References
[1] Johannes Kopf, Xucjian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1611-1621, 2021.
[2] Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric vision made easy. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recogmition, pages 20697-20709, 2024. 1, 3, 4, 5, 6, 7,8, 10
[3] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and -iuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. In International Conference on Learning Representations, 2025, 1, 2, 3, 6, 7, 8, 9, 10
[4] Qianqian Wang, Yifci Zhang, Aleksander Holynski, Alexci A. Efros, and Angjoo Kanazawa, Continuous 3d perception model with persistent state. In CVPR, 2025. 1, 7,8
[5] F. Perazzi, Pont- B. McWilliams, L.. Van Gool, M. Gross, and A. Horkine-Homuug. A henchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016. 2, 8,
[6] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre- training for 3d vision tasks by scoss-view completion. Advances in Neural Information Processing Systems, 35:3502-3516, 2022.
[7] Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow. In ICCV, 2023.
[8] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In JCML, page 2021. 2,
[9] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, body from single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975-10985, 2019. 5,5, 6
[10] Michael J. Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. BEDLAM: synthetic dataset of bodies exhibiting detailed lifelike animated motion. In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 8726-8737, June 2023. 5,
[11] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hermann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2022. 2
[12] Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. ICCV, 2023, 2, 6,7
[13] Qianqian Wang. Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin Marain-Brualla, Noah Snavely, and Thomas Funkhouser. Thrnet: Learning multi-view image-based rendering. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2021.
[14] David Charatan, Sizhe Lester Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19457-19467, 2024. 3
[15] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint arXiv:2407. 13764, 2024. 3
[16] Carl Dnersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in video. Advances in Neural Information Processing Systems, 15:13610-13626, 2022.
[17] Skanda Koppula, Ignacio Rocco, Yi Yang. Joe Heyward, Joao Carreira, Andrew Zisserman, Gabriel Brostow, and Carl Doersch. Tapvid-3d: A benchmark for tracking point in 3d, 2024, URL https:/aaxii 218,
11

---

[18] Tuan Duc Ngo, Peiye Zhuang. Evangelos Kalogerakis, Chuang Can, Sergey Tulyakov, Hsin- Ying Lee. and Chaoyang Wang. Delta: Dense efficient long-range 3d tracking for any video. In The Thirteenth International Conference on Learning Representations. 3
[19] Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragkiadaki. Tapip3d: Tracking any point in persistent 3d geometry. arXiv preprint arXiv:2504. 14717, 2025. 3
[20] Yuxi Xiao, Qianqian Wang. Shangzhan Zhang. Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406 20417, 2024.
[21] Jiahao Lu, Tianyu Huang, Peng Li, Zhiyang Dou, Cheng Lin, Zhiming Cui, Zhen Dong, Sai-Kit Yeung, Wenping Wang, and Yuan Liu, Align3r: Aligned monocular depth estimation for dynamic videos. arXiv preprint ar Xiv:24122.3377, 2025. 3, 7,8
[22] Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holynski. Sterco4d: Learning how things move in 3d from internet stereo videos. arXiv preprint, 2024. 3.7
[23] Edgar Sucar, Zihang Lai, Eldar Insafutdinov, and Andrea Vedaldi. Dynamic point maps: A versatile representation for dynamic 3d reconstruction. arXiv preprint arXiv:2503 2025. 3
[24] Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickic Yc, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. arXiv preprint arXiv:2412.04466 2025. 3, 7,8
[25] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint arXiv:2408. 16061, 2025. 3,7.8
[26] Yuzheng Liu, Siyan Dong, Shuzhe Wang. Yingda Yin, Yanchao Yang. Qingnan Fan, and Baoquan Chen. Slam3r: Real-time dense scene reconstruction from monocular rgb videos. arXiv preprint a/Xiv:24/2009401, 2024. 3
[27] Jianing Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction 1000+ images in one forward pass. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025. 3,7,8
[28] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Trans. Mach. Learn Res., 2024, 2024. URL
https: //tp://////eeeeiewe wwt. form/rrmmm 268.888U662F.. 3
[29] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny, Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Patten Recognition Conference, pages 5294-5306, 2025. 3
[30] Shizun Wang, Xingyi Yang. Qiuhong Shen, Zhenxiang Jiang, and Xinchao Wang. Gflow: Recovering 4d world from monocular video. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7862-7870, 2025.
[31] Songyan Zhang, Yongtao Ge, Jinyuan Tian, Guangkai Xu, Hao Chen, Chen Lv, and Chunhua Shen. Pomato: Marrying pointmap matching with temporal motion for dynamic 3d reconstruction, arXiv preprint arXiv:2504.05666 2025. 3
[32] Haiwen Feng, Junyi Zhang, Qianqian Wang, Yufei Ye, Pengcheng Yu, Michael J Black, Trevor Darrell, and Angjoo Kanazawa. St4rtrack: Simultancous 4d reconstruction and tracking in the world. 2025. 3
[33] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023. 3
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 4
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
12

---

[36] René Ranfil, Katrin Lasinger, David Hafner, Konrad Schindler. and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2020.
[37] Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer. Andrea Vedaldi, and Joao F. Henriques. Keeping your eye on the ball: Trajectory attention video transformers. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
[38] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 32 42, 2021. 6
[39] Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: A high- resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[40] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Schastian Scherer. Tartanair: A dataset to push the limits of visual slam. In 2020 JEFF/RS) International Conference on Intelligent Robots and Systems (IROS), 2020.
[41] Chandan Yeshwanth, dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-22, 2023.
Liu, Matthias NieBner, and Angela Dai. Scannet++: A high fidelficciity
[42] Philipp Schröppel, Jan Bechtold, Artemij Amiranashvili, and Thomas Brox. benchmark and bascline for robust multi-view depth estimation. 2022 International Conference on 3D Vision (3DV), pages 637-645. IEEE, 2022.
[43] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patniik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalahility in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, pages 2446-2454 2020. 6
[44] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2019. URL MtRLttttt////peeeeeveii
[45] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Syxtems, 35:33768-33780, 2022. 8
[46] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. naturalistic open source movie for optical flow evaluation. In Fitzgibbon al. (Eds.). editor, European Conf. on Computer Vision (ECCV), Part IV. LNCS 7577, pages 61 6115.25. Springer- October 2012. 7,
[47] Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8375-8384, 2021. 8
[48] Jürgen Sturm, Nikolas Engelhard. Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 573-580. IEEE, 2012. 8
[49] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003. 10
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: All claims in the introduction and abstract accurately reflect the contribution and scope, which are then verified in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
Section 5 discusses it.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not have theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included the implementation details in Section 4.1 and Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan release workspaces related to data generation and learning the public. We also utilize open source models for training.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
ctsilss.cc/pubbicgugu guwdes/ /CodeSubmis ssi snsoiicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the detail of the training/evaluaaioo setup, dataset, and hyperpa- rameters in Section 4.1 and Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
All experiments are conducted with the same and commonly used random
seed.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources for training Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
http//uulc//thhstttiieeee dnes?
Answer: [Yes]
Justification:
We do not have any ethical concerns regarding the paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact in Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method does not introduce risks for misuse.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited all papers and datasets used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We will release the Pytorch implementation after the acceptance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer: [NA]
Justification: We use existing benchmark datasets and do not have any crowdsourcing datasets experiments in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have human subject in the research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standart components.
Please refer to our LLM policy (https /tt///eer.pp///// cc/Confereecee///000 /LLM) for what should or should not be described.
20


Reviews:
[{"review_id": "jlRuNdq0wD", "reviewer": "Reviewer_YJq8", "summary": "To address the limitations of previous work focused on static scenes and pairwise processing, this paper introduces a novel framework that jointly predicts both the point map and point trajectories. The authors enhance the model architecture to handle multi-frame prediction, improving efficiency over post-optimization-based methods. To obtain ground-truth point trajectories, they leverage a ground-truth human motion dataset and render videos as input. Experimental results demonstrate the effectiveness of the proposed method compared to prior approaches.", "questions": "As noted in the weaknesses, there is a concern that the current point trajectory requirement may be too strict and could introduce ambiguity. It would be valuable to see how the performance changes if post-optimization is also incorporated, along with some discussion comparing this to the VGGT paper.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper reframes the 4D understanding problem as a joint task of point map and trajectory motion prediction. By incorporating a design for multi-frame processing, the proposed framework achieves improved 4D video understanding, enhancing both motion estimation and mapping accuracy and efficiency.\n2. The approach of leveraging ground-truth motion data and synthetic videos is a reasonable way to address the challenge of obtaining ground-truth point trajectories, and its effectiveness is demonstrated through the experiments.\n3. The paper further demonstrates the potential of one-pass multi-frame prediction to replace the need for post-optimization, although there is still room for improvement.\n\nWeaknesses\n1. It is inherently difficult to obtain ground-truth point trajectories for arbitrary videos, since objects or parts that appear in later frames may not be visible in the initial frames, leading to ambiguity. Why not incorporate a visibility mask to indicate whether points are present in each frame, similar to the design in CoTracker? By relaxing this constraint, it may be possible to leverage more data for the joint task.\n2. Most post-optimization-based methods achieve better absolute relative error. However, it’s unclear what the core challenge is in diverging from the pseudo ground truth—does it primarily stem from the strict requirement of complete point trajectories? Would the results of Track3R also improve if combined with post-optimization, and what would the performance look like after such refinement?\n3. The paper does not discuss topics related to VGGT. Although VGGT was originally designed for static scenes, it would be helpful to understand the advantages and disadvantages of the proposed method in handling multi-frame prediction compared to VGGT.", "was_revised": true, "final_justification": "Thank you to the authors for the responses. The answers addressed my questions. The proposed method can further motivate the community to explore one-pass point mapping for dynamic scenes. I will keep my original score.", "rebuttal": "Dear reviewer YJq8,\n\nThank you for your valuable feedback and very detailed comments. We appreciate your remarks on the strengths of our paper, including the task design of joint point mapping and trajectory prediction, one-pass multi-frame prediction framework, and employing motion data and synthetic videos for training. We will address your concerns and questions in the response below.\n\nNote: [x] indicates the reference from the manuscript; [ref-x] indicates extra references for the rebuttal.\n\n---\n\n**Why not incorporate a visibility mask to indicate whether points are present in each frame? By relaxing this constraint, it may be possible to leverage more data for the joint task.**\n\n\nTo begin, we would like to clarify that our method can incorporate visibility masks in training data, and our point mapping function (e.g., mapping objects in later frames to the camera view of an initial frame) already utilizes them for training, which denotes ambiguous or degenerate regions induced by the change of camera views among different frames. For instance, these visibility masks are employed when calculating the confidence-aware and scale-invariant regression loss [2, 3].\n\nIt is also worth noting that the visibility mask can help leverage more training data. For your interest, we have conducted an experiment which employs unlabeled stereo videos taken with VR180 cameras, along with the auto-labeling pipeline that generates 3D trajectory sequence and visibility masks [17]. Specifically, we train our model with extra 30k clips obtained from raw VR180 videos downloaded from YouTube, and test the performance in the same benchmark configuration as Table 1. The experiment indicates that our training with the additional data could provide overall improvements in the performance, e.g., relative 10.2% enhancement in EPE3D (0.431 $\\to$ 0.387 in iPhone) as follows.\n\n\\begin{array}{c | c  c |  c  c |  c  c }\n\\hline\n& \\rlap{~~~\\text{iPhone}} & \\phantom{a} & \\rlap{~~~~\\text{Sintel}} & \\phantom{a} & \\rlap{\\text{Point Odyssey}} & \\phantom{a}  \\newline \n\\text{Method} & \\phantom{a}  \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a}  \\newline\n\\hline\n\\begin{subarray}{c}\\text{Fast3R}\\end{subarray} & 0.692 & 0.419 & 0.667 & 0.517 & 0.771 & 0.214   \\newline\n\\begin{subarray}{c}\\text{Spann3R}\\end{subarray}& 0.658 & 0.431 & 0.523 & 0.622 & 0.591 & 0.231   \\newline\n\\begin{subarray}{c}\\text{CUT3R}\\end{subarray} &  0.701 & 0.408 & 0.681 & 0.428 & 0.609 & 0.177   \\newline\n\\begin{subarray}{c}\\text{Track3R}\\end{subarray} & 0.431 & 0.344 & 0.312 & 0.374 & 0.399 & 0.165   \\newline\n\\hline\n\\begin{subarray}{c}\\textbf{Track3R} \\newline\n\\textbf{(with VR180)}\\end{subarray} & \\textbf{0.387} & \\textbf{0.323} & \\textbf{0.292} & \\textbf{0.358} & \\textbf{0.371} & \\textbf{0.159}  \\newline\n\\hline\n\\end{array}\n\n---\n\n**Would the results of Track3R also improve if combined with post-optimization, and what would the performance look like after such refinement?**\n\nWhile the post-optimization algorithm may improve absolute relative errors, which indicates the (static) point mapping quality, it inevitably biases the final outputs to ignore the motion, which sacrifices the motion estimation performance, such as EPE3D. It also requires heavy computational costs (e.g., all post-optimization baselines demonstrate FPS less than 1, while ours demonstrate 23.81).\n\nIn more detail, the post-optimization algorithm in the baselines [2, 3, 16, 19] focus only on static depth maps derived from 3D point maps and minimize the warping errors between different frames. This may smooth out outliers at a specific frame (such as noisy points around the object edges in Figure 1), but can also lose temporal correspondence between the points.\n\nWe would like to emphasize that our focus is obtaining motion and geometry prior for dynamic scenes, orthogonal to post-optimization algorithms, hence we mainly compare our model to the one-step prediction baselines. For example, we compare the camera pose estimation results (Table 2), which are obtained by applying the iterative least-squares optimization to the outputs of the one-step predictions, following the experimental configuration of CUT3R [4].\n\n---\n\n**Comparison to VGGT.**\n\nBoth ours and VGGT [ref-1] modify the Siamese architecture from prior art to obtain one-pass point mapping models. However, in addition to the key difference in their task configuration, i.e., dynamic scene (ours) vs. static scene (VGGT), another notable difference between the two models is that VGGT directly modifies the cross-frame attentions within the base Siamese architecture for point mapping, while ours introduces the trajectory encoder and avoid modifying the attention layer to prevent catastrophic forgetting in the pre-trained model.\n\nInstead, VGGT chooses to encode image features using a foundation vision model (e.g., DINO [ref-2]) to provide a strong prior for 3D tasks to their model, and focuses on fine-tuning the model for the point mapping task. Since DINO can act as a strong and robust representation for static images, it significantly contributes to VGGT demonstrating state-of-the-art performance on static scene configurations. We believe their approach, employing image foundation models for 3D tasks is a reasonable direction for the point mapping task, and extending to dynamic scenes, such as employing video foundation models, can be an interesting future work.\n\n[ref-1] Wang et al., \"VGGT: Visual Geometry Grounded Transformer\", CVPR 2025\n\n[ref-2] Oquab et al., \"DINOv2: Learning Robust Visual Features without Supervision\", Transactions on Machine Learning Research (2024)"}, {"review_id": "Id2AUKEM5g", "reviewer": "Reviewer_XKzt", "summary": "This paper introduces Track3R, a novel framework for predicting point maps and trajectories from monocular videos. It makes two primary contributions: First, it proposes a new formulation that jointly predicts a 3D point map (representing object shape relative to a moving camera) and a 3D motion trajectory (representing object dynamics in a fixed reference frame). Second, it introduces a novel \"Trajectory Encoder\" module, based on factorized temporal attention, which allows a Siamese-style architecture to efficiently process multiple frames in a single forward pass while retaining valuable priors from pre-trained models. The authors demonstrate through extensive experiments that Track3R significantly improves performance on joint 3D motion estimation and point mapping tasks in dynamic scenes, outperforming feed-forward baselines and achieving competitive results with much slower test-time optimization methods.", "questions": "I am willing to raise my score if the points mentioned in the 'Weaknesses' section are addressed satisfactorily.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. Clear Problem Formulation: The core idea of disentangling camera motion from object motion by jointly predicting point maps and 3D trajectories is clear and well-motivated. This is achieved by repurposing one of the decoder's outputs to represent the 3D coordinates of a point at different time indices but from a fixed camera view, effectively defining its 3D motion trajectory.\n\n\n2. Elegant Architectural Design: The proposed Trajectory Encoder is a clever architectural modification. The use of factorized temporal attention to process multiple frames is efficient. Crucially, the integration via a layerscale initialized to a small value is an insightful technique to \"inflate\" a pre-trained pairwise model for a multi-frame task without suffering from catastrophic forgetting.\n\n\n3. Impressive Empirical Results: The experimental results are strong and convincing. Track3R shows marked improvements over state-of-the-art feed-forward methods in both motion estimation (EPE_3D) and point mapping (Abs_rel). The fact that it achieves this with a high inference speed (23.81 FPS) and is competitive with methods requiring costly test-time optimization is a significant practical advantage.\n\n\n4. High Clarity and Readability: The paper is well-written, logically structured, and easy to follow. The motivation is clearly laid out, the method is described in sufficient detail, and the notation is clear, making the method easy to understand.\n\n\n\n# Weakness\n\n1. Comparison with 3D Tracking Methods: While the proposed method shows impressive performance on jointly predicting point maps and tracking trajectories, a comparison with existing 3D tracking methods (e.g., DELTA [1], TAP3D [2], SpatialTracker [3]) on standard tracking metrics would greatly help readers understand how Track3R compares to them.\n\n2. Insufficient Related Work: There are other works that also focus on jointly predicting point maps and tracking, such as GFlow [4], POMATO [5], St4RTrack [6] which should be discussed in the related work section.\n\n3. Suggestions for Presentation:\n* The term \"shape\" could be more appropriately replaced with \"geometry\" throughout the paper for better technical precision.\n* The aesthetics of Figure 2 could be improved (e.g., by avoiding a high-contrast color palette).\n* In Line #20, the statement \"The recent advance of 3D prior models\" should be followed by citations to the relevant works.\n\n\n[1] DELTA: Dense Efficient Long-range 3D Tracking for Any video     \n[2] TAPIP3D: Tracking Any Point in Persistent 3D Geometry     \n[3] SpatialTracker: Tracking Any 2D Pixels in 3D Space     \n[4] GFlow: Recovering 4D World from Monocular Video     \n[5] POMATO: Marrying Pointmap Matching with Temporal Motion for Dynamic 3D Reconstruction     \n[6] St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World", "was_revised": true, "final_justification": "This paper presents a solid and technically sound method for joint 3D motion estimation and point mapping tasks, and it achieves better performance than other methods. In the rebuttal, the authors provided more comprehensive experimental comparisons with other methods and discussed more related works. This has addressed my concerns, and I therefore recommend \"accept\" for this paper.", "rebuttal": "Dear reviewer XKzt,\n\nThank you for your valuable feedback and very detailed comments. We appreciate your remarks on the strengths of our paper, including the clear problem formulation and presentation, the architectural designs, and the empirical results. We will address your concerns and questions in the response below.\n\nNote: [x] indicates the reference from the manuscript; [ref-x] indicates extra references for the rebuttal.\n\n---\n\n**Comparison with 3D Tracking Methods (TAPIP3D, SpatialTracker, DELTA)**\n\nFor the comparison, we allow 3D tracking models [ref-3, ref-4, ref-5] to access ground truth camera poses and simulate coordinate transforms between video frames (we note that the 3D tracking does not support point mapping functions). Since our focus is modeling 3D trajectory of pixel points in a free world space (without predicting occlusions with respect to a specific camera pose and frame pixels), we choose to experiment with the configuration of Table 1.\n\nOverall, the 3D tracking methods demonstrate relatively better motion estimation performance (EBE3D), similar to our method, than one-pass point mapping baselines [4, 20, 22]. However, the 3D tracking methods cannot match the point mapping performance (Abs-rel), even if the ground truth camera poses are provided. We argue these results are anticipated since the 3D tracking models are directly supervised for motion estimation, while being unable to learn strong 3D geometry prior from static 3D point mapping data.\n\n\\begin{array}{c | c | c  c |  c  c |  c  c }\n\\hline\n& &  \\rlap{~~~\\text{iPhone}} & \\phantom{a} & \\rlap{~~~~\\text{Sintel}} & \\phantom{a} & \\rlap{\\text{Point Odyssey}} & \\phantom{a}  \\newline \n\\text{Model Type} & \\text{Method} & \\phantom{a}  \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a}  \\newline\n\\hline\n& \\begin{subarray}{c}\\text{TAPIP3D}\\end{subarray} & 0.499 & 0.717 & 0.331 & 0.626 & 0.441 & 0.636   \\newline\n\\text{3D Tracking} & \\begin{subarray}{c}\\text{Spatial Tracker}\\end{subarray}& 0.443 & 0.699 & 0.339 & 0.582 & 0.422 & 0.791   \\newline\n & \\begin{subarray}{c}\\text{DELTA}\\end{subarray} &  \\underline{0.434} & 0.501 & \\underline{0.320} & 0.611 & \\textbf{0.397} & 0.244   \\newline\n\\hline\n& \\begin{subarray}{c}\\text{Fast3R}\\end{subarray} & 0.692 & 0.419 & 0.667 & 0.517 & 0.771 & 0.214   \\newline\n\\text{Point} & \\begin{subarray}{c}\\text{Spann3R}\\end{subarray}& 0.658 & 0.431 & 0.523 & 0.622 & 0.591 & 0.231   \\newline\n\\text{Mapping} & \\begin{subarray}{c}\\text{CUT3R}\\end{subarray} &  0.701 & \\underline{0.408} & 0.681 & \\underline{0.428} & 0.609 & \\underline{0.177}   \\newline\n& \\begin{subarray}{c}\\textbf{Track3R (ours)}\\end{subarray} & \\textbf{0.431} & \\textbf{0.344} & \\textbf{0.312} & \\textbf{0.374} & \\underline{0.399} & \\textbf{0.165}   \\newline\n\\hline\n\\end{array}\n\n[ref-3] Zhang et al., \"TAPIP3D: Tracking Any Point in Persistent 3D Geometry\" (2025)\n\n[ref-4] Xiao et al., \"SpatialTracker: Tracking Any 2D Pixels in 3D Space\", CVPR 2024\n\n[ref-5] Ngo et al., \"DELTA: Dense Efficient Long-range 3D Tracking for Any video\", ICLR 2025\n\n---\n\n\n**Concurrent works and editorial suggestions.**\n\nThank you for pointing out GFlow [ref-6], POMATO [ref-7], St4RTrack [ref-8], which are concurrent works that aim similar tasks as our method, which we will include in our updated manuscript. While these models partially share similar design as ours, we will highlight the key differences and discuss in detail. For example, the Gaussian Splatting in GFlow  [ref-6] vs. one-pass prediction in ours, the separate temporal head for motion estimation in POMATO [ref-7] vs. the modification of the semantics of point mapping $X$ in ours, and the pairwise processing architecture in St4RTrack [ref-8] vs. the multi-frame processing architecture in ours.\n\nWe will also clarify and revise the suggested terminology and improve the color palette of the figures.\n\n[ref-6] Wang et al., \"GFlow: Recovering 4D World from Monocular Video\" AAAI 2025\n\n[ref-7] Zhang et al., \"POMATO: Marrying Pointmap Matching with Temporal Motion for Dynamic 3D Reconstruction\" (2025)\n\n[ref-8] Feng et al., \"St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World\", ICCV 2025"}, {"review_id": "ExjEtxB1xL", "reviewer": "Reviewer_bXsD", "summary": "This work addresses the under constrained problem of 3D reconstruction using 2D monocular videos. Prior works have successfully provided shapes of objects and camera parameters. However, they struggle in cases with object deformation, and their architectures are compatible only for processing pairs of frames, making it difficult to capture complex motion. This work proposes Track3R, which jointly predicts point map (camera poses) and motion trajectories across multiple frames (motion trajectories). The experimental results show 25.8% improvement for motion estimation and 15.7% for point mapping.", "questions": "In addition to the \"Strengths And Weaknesses\" section, does tuning with human motion bias the model to improved performance for Human tracking. Do you observe it affecting the performance of other objects?", "limitations": "Yes, \n\nI think additional discussion/ablations on non-human object tracking would help get a better understanding of the approach.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The model's ability to process multiple frames is a valuable contribution to the challenging problem of 3D understanding.\n\n2. The reported performance gains are significant, with a 25.8% improvement in 3D motion estimation and a 15.7% provements in point mapping accuracy.\n\n**Weaknesses**\n\n1. Computation comparison are needed (#parameters, #GPU hours, training and inference times). \n\n2. ⁠The contributions are not clear (atleast not clearly stated).\n     a.  Adapting to multiple frames comes from [8]. \n     b. The main novelty is the two path simiese for disentangling motion and camera\n\n3. Sec 3.1 and Sec3.2 are a bit difficult to understand because of the notations.  X^{i|j} which represents the 3D coordinate of frame I^i and does not involve j confuses the part. I think it gets cleared later on to learn motion but better notations would help. \n\n4. Making the captions self-sufficient for Fig1. will improve the quality of the paper.\n\n\n\n\n\n\n\n[8] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for videounderstanding? In ICML, page 4, 2021.", "was_revised": true, "final_justification": "The updated computation parameters, the clarification on the novel integration of factorized attention, and the addition of experiments on non-human subjects have all effectively addressed my previous queries. I am maintaining my positive score for the paper.", "rebuttal": "Dear reviewer bXsD,\n\nThank you for your valuable feedback and very detailed comments. We appreciate your remarks on the strengths of our paper, including the multi-frame processing designs, and the empirical results. We will address your concerns and questions in the response below.\n\nNote: [x] indicates the reference from the manuscript.\n\n---\n\n**Computation comparisons.**\n\nWe note that the comparison for inference speed (frame per second; FPS) is available in Table 1, for example, ours can demonstrate 23.81 while post-optimization baselines demonstrate FPS less than 1.\n\nOther computation factors, such as #parameters, #GPU hours (training time), we provide the comparison to the best of our knowledge, referring to the papers and the open-source repositories of the baselines [4, 20, 22], as follows.\n\n\\begin{array}{c | c  c  c c}\n\\hline\n\\text{Method} & \\text{GPU Type} & \\text{\\\\# parameters} & \\text{GPU hours} & \\text{Training}  \\newline\n\\hline\n\\begin{subarray}{c}\\text{Fast3R}\\end{subarray} & \\text{A100} & \\text{648M} & - &  \\text{from scratch}  \\newline\n\\begin{subarray}{c}\\text{Spann3R}\\end{subarray} & \\text{V100} & \\text{803M} & 1920 & \\text{fine-tuning from DuST3R}   \\newline\n\\begin{subarray}{c}\\text{CUT3R}\\end{subarray} & \\text{A100} & \\text{665M} & 18831 & \\text{from scratch}   \\newline\n\\begin{subarray}{c}\\textbf{Track3R (ours)}\\end{subarray} & \\text{A100} & \\text{688M} & 288 & \\text{fine-tuning from MonsT3R}   \\newline\n\\hline\n\\end{array}\n\n---\n\n**Remarks on the architectural novelty and key contributions.**\n\nWe would like to emphasize that the factorized attention [8] is a part of our trajectory encoder module to enable information sharing over multiple frames. However, the merit of our architecture design, as recognized by the reviewers XKzt and p1Bw, is our specific way of integrating the module to the base architecture and inflating the pairwise model towards the multi-frame processing, which effectively retains strong prior in a pre-trained model.\n\nIn addition to the architectural novelty, we also would like to note that we tackle the scarcity of 4D training data for training. As pointed out as the strength of our paper by the reviewer YJq8, our method of leveraging ground-truth motion data from synthesized human action videos provides a significant effect on the point mapping and trajectory prediction performance, e.g., Table 3.\n\n---\n\n**Terminology and editorial suggestions.**\n\nThank you for pointing out the ambiguity of $X^{i|j}$, where its semantics may vary depending on whether it is used under the context of pairwise baselines, or the context of our method which incorporates motion over the frames. We will revise the presentation, and make a further effort to clarify the terminology. We will also elaborate more on the captions for Figure 1, e.g., include the definition of the trajectory prediction ($X$) and the point mapping ($Y$), and the difference between our method versus the pairwise models. \n\n---\n\n**Experiment with non-human object data.**\n\nAs also suggested by the reviewer YJq8, we have conducted an experiment which employs non-human stereo videos taken with VR180 cameras, along with the auto-labeling pipeline that generates 3D trajectory sequence [17]. Specifically, we train our model with extra 30k clips obtained from raw VR180 videos downloaded from YouTube, and test the performance in the same benchmark configuration as Table 1. The experiment indicates that our training with the additional data could provide overall improvements in the performance, e.g., relative 10.2% enhancement in EPE3D (0.431 $\\to$ 0.387 in iPhone) as follows.\n\n\\begin{array}{c | c  c |  c  c |  c  c }\n\\hline\n& \\rlap{~~~\\text{iPhone}} & \\phantom{a} & \\rlap{~~~~\\text{Sintel}} & \\phantom{a} & \\rlap{\\text{Point Odyssey}} & \\phantom{a}  \\newline \n\\text{Method} & \\phantom{a}  \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a}  \\newline\n\\hline\n\\begin{subarray}{c}\\text{Fast3R}\\end{subarray} & 0.692 & 0.419 & 0.667 & 0.517 & 0.771 & 0.214   \\newline\n\\begin{subarray}{c}\\text{Spann3R}\\end{subarray}& 0.658 & 0.431 & 0.523 & 0.622 & 0.591 & 0.231   \\newline\n\\begin{subarray}{c}\\text{CUT3R}\\end{subarray} &  0.701 & 0.408 & 0.681 & 0.428 & 0.609 & 0.177   \\newline\n\\begin{subarray}{c}\\text{Track3R}\\end{subarray} & 0.431 & 0.344 & 0.312 & 0.374 & 0.399 & 0.165   \\newline\n\\hline\n\\begin{subarray}{c}\\textbf{Track3R} \\newline\n\\textbf{(with VR180)}\\end{subarray} & \\textbf{0.387} & \\textbf{0.323} & \\textbf{0.292} & \\textbf{0.358} & \\textbf{0.371} & \\textbf{0.159}  \\newline\n\\hline\n\\end{array}"}, {"review_id": "b2tJrlg7ut", "reviewer": "Reviewer_p1Bw", "summary": "Track3R introduces a framework for joint point mapping and motion trajectory prediction from monocular videos, modeling separate object motion and camera pose trajectories across multiple frames. It reports 25.8% improvement in motion estimation and 15.7% in point mapping accuracy over feed-forward baselines in dynamic scenes.", "questions": "Why does Track3R underperform test-time optimization methods in point mapping? What specific limitations (e.g., model capacity, training data) cause this? A clear plan to narrow this gap could improve my rating. \nSynthetic human motion data may not reflect diverse real-world scenes. Can you provide evidence of performance on varied, non-synthetic data? This could address quality concerns.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The joint task of predicting point maps and motion trajectories is a meaningful advance, extending pairwise methods to multi-frame processing (Section 3). The trajectory encoder’s factorized attention is a clever adaptation for capturing dynamics (Section 3.3). Experiments on iPhone, Sintel, and Point Odyssey datasets show Track3R outperforms feed-forward baselines in 3D motion (EPE3D) and point mapping (Abs-rel) tasks (Table 1, Section 4.2). The ablation study (Table 3) clearly validates each component’s contribution. The approach has strong potential for 3D vision applications like autonomous driving or AR, where dynamic scene understanding is key (Section 5.2). The paper is well-organized, with helpful figures (Figure 2) and a detailed training setup (Section 4.1). Visualizations (Figure 3) effectively show qualitative gains.\n\n\n**Weaknesses**: Track3R trails test-time optimization methods (e.g., MegaSaM, MonST3R) in point mapping accuracy (Table 1), limiting its claim of robustness. Heavy reliance on synthetic human motion data (Section 4.1) raises concerns about real-world generalization. The trajectory encoder builds on existing factorized attention [8], and the paper doesn’t fully clarify its novelty over prior video transformer work. The joint task, while innovative, feels like an incremental step given related multi-frame efforts.", "was_revised": true, "final_justification": "The rebuttal from the authors for my review and also for other reviews from other reviewers address my concern about generalizability of the method to non synthetic data. I increased my score accordingly and this paper now fall in the region for borderline accept.", "rebuttal": "Dear reviewer p1Bw,\n\nThank you for your valuable feedback and very detailed comments. We appreciate your remarks on the strengths of our paper, including the task design of joint point mapping and trajectory prediction, the architectural designs, and the strong potential for 3D vision applications. We will address your concerns and questions in the response below.\n\nNote: [x] indicates the reference from the manuscript.\n\n---\n\n**Why does Track3R underperform test-time optimization methods in point mapping? What specific limitations (e.g., model capacity, training data) cause this?**\n\nWe first emphasize that post-optimization methods require heavy computational costs, e.g., all of them considered in our paper demonstrate FPS less than 1, while ours demonstrate 23.81. While such a post-optimization algorithm may improve absolute relative errors (by paying a huge cost), which indicates the (static) point mapping quality, it inevitably biases the final outputs to ignore the motion, which sacrifices the motion estimation performance, such as EPE3D. In more detail, the post-optimization baselines  [2, 3, 16, 19] focus only on static depth maps derived from 3D point maps and minimize the warping errors between different frames. This may smooth out outliers at a specific frame (such as noisy points around the object edges in Figure 1), but can also lose temporal correspondence between the points.\n\nEmploying more training data can be a useful solution to further boost the performance of our method. As also suggested by the reviewers YJq8 and bXsD, we have conducted an experiment which employs non-human stereo videos taken with VR180 cameras, along with the auto-labeling pipeline that generates 3D trajectory sequence [17]. Specifically, we train our model with extra 30k clips obtained from raw VR180 videos downloaded from YouTube, and test the performance in the same benchmark configuration as Table 1. The experiment indicates that our training with the additional data could provide overall improvements in the performance, e.g., relative 10.2% enhancement in EPE3D (0.431 $\\to$ 0.387 in iPhone) as follows. We note that we are in progress to further scale up the training dataset using more videos from YouTube, and will include the results in the final manuscript.\n\n\\begin{array}{c | c  c |  c  c |  c  c }\n\\hline\n& \\rlap{~~~\\text{iPhone}} & \\phantom{a} & \\rlap{~~~~\\text{Sintel}} & \\phantom{a} & \\rlap{\\text{Point Odyssey}} & \\phantom{a}  \\newline \n\\text{Method} & \\phantom{a}  \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a} & \\phantom{a} \\text{EPE}\\_\\text{3D} \\phantom{a} & \\phantom{a} \\text{Abs-rel} \\phantom{a}  \\newline\n\\hline\n\\begin{subarray}{c}\\text{Fast3R}\\end{subarray} & 0.692 & 0.419 & 0.667 & 0.517 & 0.771 & 0.214   \\newline\n\\begin{subarray}{c}\\text{Spann3R}\\end{subarray}& 0.658 & 0.431 & 0.523 & 0.622 & 0.591 & 0.231   \\newline\n\\begin{subarray}{c}\\text{CUT3R}\\end{subarray} &  0.701 & 0.408 & 0.681 & 0.428 & 0.609 & 0.177   \\newline\n\\begin{subarray}{c}\\text{Track3R}\\end{subarray} & 0.431 & 0.344 & 0.312 & 0.374 & 0.399 & 0.165   \\newline\n\\hline\n\\begin{subarray}{c}\\textbf{Track3R} \\newline\n\\textbf{(with VR180)}\\end{subarray} & \\textbf{0.387} & \\textbf{0.323} & \\textbf{0.292} & \\textbf{0.358} & \\textbf{0.371} & \\textbf{0.159}  \\newline\n\\hline\n\\end{array}\n\n---\n\n**Synthetic human motion data may not reflect diverse real-world scenes. Can you provide evidence of performance on varied, non-synthetic data?**\n\nWe note that the iPhone dataset benchmark in our experiments is already of the real-world scene, equipped with the sensory 3D point annotations for testing. While our method trained on the synthetic human motion data already demonstrates reasonable performance on this benchmark, we also would like to emphasize that our experiments with the YouTube VR180 data can further boost the performance in this benchmark covering real-world scenes (see the table above)."}]

Author rebuttal:

Meta-review: Track3R delivers a technically sound and efficient approach to spatiotemporal 3D understanding in dynamic scenes, with clear gains over feed-forward baselines and a clean path to further scaling. For the camera-ready, I encourage the authors to: (i) tighten claims around point-mapping vs optimization, (ii) further clarify notation/figures and adopt “geometry” consistently, and (iii) expand related-work positioning/comparison (VGGT, GFlow, POMATO, TAPIP3D, St4RTrack) and explicitly list limitations and compute trade-offs.  (IV) include more real-world benchmark comparison as Reviewer p1Bw mentioned and also considering the TAPVid-3D benchmark.
Decision: Accept (poster)