Fast Encoder-Based 3D from Casual Videos via Point Track Processing
Authors: Yoni Kasten
Wuyue Lu
Haggai Maron
Venue: NeurIPS 2024 poster (2024)
Keywords: Dynamic Videos
Dynamic 3D Reconstruction
Equivariance
Symmetries
Primary area: machine_vision

Abstract: This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. 
  Aiming to significantly improve the efficiency of previous approaches, we present TracksTo4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation. TracksTo4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TracksTo4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95\%.  We further show that TracksTo4D  generalizes well to unseen videos of unseen semantic categories at inference time.

Fast Encoder-Based 3D from Casual Videos via Point Track Processing
Yoni Kasten' !
Wuyue Lu² Simon Fraser University
Haggai Maron 1.3 Technioo
NVVIIA Research
Abstract
This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. Aiming significantly improve the efficiency of previous approaches, we present TTAAKKSTo4D a learning- based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing tailored for processing 2D point tracks. Our proposed architecture is designed with key principles in mind: (1) takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using low-rank approximation. TRACKSTO4D is trained in an unsupervised way on dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TRACKSTo4D can reconstruct temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the- art methods, while drastically reducing runtime by up to 95%. We further show that TRACKSTo4D generalizes well to unseen videos of unseen semantic categories at inference time. Project page: halps::/tracks-to-dd....u
architecture
1 Introduction
Predicting 3D geometry in dynamic scenes is challenging problem. In this problem setup, we are given access multiple images
scene taken sequentially, e.g., from a monocular video camera, where both the content in the scene and the camera are moving, Our task is to reconstruct the dynamic 3D positions of the points seen in the images and the camera poses. This fundamental problem has gained significant interest from the research community over the years 7 37 26 64),
mainly due to its important applications in many fields such as robot navigation, autonomous driving and 3D reconstruction of general environments [19]. Importantly, in contrast static scenes where the epipolar geometry constraints hold between the corresponding points of different views [17],
determining the depth of moving point from monocular views an ill-posed problem B This causes standard Structure from Motion techniques 42 55 33
to be inadequate in this setup 25
Previous work and limitations. Many existing approaches for above problem make simplifying assumptions that limit their applicability to real-world scenarios. Methods based on orthographic camera models and low-rank assumptions use matrix factorization techniques 7 26] but the ortho- graphic camera assumption might not be realistic and may cause reconstruction errors. Techniques that incorporate depth priors often require lengthy optimization processes in order to make the depth estimates across frames consistent 255 64) Other physics-based approaches make assumptions
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

topet 3D
Equiearant anchieemure
Posfure Cleade
Towar
Oriz Iaratas:
Crace Puuhh (a)
Invriiad Lipere
present TRACKSTO4D, a method for mapping a set of point tracks extracted from casual dynamic videos into their corresponding 3D locations and camera motion. At inference time, our network predicts the dynamic structure and camera motion in a single foed-forward pass. Our network takes as input set of 2D point tracks (left) and uses several multi-head attention layers while alternating between the time dimension and the track dimension (middle). The network predicts cameras, per-frame 3D points, and per-world point movement value (right). The 3D point internal colors illustrate the predicted 3D movement level values, such that points with high/low 3D motion are presented in red/purple colors respectively. These outputs are used to reproject the predicted points into the frames for calculating the reprojection error losses. See details in the text. The reader encouraged watch the supplementary vidco visualizations.
Figure 1:
about rigid bones 160 58 or isometric deformable surfaces 37 and typically involve complex, slow optimization per video. In addition, they may require foreground-bacckrooun segmentation of the moving content, which is not always easily obtained. Alternatively, some methods are specifically tailored to certain object classes like humans [52], restricting their domain to those limited cases. Consequently, these prior methods are either not directly applicable to casual videos, require long optimization time per video.
Our approach. We propose TRACKSTO4D novel approach for fast reconstruction of sparse dynamic 3D point clouds and camera poses from casual videos. Our main idea is to train a neural network multiple videos to learn the mapping from the input image sequence the scene's ' 3D point clouds and camera poses. After training, the trained network can be efficiently applied to new image sequences using single feed-forward pass, avoiding costly optimization.
sequence of
To enhance the method's ss ability to generalize across different types of videos and scenes, we made crucial design choice: our approach processes point track tensors input, rather than operating directly on the image sequence. Specifically, each entry (n, p) in these tensors represents the 2D position of tracked point p in specific video frame 77. Our main insight is that point track tensors may exhibit more common motion patterns across casual video domains compared to image pixels. In other words, argue that processing the raw point track data rather than scene-specific pixels or features may enable learning class and scene- aannstano internal feature representations for improved generalization. Importantly, recent advances in point tracking [I] 20 enable efficiently inferring these point tracks from casual videos using pre-trained models. These two properties make point track matrices attractive input for learning method.
Following this design choice, we design our architecture according to two principles: (1) process point track tensors, which have unique structure, and (2) encode meaningful prior knowledge about the reconstruction problem, as the problem is ill-posed in general. In the following, we address these desired properties.
First, we design network architecture that can effectively and efficiently handle point track inputs. To do that, we propose novel layer design that takes into account the symmetries of the problem: the mapping we aim to learn, from point track matrices 3D point clouds and camera poses, preserves two natural symmetries: (i) the points being tracked can be arbitrarily permuted without affecting the problem; (ii) the frames containing these points exhibit temporal structure, adhering to an approximate time-translation symmetry. Following the Geometric Deep Learning paradigm (8) we build upon
recent theoretical advances in equivariant learning 30 and integrate these two symmetries into our network architecture using dedicated attention and positional encoding mechanisms.
4D since have three Euclidean coordinates with an additional time coordinate

---

Second, key challenge in predicting 3D dynamic motion and camera poses from 2D point tracks that this problem is inherently ill-posed without additional constraints [3. To address this, we integrate low-rank movement assumption into our architecture, following the seminal work of 7 which constrained output point clouds be linear combinations of basis elements. Specifically, given an input point track tensor, our architecture equivariantly predicts small set of input-specific basis elements. The output point clouds at each time frame are then defined as linear combination of these basis elements, with the coefficients also predicted by the network. Notably, the first basis assumed to fully represent the 3D static points in the video, while the remaining basis elements capture the 3D dynamic deviations. This structure effectively restricts the predicted point clouds to have a more specific form, making the problem more constrained.
Our network is trained on dataset of extracted point track matrices 20 from raw videos without any 3D supervision by simply minimizing the reprojection errors, aiming to predict output point clouds that, after undergoing perspective projection, will return the original 21) point tracks. In our experiments, TRACKSTo4D trained on the Common Pets dataset test data with GT camera poses and GT depth information for point tracks, and demonstrate that it produces comparable results to state-of-the-ar methods, while having significantly shorter inference time by up to 95%. In addition, we show the method's 's ability to generalize to out-of-domain videos.
44 We evaluate our method on
Contributions. In summary, our contributions are (1) A novel modeling of the dynamic recon- struction problem via learning on point tracks without 3D supervision; (2) A novel deep learning architecture incorporating
key principles: accounting for the symmetry of the data and en- coding low-rank structure in the predicted point clouds (3) Experiments demonstrating extremely fast inference time compared to baselines, accurate results, and strong generalization across other categories.
2 Method
Problem formulation. Given a video of N frames, let M E NxP3 be a pre-extracted 2D point tracks tensor (Fig. [ left side). This tensor represents the two-dimensional information about a set of P world points that are tracked throughout the video. Each element in the tensor, Mi stores three values: (r. y, o) where æ; y E are respectively the observed horizontal and vertical locations of point j in frame i, and E [0.1} indicates whether point j is observed in frame i or not. Our goal to train a deep neural network to map the input point tracks tensor M into set of per-frame camera poses (R, (,MM) j==1 and per-frame 3D points (X,(M) where (MM E 80(3), t, (M) E R XMMM E Px3 (Fig. right side).
Overview of our approach. Our method receives M e Nxxxx input. This tensor is being processed by a neural architecture composed of multi-head attention layers where the attention is applied in alternating fashion the P and the N dimensions in each layer. These layers are defined in Sec. 2.T After a composition of several such layers, the network uses the resulting features RN Pxx to predict camera poses 8O x IR3 and point clouds in N xx33, These N point clouds are parameterized as linear combination of K input specific point cloud bases B1 (M), ........ Bx (M) E xx3 This discussed in detail in Sec. 2.2 Our network is trained in an unsupervised way dataset of videos by minimizing the reprojection error and other regularization losses (Sec. 2.3) that are used to update the model parameters. Our pipeline illustrated in Fig. [
2.1 Equivariant layers for point track tensors
Following the geometric deep learning paradigm, our goal is to design a neural architecture that respects the underlying symmetries and structure of the data.
Symmetry analysis. Our input is tensor M E NxPx3 representing sequence of N frames each with P point coordinates. This structure gives rise to two key symmetries: First, the order of
3

---

the P points within each frame does not matter in other words, permuting this axis results in an equivalent problem [0 39
Formally, this axis has a permutation symmetry Sp where Sp is the symmetric group on P elements. Second, along the temporal N axis, we have an approximate translation symmetry arising from the ordered video sequence. This means that shifting the time frames is required to result in the same shift in our output. We model this with cyclic group CN of order N. Both symmetries are illustrated in Fig. 2 We note that while the cyclic group assumption may not be entirely accurate, we still find useful as helps us to derive appropriate parametric layers for our data, similar to how the convolutional layer is derived for data with translational symmetries such as images. Taken together, the full symmetry group of the input space is the direct product G Cy x Sp combining these time and point permutation symmetries, acting on R xxxx by ((t, σ) M M.pp = M o (p)) for (t, o)) E O Next, we will design architecture equivariant to G. to ensure that the model takes into account the symmetries above.
Points
Linear equivariant layers. Point track tensors can be viewed as a collection of N individual point tracks,
Permutation syreutry
each of which exhibits translational symmetry. The scenario where an object comprises a set of elements with their own symmetry group, such as set of images or graphs, was previously explored in [30]. . In that work, the authors characterized the general linear equivariant layer structure in such cases, termed the Deep Sets for Symmetric Elements (DSS) layer. Building on the DSS approach, our basic linear equivariant layer for the point track tensors M would take the form:
Frames
Time mnntattoo ontimoy 22
Figure 2: The symmetry structure of our problem. Frames (vertical) have time trans- lation symmetry while points (horizontal) have set permutation symmetry.
F(M).3 = LM
Σ L2(M 8²=1
where L, are linear translation equivariant function (i.e. convolutions), M., E Rxx the columns of M representing all the inputs for a specific tracked point, F(M).., E R Nxd' is the output column and d, d' are the input and output feature channels respectively. To construct neural network, these layers can be interfeaved with pointwise nonlinearities, similar to basic convolutional neural networks.
Implementation via transformers and positional encoding. While the linear layer design is reasonable, it may not be the optimal choice. To enhance the model, we design a new layer whose structure follows Equation (L) but incorporates nonlinear layers in the form of transformers (51). We note that the idea of using linear layers as inspiration for non-linear layers aligns with common practices in geometric deep learning, as described in several previous works 30 14]
Specifically, our layer F is formulated similarly to Equation ① but instead of convolutions (I,) and summations, it utilizes two self-attention mechanisms and suitable temporal positional encoding across the N dimension. Formally, our basic layer F N + Nxxd is computed via four steps, which are described below:
exp(d.j kez)
q. = W. Kij W. Maj, Vij WV M.y.M Mi
(2)
F=1 i= exp(dij k1j)
exp( kiy))
dg -WOM kij W Myj. Vig WV F(M)
Σ
Vej (3)
Di=1 exp(qi) ka)
Here, M.j E Rd are the features associated with the j-th point in the i-th frame. The attention mechanism defined in the first equation above is augmented with standard temporal positional
Our work builds upon the pioneering works PointNet (39 and Deep Sets (63 that studied permutations symmetries general and for point clouds in particular. As we shall see next, in our case, the symmetry group a bit more complicated and fits nicely the setup of 20).
This is different from the symmetry group studied in [2). where the temporal structure exploited.

---

encoding in the first layer and replaces the translation equivariant function L, applied to the columns of MM (Eq.IT). The attention in the second equation (3) implements the set aggregation (summation) (also in ()) applied to the rows of M. As commonly done, we use transformers with 16 attention heads 51
2.2
Constraining 3D motion and camera poses via low-rank assumption
Given our 2D tracks, we aim characterize the motion of the points by decomposing them into the global camera motion and the 3D motion of objects in the scene. The 2D motion of static scene points provides useful constraints for estimating the camera motion. However, as previously mentioned, predicting camera and dynamic 3D motion solely from 2D motion is an ill-posed problem without additional constraints + We tackle this challenge by adding two mechanisms to our architecture: (1) low-rank movement assumption; and (2) specific modeling of the static scene for camera estimation.
Low-rank movement assumption. First, motivated by classical orthographic Non-Rigid Structure from Motion 77]. constrain the output points to be formulated by linear combination of input- specific basis elements. Specifically, given the input 2D point tracks, M E Nxxxxx our network predicts K point clouds: Bi (MM, (MM).... Bx (M) E IR Px3 and N(K - 1) linear coefficients, k= {CNx(M) [M)] such that
K
X.(M - (iM + Σ Ca (M)
where (MM E RR3 is the 3D point cloud at frame i. The point clouds and coefficients are computed by taking the output of the last equivariant layer as defined in the previous section and applying invariant aggregations on the respective dimension resulting in equivariant and invariant outputs. See more details in the appendix. We note that we deliberately chose the coefficient of (M) to be the constant 1, the reason is explained in the next paragraph.
Specific modeling of the static scene for camera estimation. Frequently, casual video data of dynamic scenes contains many static regions, which can be used to determine camera poses [66]. We leverage this observation by treating the first basis element BN (M) E I Px3 as a static approximation for all scene points and encourage Bi (M) as well as the output camera poses explain the 2D observations according to this approximation using a "static" reprojection loss (LStatic defined the next section). We note, however, that static point cloud is not likely to produce low reprojection errors for the non-static components, thus the reprojection error necessitates robustness to substantial errors from the non-static elements. To address this, network predicts (cquivariantly) P motion level values ?1 (. (MM ) E IR,, one for each point in our dynamic point cloud, which we use to weight the reprojection errors from B, (M). The main idea is to give less weight to non-static points so that the static projection loss can disregard them. Specifically, inspired by [64], each % (M) defines Cauchy distribution that models the reprojection errors for its associated world point, such that a world point with higher y is expected to produce a wider error distribution. Empirically, as noted by [64], the Cauchy distribution tends be more robust for modeling reprojection error uncertainties compared to Gaussian noise modeling [22]. Then, Lstatie minimizes the negative
log-likelihood under this assumption. See details in Sec. 2.3
Comparison to supervised learning setups. A central aspect of our method is addressing the iol-posed nature of this problem through an architecture that heavily regularizes the output dynamic point clouds, enabling training without supervision. Alternatively, one could use video data with ground truth depth for supervised training. potentially allowing the output to have more general form. Unfortunately, such supervision is both challenging acquire and may lead to poor generalization on novel motion patterns.
2.3 Training and losses
Model outputs. Given the input 2D point tracks M E NxPx3 our network produces outputs as a function of M: linear bases and coefficients,
5

---

(.M))...,BBI..... E I Rx3 (c 1 (o point cloud X1 (M),, (.MMM E R gxx3 (MM)............... (P(M) R2, movement level values, and (R1 (M), (M)),, .... (Ry
E R which define a dynamic
SO(3) I2 camera poses.
We use these network outputs to define a self-supervised loss function with respect to the current network weights and M which is defined by:
ARepmjost Repmoeet + Attari CSmiic + ANeeaaivve + ASparse
(5)
Reprojection loss. The reprojection loss encourages the consistency between the output 3D point clouds and camera poses, to the input 21) observations:
N
P
Reproject
ΣE
MiR((X)
R,,
(6)
Σ M =1
where R(Xej, R2, t,, My is the reprojection enror when projecting the point X with the camera pose (R, t;) with respect the measured point M,
(R) (X.; (X., (n) (X.j 1;))3
- t1))1,2
R(Xij, Ri, t, Ma
Mg
Static loss. As discussed in Sec. 2.2 to better constrain the camera poses, the first predicted basis element (M) E gxx3 defines static (fixed in time) point cloud approximation. Our network also predicts movement coefficient % (M) for each world point that defines zero-mean Cauchy distribution. Given "i and the reprojection enror Tij R(B1,, R2 t;, M,," - of the the jth point of B that projected by the jth camera, the negative log-likelitne eegllikelihood of rı distributed according to the 7y-zero-mean Cauchy distribution is proportional to:
r
C(rig, (5)r)>7): = log
Yj
+
(8)
%
Note, that this loss reduces the contribution of the reprojection errors for points with high y, but also encourages y to be small, i.e. encouraging the static point cloud to represent the dynamic scene when possible. Our static loss is the mean negative log-likelihood over all observed points in all frames:
P
Static
MrfC(R(BI,, t.,
t, M, ).7)))
(9)
Mm
M
j=1
==
Regularization losses.
32
we regularize the observed points to be in front of the camera by:
Negaative
M2 Min (Xij t;))a, 0)
(10)
We further find it beneficial to regularize the deviation from the static approximation B1 to be sparse for static points, i.c. points with low 7 values:
P ΣΣ
Csparse P(K-1
(|Bkjal+ |Byz2l (Byss))
(11)
k= 3%
where 7 is detached from the gradient calculation for this loss.
3 Experiments
In this section, we conduct experiments to verify our proposed network's performance on real-world casual videos. We began by training the network on specific domains and then evaluated its accuracy and running time on unseen videos from both, training and unseen domains.
We denote the 5th 3D point of By Ie Px3 by Buj E R. The 3 elements of this point are denoted by
Bej Bagz, Baja E 18 (see [I

---

Table 1: Pet evaluation. Top: Baseline method results for structure or camera estimation (or both). Bottom: Our results with several configurations. (C),(D) (CD) respectively indicate the object categories in the training set: cats, dogs, or both. and respectively indicate post- post-processee processing Bundle Adjustment aue-tuning.
Abs Rel 1
< 1.25 1
<
1.25? f
<
1.25
ATE 4 (mm)
RPE (2a0))
RPEgac (dog)
Time (min)
Dyn.
All
Dyn.
All
Dyn.
All
Dyn.
All
AM 43
5.08
3.60
0.20
0.16
SMM 66
12.79
6.95
11.00
3.6E+07
AM
6
0.09
0.06
0.93
0.97
0.99
0.99
1.00
1.00
6.90
3.95
0.22
130
MiDuS S
0.16
6.20+01
0.78
0.71
0.97
0.58
1.00
0.93
0.15
Ours (C)
0.11
0.08
0.88
0.92
0.99
0.98
1.00
1,00
3.96
3.79
0.25
0.15
Durk (C)RRA
0.11
0.8
0.88
0.92
0.99
0.98
1.00
1,00
4.22
2.56
0.17
0.15
(CHFT
Ours (T)
0:12
0.08
0.85
0.91
0.99
0.99
1.00
1.00
8.03
0.23
0.15
Durs (D)+RA
0.12
078
0.85
0.91
0.99
0.90
1.00
1.00
4.19
2.83
0.17
0.15
Ours (D>+FT
0.09
0.06
0.88
0.96
1.00
0.99
1.00
1,00
3.98
2.74
0.16
4.86
Durs (CD))RA
0:12
0.91
0.98
0.98
1.00
1.00
4.21
Ours (CDHFF
0.09
0.06
0.90
0.96
1.00
0.99
1.00
1.00
3.98
2.74
0.16
-186
Table 2: Out-of-training-domaii evaluation. Evaluation metrics on monocular videos from has the same structure as Tab. M
The table
1.25
<
1.252
<
1.25 All
ATE (mmm
RPS (mmm
RPE (deg)
Dyn
Dyn.
Dyn.
All
Dyn.
(min)
D-SLAM
0.07
0.18
PunsleSFFM 66
26.66
23.83
0.20
2.13
RCVD 25
0.19
2.6F-05
0.69
0.75
0.95
0.95
0.96
0.98
160
130
3.43
7,00
CanalSAM
x
0.95
0.99
1.00
1,00
1.00
0.06
22.00
2,8E-04
2.75:05
0.75
(C)
0.99
0.04
Ours (C+BA
0.08
0.06
0.89
0.95
0.99
0.99
0.99
1.00
8.67
12.36
0.08
0.04
Ouss (C)+FF
0.07
0.03
0.94
0.95
0.99
1.00
1.00
1.00
7.58
11.64
0.08
0.59
(D)
0.08
0.07
0.92
0.99
0.98
0.99
1.00
51.64
0.61
0,34
(D)
0.08
0.92
0.93
0:99
1.00
12.06
0:04
Ours (DD+1
0.05
0.03
0.97
0.99
0.99
1.00
11999
1.00
8.15
11.88
0.09
0.59
Ours (CD)
0.10
0.08
0.95
0.94
0.99
0.99
1.00
1.00
36.17
53.94
0.57
0.04
(CD)-BA
0.99
1.00
1.00
0.08
0.04
(UD))FTT
1.00
0.09
Training procedure. We trained our network on the cat and dog partitions from the COP3D dataset 44], which contains a diverse set of casual real-world videos of pets. Our networks were trained from scratch three times to test our generalization capability between semantic categories: once on the cat partition, once the dog partition, and once both partitions combined. Training technical details are provided in the appendix. We use K 12 bases in all our experiments (Sec. 2.2
Evaluation data. To assess our framework's performance on pet videos, we curated new datase consisting of 21 casual videos of dogs and cats, each video containing 50 frames. These videos were captured using an RGBD (RGB-Depth) sensor. The depth maps were used as ground truth for evaluating the reconstructed structure. We extracted the cameras by running COLMAP on the images while masking out the pet areas with dilatated masks provided by [71). The cameras were scaled to millimeter units using the provided GT depth. Note that our network did not see this test data during training and it was not used to tune hyperparameters.
Additionally, to evaluate our method on out-of domaan evaluation data, we used the Nvidia Dynamic Scenes Dataset [62]. Specifically, while our network was trained on pet videos, this dataset contains other dynamic object types, e.g. human, balloon, truck, and umbrella, with a different camera motion type and a variety of motion profiles. The dataset contains 8 dynamic scenes which are captured by 12 synchronized cameras, enabling accurate depth estimation which is treated as GT for evaluating monocular depth estimation. The ground truth camera poses were calculated by [42 with the synchronized multiview camera rig and the ground truth dynamic masks. Similarly to we
'While the COP3D dataset provides cameras that were extracted by COLMAP 42
we note that this
evaluation data insufficient in our case. This is because the dynamic structure was captured as well in part of their reconstruction which indicates that its reconstruction might not be accurate. Furthermore, the coordinates system units of these reconstructions are unknown. Finally, this dataset does not have any depth map information for evaluating the dynamic structure.
7

---

Table 3: Ablation study. The contribution of different parts from method. See details in the text.
Abs Rei
1.25
1.25
1.25
(pix))
ATE: (mm)
RPE Trais 4
RPE Rot
Dyn
All
Dyn
All
Dyn
Dyn.
Dyn
All
(mm)
(deg)
Set Sets
0.15
0.77
0.87
0.99
9.86
5.33
16.87
0.39
No Siuai
0.77
0.36
0.25
0146
01.48
01.70
01.68
0.82
1.00
0.56
96.20
29.86
0.99
No
0.22
0.16
0.56
0.73
0.95
0.91
(3599
0.97
4.54
2.41
13.91
4.36
0.29
K-30
0.09
0.50
2.78
0.23
K-2
0.08
0.91
0:98
1.00
8.58
3.56
DSS
1.00
0.38
0.19
0.35
0.34
0.00
0.47
0.74
63.75
70.00
32.50
22.6.
1.64
No Sppare
0.17
0.13
0.79
0.50
0.95
0.34
1.00
0.99
4.57
2.73
11.79
7.99
0.55
Full
0.08
1.97
simulated 8 monocular dynamic video sequences using the camera rig, each with 24 frames, and used them for evaluation.
Evaluation results. Qualitative visualizations are presented in Fig. 6 We also show a visualization of the movement level values, 7 in Fig- 4 For comparisons, we chose state-of-the-art methods that as our method, can be applied to raw casual videos that were captured by standard pinhole camera models and do not need any static or dynamic segmentation. We evaluated both, the camera poses and the structure accuracies. Comparison results for the pet-test-set and out-of-domain dataset are presented in Tables 2 respectively The camera poses are evaluated compared to the GT, using the Absolute Translation Error(ATE), the Relative Translation Error(RTE), and the Relative Rotation EREE)(RRE) metrics after coordinates system alignment. We compare three training configurations of our method of training only on cats, only on dogs, and on both. As can be seen in the tables, the results are similar in all cases. Our output camera poses as inferred by the network are already accurate and outperform some of the prior methods. We further show the results of our method after a single and short round of Bundle Adjustment, which makes our method better than all baselines on the pet sequences, and comparable on the nut-of-domail cases.
Importantly, Tables [ and 2 also compare the method's runtime. It can be seen that our method, even with bundle adjustment, is the fastest camera prediction method. Note that our method's runtimes include the point tracking time that is performed by [20] as a pre-process, In the appendix (Tab. [ we present tracking time versus inference time, showing that most of the time is spent on tracking, while our inference very fast. Tables I and alsoo show structure evaluation with the depth evaluation metrics [12] on the sampled point tracks. They demonstrate that our inferred structure is almost comparable the state-of-the-art (64] while taking significantly shorter running times (a few seconds for our method versus more than two hours for [64] on pet videos). Short (0.6-5 minutes), per-sequence fine-tuning makes our method's s accuracy comparable to (44 In the appendix, we present per-scene output accuracy, demonstrating our ability to generalize across different speed profiles. In terms of running time, our method is a bit slower than MiDaS [5] which only provides depth maps without cameras, but achieves much better result:] We note that in contrast to the other prior.
methods that predict the dynamic depth, ours does not use any sorphh-froo-iigll-nnne
Ablation study To evaluate the contribution of our different method parts we run an ablation study which is presented in Tab. 3 In this study, the training was always done on the cat partition from COP3D and evaluated on our test data which contains dogs and cats. First, we performed an ablation study on our transformer architecture by taking the architecture suggested by [32 ("Set of Sets") or the DSS architecture that uses only linear layers [30 ("DSS"). As the table shows our architecture ("Full") achieved significantly better results. To test the losses our framework, we also evaluated the following: (1) ignoring the y outputs and using regular reprojection error on B₁ for all points ("No y"): (2) removing our sparsity loss ("No Lsparse "); and (3) removing the static loss ("No Csaatie)) "). In all cases, the error increased whereas in the later one, the results became much worse. We further ablate the choice of K 12 as the number of linear bases, by trying 2 extreme numbers of K = 30, K 2 (we saw no significant differences when we used nearby choices such K = 11). As can be seen in the table, when K = 30 the output is not regularized enough and produces a higher
'The reader is encouraged watch the supplementary videos for better 4D perception. 7 "Our comparisons were made using MiDaS 3.1, specifically the beit large_rre__1 version (Birkl et al. 2023). This is an improved version of MiDaS that utilizes the DPT architecture, representing a more current state of the method.
8

---

Figure 3: Qualitative Results. Top. Frames from 2 different test video sequences with point tracks marked with corresponding colors. Bottom. A 3D visualization of our method's outputs, from two time stamps. The camera trajectory present as gray frustums, whereas the current camera marked in red. The reconstructed 3D scene points are presented corresponding colors to input tracks on the top. The scene is observed from the same viewpoint, enabling the visualization of the dynamic reconstructed structure.
Figure 4: y Visualization. We show visualization of the y outputs of network that described in Sec. 2.2 In each video sequence, we show the input tracks, where each color visualizes movement level value, 7. Purple marks static points with low 7 whereas red marks dynamic points with high y. Note, that our network did not get any direct supervision for these values, but only the raw point tracks predictions from (20). The visualizations for cats were produced by the model that was only trained on dogs and vice versa. We note that model generalizes well out-of-domain (non-pet) cases well.
depth error for the dynamic part. For = 2 the depth is regularized but the reprojection error ("Rep. (pix.)") gets higher due over-regularization. Overall, this study justifies our design choices ("Full").
Related Work
SIM pipelines seek to recover static 3D structure and camera poses from unordered images. 49 45 42
Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SIM)
[5
42 effective but require repeated applications of Bundle Adjustment 32 9 presented method for learning prior from a dataset multiview image sets, to accelerate SfM pipelines by using equivariant deep networks. Monocular Simultaneous Localization
Learning- Dearning-free pipelines
(BA)
[50]
extract camera poses from video sequences while defining scene map with keyframes. These methods assume static scenes, fail to produce the cameras in scenes with large portions of dynamic motion, and cannot reproduce dynamic parts of the scene. PROID-SLAM 488 used synthetic data with ground truth 3D supervision for learning to predict camera poses via deep-based BA on keyframes while excluding dynamic objects. ParticleSIM [66] filters out 2D dynamic content for reproducing the cameras in dynamic scenes, using its pre-trained motion prediction network. Both, 48 66
and Mapping (SLAM) methods 33 34
13
61
6
54|
67
69 46)
do not infer the dynamic 3D structure.
Orthographic Non-Rigid SIM (NRSIM) Bregler al. introduced factorization method for computing non-rigid structure and rotation matrices from point track matrix, by assuming low

---

dimensional basis model. While follow-up papers improved accuracies with different regularizations (26] 10 36 [8] or neural representations [35 24 the orthographic camera model assumption
is in general not valid for casual videos. Furthermore, these methods often assume background subtraction as a preprocessing. Even though follow-up work proposed factorization solutions for pinhole cameras [16], its sensitivity to noise
19), makes it impractical for casual videos.
Test-time optimization for dynamic scenes monocular depth estimation from a pre-trained model 44
Recent methods 22
25
64
65] finetuned the
40] using optical flow constraints [47], for obtaining consistent dense depth maps for a monocular video. [64] further optimized motion maps
for handling scenes with large dynamic motion. [56 15 use depth from single image estimations, to
improve novel view synthesis in dynamic scenes. (28) further optimizes for the unknown camera
poses together with the dynamic radiance field optimization. [3 38 model a single deformable
surface from a monocular video by applying isometric constraints. LASR (58], ViSER [59] and
BANMo (60) optimize for a dynamic surface by assuming rigid bones and linear blend skinning weights. However, all the above-mentionel methods require per-scene optimization, resulting in slow inference. Recently, [4] presented the Common Pets in 3D (COP3D) dataset that contains casual, in-the-wild videos of pets, and used it to learn priors for novel view synthesis in dynamic scenes.
Point tracking There has been a recent advance in 2D point tracking by learning 20 or optimization [53] techniques. Concurrently, [57] presented method for jointly performing 2D tracking and 3D lifting, by learning to track with depth supervision while applying an as-rigid-as- possible loss. However, their method cannot predict camera poses or identify static parts directly.
5 Conclusions and limitations
We presented TRACKSTO4D, novel deep-learning framework that directly maps 2D motion tracks from casual vidcos into their corresponding dynamic structure and camera motion. Our approach features deep learning architecture that considers the symmetries in the problem with designed intrinsic constraints to handle the ill ill-posed nature of this problem. Notably, our network was trained using only raw supervision of 2D point tracks extracted by an off-the-shelf method (20) without any 3D supervision. Yet, it implicitly learned to predict camera poses and 3D structures while identifying the dynamic parts. During inference, our method demonstrates significantly faster processing times compared to previous methods while achieving comparable accuracy. Furthermore, our network exhibits strong generalization capabilities, performing well even on semantic categories that were not present in the training data.
Limitations and future work. While our experiments demonstrated that our network efficient, accurate, and capable of generalizing unseen video categories, there are several limitations and future work directions that we would like to address. First, our method is limited in handling videos with rapid motion, it depends on the accuracy of the tracking method used, specifically CoTrracker (20). which has limitations in tracking fast movements. We observed that (20) often fails in automotive scenes due to high motion blur, particularly on the road, limiting our ability to perform large-scale evaluations on automotive datasets given the current tracking method performance. We note that any future improvements with point tracking in terms of accuracy and inference time will directly improve our method as well. Second, our method assumes enough motion parallax to constrain the depth values and fails generate accurate camera poses without it. An interesting direction for future work would be to leverage depth models and 3D tracking techniques, which could potentially improve accuracy, especially in cases with minimal motion parallax. Yet. our current attempts to incorporate MiDaS-inferred depth as an additional input and apply depth loss relative to MiDaS have not yielded any performance improvements, indicating that this approach needs further exploration. Third, while we found K = 12 basis elements to be effective for our evaluation set, balancing complexity reduction and motion representation, we acknowledge this fixed number may not capture all possible scene dynamics. Future work could explore automatically inferring the optimal number of bases per scene. Lastly, our network can handle up to about 1000 point tracks in 50 frames in one inference step when running on single GPU. A possible extension to handle denser point clouds and longer videos could involve querying point tracks iteratively while maintaining a shared state, but this approach remains to be explored.
10

---

Acknowledgments HM is the Robert J. Shillman Fellow, and is supported by the Israel Science
Foundation through personal grant (ISF 264/23) and an equipment grant (ISF 532/23).
References
[1] Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R.: Building rome in day. Communications of the ACM 54(10), 105-112 (2011)
[2] Agarwal, S., Mierle, K., Team, T.C.S.: Ceres Solver (10 2023), https://githhb.ccm com/ ceres- soeeeeeeeeeeeeeeeeeeee
[3] Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Nonrigid structure from motion in trajectory space. Advances in neural information processing systems 21 (2008)
[4] Bevilacqua, B., Frasca, F., Lim, D., Srinivasan, B., Cai, C., Balamurugan, G., Bronstein, M.M., Maron, H.: Equivariant subgraph aggregation networks. arXiv preprint (2021)
[5] Birkl, R., Wofk, D., Müller, M.: Midas v3.1 model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307. 14460 (2023)
[6] Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S., Davison, A.J.: Codeslam learning a compact, optimisable representation for dense visual slam. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2560-2568 (2018)
[7] Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3d shape from image streams. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662). vol. pp. 690-696. IEEE (2000)
[8] Bronstein, M.M., Bruna, J., Cohen, T., Veličković, P.: Geometric deep learning: Grids, groups, graphs geodesics, and gauges. arXiv preprint arXiv:2104. (2021)
[9] Brynte, L.,, Iglesias, J.P., Olsson, C., Kahl, F:: Learning structure-from-mootion with graph attention networks. arXiv preprint arXiv:2308. 15984 (2023)
[10] Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structurefrrom-motiin factorization. International Journal of Computer Vision 107, 101-122 (2014)
[11] Doersch, C., Yang, Y., Vecerik, M., Gokay, D., Gupta, A., Aytar, Y., Carreir,, J., Zisserman, A.: Tapir: Tracking any point with per-frame initialization and temporal refinement. arXiv preprint arXiv:2306.0866 (2023)
[12] Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from single image using multi-scale deep network. Advances in neural information processing systems 27 (2014)
[13] Engel, J.. Schöps, T., Cremers, D.: Lsd-slam: Large-scale direct monocular slam. In: European conference on computer vision. pp. 934-849. Springer (2014)
[14] Frasca, F., Bevilacqua, B., Bronstein, M., Maron, H.: Understanding and extending subgraph gnns by rethinking their symmetries. Advances in Neural Information Processing Systems 35, 31376-31390 (2022)
[15] Gao, C., Saraf, A., Kopf, J., Huang, J.B.: Dynamic view synthesis from dynamic monocular video. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 5712-5721 (2021)
[16] Hartley, R., Vidal, R.: Perspective nonrigid shape and motion recovery. In: Computer Vision- ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10. pp. 276-289. Springer (2008)
[17] Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)
11

---

[18] Iglesias. J.P., Olsson, C., Valtonen Ornhag, M.: Accurate optimization of weighted nuclear norm for non-rigid structure from motion. In: Computer Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16. pp. 21-37. Springer (2020)
2020: 16th European
[19] Jensen, S.H.N., Doest, M.E.B., Aanæs, H., Del Bue, A.: A benchmark and evaluation of non-rigid structure from motion. International Journal of Computer Vision 129( 882-899 (2021)
[20] Karaev, N., Rocco, I., Graham, B., Neverova, N.. Vedaldi, A., Rupprecht, C.: Cotracker: is better to track together. arXiv preprint arXiv:2307.07635 (2023)
[21] Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Repurposing diffusion- based image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[22] Kendall, A., Gal, Y.: What uncertainties do need in bayesian deep learning for computer vision? Advances in neural information processing systems 30 (2017)
[23] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
[24] Kong, C., Lucey, S.: Deep non-rigid structure from motion. In: Proceedings the IEEE/CVF International Conference on Computer Vision. pp. 1558-1567 (2019)
[25] Kopf. J., Rong, X., Huang, J.B.: Robust consistent video depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1611-162 (2021)
[26] Kumar, S., Van Gool, L.: Organic priors in non-rigid structure from motion. In: European Conference on Computer Vision. pp. 71 88. Springer (2022)
[27] Li, Z., Niklaus, S., Snavely, N., Wang. O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 6498-6508 (2021)
[28] Liu, Y.L., Gao, C., Meuleman, A., Tseng, H.Y., Saraf, A., Kim, C., Chuang, Y.Y., Kopf, J., Huang, J.B.; Robust dynamic radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13 (2023)
[29] Luo, X., Huang, J.B., Szeliski, R., Matzen, K., Kopf, J.: Consistent video depth estimation. ACM Transactions on Graphics (ToG) 39(4), 71-1 (2020)
[30] Maron, H., Litany, O., Chechik, G., Fetaya, E.: On learning sets of symmetric elements. In: International conference machine learning. pp. 6734 6744. PMLR (2020)
[31] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM 65(1), 99-106 (2021)
[32] Moran, D., Koslowsky, H., Kasten, Y., Maron, H., Galun, M., Basri, R.: Deep permutation equivariant structure from motion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5976-5986 (2021)
[33] Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam: versatile and accurate monocular slam system. IEBE transactions on robotics 31(5). 1147-1163 (2015)
[34] Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: Dense tracking and mapping in real-time. 2011 international conference on computer vision. pp. 2320-2327. IEBE (2011)
[35] Novotny, D., Ravi, N., Graham, B., Neverova, N., Vedaldi, A.: C3dpo: Canonical 3d pose networks for non-rigid structure from motion. In: Proceedings of the IEEF/CVF International Conference Computer Vision. pp. 7688-7697 (2019)
12

---

[36] Oskarsson, M., Batstone, K., Astrom, K.: Trust no one: Low rank matrix factorization using hierarchical ransac. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5820-5829 (2016)
[37] Parashar, S., Pizarro, D., Bartoli, A.: Isometric non-rigid shape-from-motion with riemannian geometry solved in linear time. IEEE transactions on pattern analysis and machine intelligence 40(10), 442-2454 (2017)
[38] Parashar, S., Pizarro, D., Bartoli, A.: Robust isometric non-rigid structure-frommmotion. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(10), 6409-669990 6423 (2021)
[39] Qi, C.R., Su, H., Mo, K. Guibas, LJJ.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652-660 (2017)
[40] Ranftl, R., Bochkovskiy, A., Koltun, V: Vision transformers for dense prediction. ICCV (2021)
[41] Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V:: Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(3) (2022)
[42] Schonberger, J.L., Frahm, J.M.: Structure- motion revisited. In: Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4104-4113 (2016)
[43] Sidhu, V.. Tretschk, E., Golyanik, V., Agudo, A., Theobalt, C.: Neural dense non-rigid structure from motion with latent space constraints. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16. pp. 204-222. Springer (2020)
[44] Sinha, S., Shapovalov, R., Reizenstein, J., Rocco, I., Neverova, N., Vedaldi, A., Novotny, D.: Common pets in 3d: Dynamic new-view synthesis of real-life deformable categories. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.
4881-4891 (2023)
[45] Sturm, P., Triggs, B.: A factorization based algorithm for multi-image projective structure and motion. In: Computer Vision-EECv' 4th European Conference on Computer Vision Cambridge, UK, April 15-18, 1996 Proceedings Volume 4. pp. 709-720. Springer (1996)
[46] Teed, Z., Deng, J.: Deepv2d: Video to depth with differentiable structure from motion. arXiv preprint wr.Xiv11812.00
(2018)
[47] Teed, Z.,, Deng, Raft: Recurrent all-pairs field transforms for optical flow, In: Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part 16. pp. 402-411 419. Springer (2020)
[48] Teed, Z., Deng, J.: Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems 34, 16558-1653 (2021)
[49] Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factoriza- tion method. International journal of computer vision 9, 137-154 (1992)
[50] Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment modern synthesis. In: Vision Algorithms: Theory and Practice: International Workshop on Vision Algorithms Corfu, Greece, September 21-22, 1999 Proceedings. pp. 298-372. Springer (2000)
[51] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.. Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
[52] Wan, Z., Li, 7., Tian, M., Liu, J., Yi, S., Li, H.: Encoder-decoder with multi-level attention for 3d human shape and pose estimation. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 13033-13042 (2021)
13

---

[53] Wang. Q. Chang. Y.Y., Cai, R., Li, Z., Hariharan, B., Holynski, A., Snavely, N.: Tracking
everything everywhere all once. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19795-19806 (2023)
[54] Wang, W., Hu, Y. Scherer, S.: Tartanvo: A generalizable learning-based vo. In: Conference on Robot Learning. pp. 1761-1772 PMLR (2021)
[55] Wu, C.: Towards linear-time incremental structure from motion. In: 2013 International Confer- ence on 3D Vision-3DV 2013, pp. 127-134. IEEE (2013)
[56] Xian, W., Huang, J.B., Kopf, J., Kim, C.: Space-time neural irradiance fields for free-viewpoint video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 9421 9431 (2021)
[57] Xiao, Y., Wang, Q. Zhang, S., Xue, N., Peng. S., Shen, Y., Zhou, X.: Spatialtracker: Tracking any 2d pixels in 3d space. arXiv preprint arXiv:2404.04311 (2024)
[58] Yang. G., Sun, D., Jampani, V., Vlasie. D., Cole, F., Chang, H., Ramanan, D., Freeman, W.T., Liu, C.: Lasr: Learning articulated shape reconstruction from a monocular video. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 15980-15989 (2021)
[59] Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Liu, C., Ramanan, D.: Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems 34, 19326-19338 (2021)
[60] Yang, G., Vo. M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Building animatable 3d neural models from many casual videos. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2863-2873 (2022)
[61] Yang, N., Stumberg, L.v., Wang. R., Cremers, D.: D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1281-1292 (2020)
[62] Yoon, J.S., Kim, K., Gallo, O., Park, H.S., Kautz, J.: Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5336-5344 (2020)
[63] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. Advances in neural information processing systems 30 (2017)
[64] Zhang, Z., Cole, F., Li, Z., Rubinstein, M., Snavely, N., Freeman, W.T.: Structure and motion from casual videos. In: European Conference on Computer Vision. pp. 20-37. Springer (2022)
[65] Zhang, Z., Cole, F., Tucker, R., Freeman, W.T., Dekel, T.: Consistent depth of moving objects in video. ACM Transactions Graphics (TOG) 40(4), 1-12 (2021)
[66] Zhao, W., Liu, S., Guo, H., Wang, W., Liu, Y.J.: Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In: European Conference on Computer Vision. pp. 923-542, Springer (2022)
[67] Zhao, W., Liu, S., Shu, Y., Liu, Y.J.: Towards better generalization: Joint depth-pose learning without posenet. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9151-9161 (2020)
[68] Zheng, Y., Harley, A. W.W., Shen, B., Wetzstein, G., Guibas, L.J.: Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In: ICCV (2023)
[69] Zhou, H. Ummenhofer, B., Brox, T.: Deeptam: Deep tracking and mapping. In: Proceedings of the European conference on computer vision (ECCV). pp. 822-838 (2018)
[70] Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5745-5733 (2019)
14

---

[71] Zou, X., Yang, J., Zhang, H., Li, Li, L.,, Wang, J., Wang, L., Gao, J., Lee, YJ..: Segment everything everywhere all at once, Advances in Neural Information Processing Systems 36 (2024)
15

---

A Supplementary results
A.1 Video results
We provide supplementary video outputs of several cases from our test set. Each video presents the input video frames with a set of pre-extracted point tracks that are used as input to our network and presented in corresponding colors (left side), and the output cameras and dynamic 3D structure (right side). The output camera trajectory presented as gray frustums, whereas the current camera is marked in red. The reconstructed 3D scene points are presented in corresponding colors to the input tracks. Note that the outputs presented in the videos were obtained at inference time, with a single ferd-forward prediction, without any optimization or fine-tuning, unseen test cases.
A.2 Extended quantitative evaluation
Per sequence quantitative numbers Per-sequence and mean quantitative comparisons for our 21 pet test videos are presented in Tab. Tab.9lam and Tab. T0 Tables with similar structure for the out-of-domain dataset are presented in Tab. [T and Tab. 12 Tables [ and T2 also demonstrate our robustness across various levels of dynamic motion. However, some differences in tolerance to motion patterss are evident. For instance, our depth accuracy on 'Skating' is slightly higher than on 'Truck'.
Handling tracking errors We note that the input point tracks extracted by CoTracker [18] are imperfect and contain noise and outliers. Nevertheless, our model shows significant robustness to these errors. Our model handles imperfect input through two features:
Static scene modeling: Ensures that only 2D motion that can be represented by camera motion and truly static points modeled statically. This makes our camera estimation robust to errors while pushing the non-modelled errors to the dynamic part.
Dynamic scene modeling: Using limited basis elements for dynamic parts inherently resists outliers and extreme anomalies.
To quantify this robustness, we conducted tests by adding Gaussian and uniform noise to CoTracker points. The results (see Tab. 4 confirm that our method tolerates significant noise levels, further validating its effectiveness in handling imperfect input data.
Robustness to tracking method Additionally, we tested using point tracks extracted by TAPIR ||] as input at inference, instead of the CoTTracker point tracks [20]. This evaluates our method's ability to generalize a different tracking method at inference than the one used during training. The results shown in Tab. [5] Although we observed some degradation in accuracy, the results are still good especially after finetunning, demonstrating our method's robustness to different point trackers.
Synthetic Data Evaluation We evaluation multiple test cases from PointOdyssey [68] and compared them with CasualSAM (64,, which is the most accurate baseline. The results are presented in Tab. 6 We observed that method generalizes well these cases while taking much less time than CasualSAM and maintaining high accuracy.
Comparsion with Marigold We also add new experiments with the Marigold depth estimation method (21). The numbers are presented in Tab. demonstrating that our method is more accurate in terms of depth accuracy.
B Implementation details
B.1 Architecture technical details
For learning high frequencies we map each input coordinate to sinusoidal functions as in BI with L = 12. We use 3 pairs of attention layers, each of frames attention followed by point attention. Each point (after sinusoidal functions embedding) is mapped into 256 with a linear layer. Each attention layer is function of the form F R Px2 - R Nx Px256 (see details above). Each attention uses 16 heads with K, Q, V IR (N P) followed by fully connected network with I
16

---

Table 4: Tracking Error. The table demonstrates our point track error handling and contains four row sections. 1. Ours: results when evaluating the original CoTracker point tracks from the pet test our results using the original CoTracker point tracks with added Gaussian noise with corresponding standard deviation in pixels. 10,20,50% outliers: our results using the original CoTracker point tracks where respectively 10,20,50 percent of the tracks are replaced with uniformly sampled pixels. 4. 10,20,50% occlusions: same the outlier setup, but the outlier points marked occluded by setting 0 (defined in Sec.2). which improves outlier handling. Overall see that method can robustly handle the noisy CoTracker imputs, and can further tolerate significant level of synthetically added noise.
set. 2. o
Abs Rel
1.25 1
1.25"
1 All
8 1.25
f All
ATE J. (mm)
RPF Trans (mno)
RPF Rot (deg)
Dyn.
All
Dyn.
All
Dyn.
Dyn.
Durs
0.11
0.08
0.88
0.92
0.99
0.98
1.00
1,00
8.96
3.79
0.25
0,11
0.08
0.88
0.92
0.99
0.98
1.00
1,00
9.07
3,87
0.24
0.11
0.08
0.88
0.92
0.98
1,00
10.96
5,05
0.30
0.42
10% autliers
0.74
14.45
5.92
0.45
205 ourliers
0.35
0.23
0.62
0.65
0.82
0.85
0.90
0.94
15.26
5.96
0.46
50% autlies
0.61
0.32
0.36
0.51
0.60
0.75
0.78
0.88
19.68
11.41
0.69
10% acclusions
0.22
0.16
0.80
0.80
0.92
0.93
0.95
0.97
11.68
4.58
0.30
2055 ocelusians
0.30
0.20
0.70
0.71
0.85
0.89
0.92
0.95
13.15
5.04
0.35
50% acclusions
Table 5: TAPIR point tracks. Comparison of using our method with TAPIR [ point tracks as input at inference time versus using CoTracker (20), which used for training method, on the pet test As be seen, our method robust to the point tracks obtained by TAPIR, and the results are further improved when applying test time finetuning (FT).
Abs Rel J Dyn.
8 <
1:25 1
8 < 1.25
8 <
1.25
ATE
RPE Trans (mm)
RPE Rei (deg)
Dyn
Dyn.
All
Dyn.
All
Ours (C) CoTracker
0.11
0.08
0.55
0.92
0.99
0.98
1,00
1.00
8.96
3.79
0.23
(C) TAPIR
0.34
0.83
0.96
0.96
0.90
0.99
Ours (C) TAPIR FT
0.09
0.06
0.58
0.91
0.99
0.99
1.00
0.99
6.66
3.88
0.22
hidden layer of 2048 features. We then average over the rows to get per-point features Po E R Px256 and over the columns to get per-frame features Fo E RR x236 Finally, we map Po to per-point outputs P1 E IR. (3K++11 ( K basis points and 7) with a linear layer, and Fo into per-camera outputs Fi E (6++++++)) (6 for the rotation parameters (70), 3 for the camera center, and K linear coefficients) using convolutional layer with a kernel size of 31.
B.2 Training details
In total, we used 733 cat videos and 753 dog videos for training. We trained our networks for 7000 and 3500 epochs for the single-class and multi-class setups respectively. Training method lasts about one week on a single Tesla V100 GPU with 32GB memory. We used the Adam optimizer 23 with learning rate of 10 Our method assumes known camera internal parameters which are provided by the dataset and used to normalize the point tracks as a preprocessing step.
B.3 Licenses and links for existing assets
http::/ggitabb.coo///ccooreeeeet tracker. is released under Creative Commons Curibution-Nonoommnnn 4.0 International Public License
CoTracker 20 The code is available here:
Common pets dataset 44
The data is available here: https///gthha.com//cccooossssres
is released under Creative Commons Attibutioon-NonCommeroe 4.0 International Public License.
Nvidia Dynamic Scenes Dataset
62
The data is available here:
It is released under the CC. BY-NC-ND License.
17

---

Table 6: PointOdyssey. Average results on 9 test samples from the PointOdyssey dataset [68]. Metries are the in the main paper. Note that since the input point tracks are provided by the dataset, running times are computed without considering point track extraction time. As can seen, our model trained only on cats, can efficiently and accurately handle sequences from the PointOdyssey dataset.
Abs Rel
6
1.25
5
1.25?
5 <<
1.25
ATE L
RPE Trans
RPE Rot (deg)
Time J (minuses)
Dyn.
All
Dyn.
All
Dyn.
All
Dyn.
All
Ours (C)
0.99
0.09
0.91
0.92
1.00
1.00
1.00
1.00
0.05
0.02
0.18
0,003
Our
0.007
CasulSAM
0.12
0.11
0.85
0.86
0.98
0.99
0.99
1.00
0.02
0.01
0.03
62
Table 7: Comparisons with Marigold. Comparison with the Marigold depth prediction method 21 on the pet test set. As can be seen, our depth accuracy is much higher.
Abs L.
1.25
<.
1,252
1.25
Dye.
Dyn.
Dyn.
All
Dyn.
All
0.88
0.98
1,00
1.00
Marigodd
0.48
0.31
0.28
0.51
0.50
0.75
0.64
0.36
B.4 Other implementation details
Point tracks sampling For building M E N we use the implementation of [20 We sample a uniform grid of 15 15 2D points, starting from frame number 0, 20, 40...., and then track these points throughout the entire video (backward and forward). In Tab. 8 we show the effect of reducing
the grid size at inference time, in terms of camera pose accuracy and running time. During training, at each iteration, we randomly sample 20-50 frames from the training videos and 100 point tracks, i.e. 20 < N < 50 and P = 100. When sampling cameras and point tracks of size N x P x 3 from a larger tensor of size N' x P' x we only take point track if its starting tracking time is in the range It N t + 3.V where t is the first sampled index. At inference time we take all the available point 2
tracks. In both, training and inference time, keep only point tracks that are observed in more than 10 frames.
Finetuning details For our fine-tuning (FT) in the main paper, applied per-sequence fine-tunning of 500,100 100 iterations starting from our final checkpoint, for pets, pets, out-of-doomiin data respectively. The fine-tuning is done as post-processing by minimizing the original loss function the given test video.
Test Set We used the RGBD camera of the iPhone 1l to record our 21 test videos of dogs and cats. Each frame has a resolution of 640 x 480 pixels. Note that the training set contained various types of resolutions. All pet owners who were photographed gave their permission for the animals to be photographed. For evaluation only, we define point track as dynamic if its associated GT mask value is for at least 40 frames. The GT masks are obtained by running 77] and searching for labels of dogs and cats. They were only used for evaluation and not used by our method at all. We verified that this data includes enough dynamic motion, by also including several videos that COLMAP failed to reconstruct without the masks. We further verified manually that the camera trajectories look reasonable.
The out-of-domain dataset contains video sequences with 24 frames, each of resolution of 546 x 288. The GT dynamic masks are provided by the dataset, For this dataset, for evaluation only, we define point track as dynamic if its associated GT mask value is 1 for at least 15 frames.
Bundle Adjustment After inference, as optional refinement, we take the output static approximation B1 E I Px3 and the output camera poses (RI..... Ry }. Riih...,,,,, (BA). We use a 3D world point from B, if its associated y is below 0.008 for the pets dataset and 0.005 for out-of-domain dataset. We optimize reprojection errors of a given observation Mi,, only if it is observed, i.e. M j = 1 and if the initial reprojection error below 10 pixels. We use the BA implementation provided by (32), which is based on the Ceres package p
(t1..... tx and apply Bundle Adjustment
Running times All inference running times were computed on a machine with NVIDIA RTX A6000 GPU and Intel(R) Core(TM) 17-9800X 3. 3.80GHz CPU, Extracting point tracks with 20 took
18

---

Grid
ATE 4
RPE Trans $
RPE Rot (deg)
Inference +
size
(mm)
(mm)
Time
Ours (C)
15
8.96
3.79
0.23
0.16(+8.6) seconds
Ours +C)
15
4.22
2.86
0.17
0.40(+8.6) seconds
Ours (C)
12
9.18
3.81
0.24
0.09(47.8) seconds
Ours (C)+BA
12
4.36
2,97
0.17
0.24(+7.8) seconds
Ours (C)
10
8.91
3,91
0.23
0,05(47.7) seconds
Ours (C)+BA
10
4.44
3.01
0.18
0.16(+7.7) seconds
Ours (C)
7
9.06
4.11
0.25
0.02(47.6) seconds
Ours +C)+BA
4.93
3.52
0.20
0.08(47.6) seconds
Ours (C)
5
10.29
4.97
0.31
0.01(47.6) seconds
Ours (C)+BA
5
8.08
6.33
0.38
0.05(+7.6) seconds
Table 8: Tracking Grid Size Effect Quantitative evaluation of the effect of reducing the number of sampled point tracks at inference time on our model that was trained on cats (C). We measure the camera pose accuracy and the running time. We also mention the point tracks extraction time in parenthesis (c.g. +8.6 seconds) which is performed by (20) as preprocess. As can be seen, our method can handle smaller number of points but the accuracy slightly drops with fewer sampled points
8.6 and 2.5 seconds for each video on the pet-videos and out-of-domain videos respectively and included in the running time tables as part of our method inference time.
Training technical details In training setups, we used: RRepojeection 50.0, Astaic 1.0. ANegative 1.0, ASparse = 0.001. At the beginning of the training, pre-train the camera poses to be located behind and facing the origin. This prevents cases which the cameras are located in the middle of the initial point cloud s.t. many points have negative depths, which may result in bad convergence. More specifically, the pre-train loss || -
Preetrain
100 ||
[0,0, 0, 15]
The pretrain runs until convergence (Lrretriin < 10-4). During the main training, we detach gradients from B, and (R1, t1) ... (Rx, ty) for RReproject stabilize the training. Until epoch 50 sample sequences of in the range [20, 22), and then we increase the range to [20, 50].
C Broader Impacts
Our work focuses on mapping video data dynamic 3D point clouds and camera motion. Our network efficiently takes this video input and generates dynamic point cloud representations that model the evolving 3D structure across the video frames. This could improve applications in domains where fast understanding of spatiotemporal evolution of 3D structure from video data is valuable, such as motion capture, robotic perception, augmented reality, and dynamic scene reconstruction. As robotic capabilities advance, could fundamentally reshape how society perceives and interacts with the domains that become automated by robots, as well as the individuals previously employed in those domains being replaced.
set of points in space, enabling analysis of geometry changes
RCVD 25
MiDaS 3
CasunSAMMf3
Ours (CAD)
Ours (C&DD) FT
Our
Our (C)
0.12
0.05
0.85
0.06
All
Seg0
< 1.25 f
All
1yn
0.71
0094
0.74
0.96
E.00
(139
0.98
0.57
0.95
0.98
1139
0.0
0.97
< 1.25"
1.000
1,00
< 1.233
Dyn
1.00
1,00
1.00
1.00
1.00
1.00
1,00
All
0.89
0.91
0.97
1,00
1,00
1.00
1.00
0.16
0.1.
0.11
All
0.18
0.09
0.09
0.07
Sey!
<. 1.25 f
All
Pyn
0.68
0.45
0.75
0.76
0.31
0.92
0.81
0.50
0.35
0.92
0.94
1.900
0.37
0.92
< 1.25"
0.95
1.25
15m
0000
1.00
1,00
1,00
1.000
1.00
1,00
All
1.00
0.99
1.00
1.00
1.00
1,00
1.00
19

---

Abs Relp
1yn
0.54
0.10
0.96
0.06
0.035
0,03
0.04
All
0.20
0.55
0.06
0.00
0.06
0.06
0.06
0.20
0.94
0.39
0.99
0.99
0,99
0.99
Seq2
<. 1:25
All
All
Dyn
0.89
0.52
8.6
0.35
1,00
0.99
0.99
0.99
1.00
0.99
1.00
0.99
0,57
1,00
1.00
0.99
< 1.25
1,00
1,00
1.00
All
Abs Relp
15n
0.79
0:25
0.07
0.15
0.05
0.07
6005
All
0.22
0.24
0.06
0.09
0.06
0.08
0.06
0.10
0.53
0.97
0.95
0.99
0.97
0.98
Sog3
< 1.25
All
All
Dyn
0.85
0.27
0:74
0:92
0:99
1.00
0.99
0.98
1.00
0.99
1.00
0.98
1.00
0.97
1.00
0.99
< 1.25
1,00
1,00
1.00
All
0:95
Abs Rell
1yyn
0.31
0.08
0.116
0.22
0.15
0.19
05:15
All
0.17
0.27
0.09
0.12
0.11
0.10
0.31
0.44
0.98
1.00
0.05
0.777
Soq4
< 1.25
All
All
Dyn
0.97
037
1,00
0.65
0:50
1.00
0.98
0.35
0.98
0.99
1.00
0.99
1.00
0.92
0.99
1,00
< 1.25
1,00
1,00
1.00
All
0.94
0.99
1,00
Abs Rell
Pyn
0009
0.05
0.115
0.07
0.07
0.05
0107
All
0.12
4.75
0,113
0.08
0.05
0.06
0.04
0.98
1.00
0.98
5og5
All
0.86
0.96
0.98
<C 1.25"
1yn
1.00
1.00
1,00
1.00
1,00
1.00
1,00
All
0.97
0:95
1,00
0.90
1,00
0.98
1.00
< 1.25
1,00
1,00
All
0:98
1.00
1,00
Abs Rell
byn
0.35
0.10
0.34
0.03
0.83
0.04
6003
All
0.14
0:15
0.05
0.04
0.05
0.05
0 OS
0.34
0.99
0.99
Sca5
All
0.83
0.87
0.96
0.96
0.97
1.96
<c 1:25"
Pyn
0063
0.99
0.39
1,00
1,00
1.00
1,000
All
0.91
0:95
0.99
0.99
0.98
0.99
0.588
< 1.25
1,00
0.99
0:98
0.99
1,00
0.99
Abs Rell
Dyn
0.39
0.00
0,09
0.15
0.11
0.12
0.10
All
0.22
0.17
0.06
0.10
0.06
0.09
0.06
00005
Seq7
<
All
0.65
0.80
0.97
0.96
0.91
0.97
<c 1:25?
Pynn
0068
1.00
0.30
0.00
0.05
0.0%
0.08
All
0.86
0:92
0.99
0.58
0.99
0.98
0.99
< 1
1,00
0.97
0.98
1.00
0.99
Abs Rell
All
1ynn
0.41
0.20
3.805404
D.25
0.115
0,03
0.07
0.08
0.04
0.05
0.07
0.00
0.04
600S
Seqs
f
All
0.72
0.64
0.99
0.89
0.97
0.99
< 1:25?
Dyn
0.69
0.97
1,00
1.00
1,800
1.00
1.00
All
0.91
0.84
1,00
0.99
1.00
1.00
1.00
<t 1.25
1,000
All
0.99
0.92
1.00
1.00
1.00
1.00
Abs Rell
All
Pyn
0386
0.33
0.25
0.20
0.1%
0.09
0.22
0.21
0.17
0.09
0.22
0.22
0.11
0.09
Seqo
< f
All
0.65
0.74
0.92
0.55
0.88
0.52
0.87
1.25?
Dyn
0.28
0.85
0.95
0.91
1,000
0.85
1.00
All
0.78
0.91
0.98
0.89
0.99
0.86
0. 98
< 1.25
1,000
All
0.90
0.98
1.00
0.97
1.00
0.97
0.99
Abs Rel]
All
byn
008
0106
0.24
0.072
0.03
0.00
0002
0.02
0.02
0.04
0.002
0.02
0.02
1,00
Seqio
< 1.25 f
All
0.92
0.73
1.00
1.00
1.00
0.99
1.00
< 1.25?
Dyn
1.00
L.00
1.00
1.00
1,000
1.00
1.00
All
0.99
0.67
1.00
1.00
1.00
1.00
1.00
< 1.255
All
1.00
0.95
1.00
1.00
1.00
1,00
1.00
Abs Rel]
All
Dyn
0.17
0.34
0.14
0.38
0.06
0.05
0.10
0.07
0.05
0.07
0.10
0.07
0.05
0.79
Seqll
< 1.25 f
All
0.74
0.67
0.97
0.50
0.96
0.94
0.96
1.252
All
Dyn
0.94
0.86
1.00
0.76
0.599
0.99
0.99
0.96
0.99
0.95
0.99
0.97
0.99
0.98
<t 1.25
1.00
All
0.99
0.86
1.00
1.00
1.00
1,00
1.00
Abs Rell]
Dyn
0.37
0.16
0,05
0.10
0.05
0.00
6.04
All
0.13
0.05
0.08
0.07
Seq12
< 1.25 t
All
0.73
0.79
0.85
0.98
0.94
0.95
0.97
0.94
< 1.225
All
Dyn
0.98
0.97
0.98
1.00
1,00
1,00
1,00
0.98
1,00
0.99
1.00
0.98
1.00
<t 1.25
1.000
All
1,00
1,00
1,00
1.00
1.00
1,00
1.00
Abs Rell]
All
Dyn
3,518508
0.31
0.29
0.35
0.09
0.06
0.15
0.11
0.09
0.14
0.07
0.11
<: 1.25
Dyn
Sog13
20

---

< 1.253
0.84
1,80
All
0.81
0.87
0.99
0.98
0.99
0.99
0.99
1.25
Dyn
1,00
1,00
1,00
1,80
1,00
1,00
1.00
All
0.9
Abs Rell
All
0.18
0.34
0.03
0.04
0.03
0.03
0.63
1:25
15p
0.79
0.72
11.99
0.99
0.99
1..99
0,99
Seql4
< 1.253
All
0.97
0.55
1.00
All
0.94
0.83
1,00
1.00
1.00
1.00
1,00
1.25
1yn
0.99
1.00
1,00
1,000
1,00
1,00
1.00
All
1,00
Abs Rell
All
0.33
0.39
0.09
0.15
0.09
0.18
0.09
1:25
15y
0101
0.77
0.3%
01202
0.38
01.79
0.39
Seg15
All
< 1.263
1200
0139
All
0.81
0.83
0.99
0.97
0.99
0.92
0.99
1.25
0.46
1.00
0.399
1.00
1,000
1,000
1,00
Abs Kell
All
0.21
0.21
0.05
0.06
0.06
0.06
0.066
1:25
0.42
0.92
0,98
0.81
0.85
0.80
0.500
Seg1ó
< 1.26
1200
All
0.95
0.93
1,00
1.00
1.00
1.00
1.00
1.25
1.00
1.00
T,00
1300
1,00
1,000
1.000
Abs Kell.
All
0.23
0.12
0.15
0.08
0.14
0.07
0.06
0.07
0.07
1.25
0.40
0.85
0.76
0.93
0.89
0.91
0.35
0.81
Seq17
< 1.25"
0.99
033
0138
All
0.87
0.94
0.98
0.98
0.99
0.99
0.99
1,25
0391
1.00
T,00
01900
0.99
0.39
(1390)
Abs Rell
All
0.14
0.48
0.21
0.32
0,05
0.10
0.12
0.10
0.09
0.08
0.11
0007
0.09
1.25
0.46
0,80
0.95
0.85
0.96
Seg18
< 1:25?
0.95
0.9%
0039
All
0.91
0.84
0.99
0.98
0.98
0.98
0.98
1.25
0.84
1.00
0.30
1.00
1,00
1,00
1.00
All
Abs Rell
Dyn
All
0.15
0.33
0.14
0.27
0,03
0.05
0.03
0.04
0.05
0.03
0.05
0.05
0003
0,83
0.28
0.80
0.97
0.98
0.99
0.74
Seq19
<t 1:25"
0.97
All
0.98
0.87
1.00
1.00
1,00
1,00
1.00
1.25
1.00
0,90
1,00
1,000
1,000
1,000
1.00
All
Abs Rell
All
Dyn
2,42E-08
0.33
0.27
0.45
0.21
0.04
0.27
0.05
0.05
0.29
0.05
0.25
0.30
0.05
0,34
0.49
0.61
0.49
0.34
Sog20
<t 1:25?
0.93
0.30
1,000
1,000
1.00
All
0.77
0.72
1.00
1,00
1,00
1.00
1.00
1.25
0.98
0,99
1,00
1,000
1,00
1,00
1.00
All
Abs Rell
All
Dyn
3.63E-07
0.40
6. 16E+00
0.16
0.09
0.06
0.12
0.08
0.06
0.09
0.08
0.11
0.09
0.06
0.78
0.93
0.50
< 1.25?
Dyn
0.75
0.97
0.99
1.00
0.99
1.00
All
0.90
0.88
0.99
0.58
0.99
0.98
0.99
1.25
1,00
1,00
1,00
1.00
All
Abs Rell
All
Dyn
9.39E-07
0.27
2.01E+05
0:07
0.05
0.02
0.04
0.08
0.02
0.05
0.04
0.07
0.06
0.02
0.17
0.16
0.T6
STD
<: 1.25?
Dyn
0.26
0:05
0.02
0.00
0.01
0.03
0.01
All 0.07
0:17
0.01
0.00
0.01
0.015
0.01
1.225
0,00
0.00
0.00
0.00
Tuble Depth accuracy, pet test We show comparison to previous metieds on predicted deyth point trajectories compored to their GT deptts. We compare 4 ways running our method. Oas (C&D)) Oua (C): Our
inference time outputs for model that trained on cats and dogs or only respectively. Ours (C&D) FT, Ours aututs
trained video. can seeo, fine-anning improves our accuracy even more.
resportively, fine-tuning
specific
21

---

DROID-ST
ParticteSMM
RCVD X3
CasuiIAM
Ours (C)+A
Our
(C)
(C)+T
ATH(mm)
3.71
6.10
64.67
5.36
5.60
5.43
4.42
Secp
RPE L1mm)
3,05
3.22
26.92
3.13
3.63
3.04
2,89
Seq1
RPL: I.Cmm))
1.32
1.49
25
3.00
2.65
1:72
1.33
RPE (deg-)
0.18
0.23
2,43
0.67
0.29
0.19
0.16
ATErmm)
Sng?
Trim))
3,62
1,97
2.49
RPE (deg
0.08
0.21
1.89
0.06
0.12
0.09
0.00
ATHrmm)
5.15
1.26
29.55
2.49
7.83
2.66
2.92
Sog3
(dog.)
0.08
ATELmmm)
2.59
4.38
56.16
2.65
4.21
2:38
2.25
Sog4
RPF. (mmm)
2,05
2.07
21.36
1.74
2.40
2,08
2.04
ATEimm)
1.07
0.53
Sog5
RPF. Trimm)
1.02
0.74
6.51
0.52
0.88
0.68
0.68
RPE (deg.)
D409
0.07
1.63
0.15
0:12
0.07
0.06
Soa6
Trimm)
10.57
10.31
RPE (dog)))
0.66
0.67
4.03
0.62
0.69
0.69
0.68
ATE)mm))
2.25
25,48
47.23
1.78
4.33
2.39
2:25
Soq7
2.34
23.81
RPE R (deg.
0.15
0.59
2.29
0.11
0.20
0.16
0.16
ATE1mm)
1223
35.79
44.06
2.00
4.78
0.84
0.30
Soe8
RPF ((mm)
1.07
50.57
25.46
1.24
1.54
0.71
0.77
21.74
34.95
Soep
RPE: T(mmm)
9.04
7.96
21.35
21.06
6.89
2.77
2.08
RPE (deg.)
0.62
0.47
3,47
0.71
0.37
0.12
0.10
22.30
1.40
SogtD
T(mm))
1.84
1.22
RPE R (deg.)
0.22
0.12
2.18
0.10
0.21
0.12
0.12
ATF)mm)
1.71
3.00
19.10
1,32
3.12
1.66
1:29
Sequ1
T(mm)
(dey.
0.08
0.08
0.06
0.12
0.08
0.07
ATFimm)
1,70
3.44
2032
2311
6.23
3.1
2.54
See12
RPE T(mm)
2.24
2.63
18.50
2.74
4,04
3.02
2.84
1:23
1.49
Seq13
RPE Timm)
1.19
1.40
17.33
1.00
1.72
1.12
1.09
RPE (deg.)
0.13
0.14
2.12
0.12
0.19
6.12
0.12
Seq14
RPE Trmm)
3.60
3,38
9:45
4.07
3.57
3
RPE (dkg.)
0.18
0.19
3.81
0.65
0.26
0.20
0.19
ATErmm)
7,89
61106
36.70
7.30
36.51
4.98
5.41
(deg.)
0.22
0.30
AT(amm)
5.84
5106
36.42
3.69
4.52
4.11
3.81
Seq16
RPE T(mm)
4.53
4.54
20.11
4.02
4.47
4.25
4.15
R.(dee.)
ATErmm)
1.13
34.07
77.91
2.08
6,32
2.31
2.36
Seq17
RPE Timm)
1.15
12.30
39 64
1.18
2.14
1,50
1.55
(dde..)
0.10
2.20
0.08
Seg18
RPE Timm)
4.05
24.78
3,89
4.34
3.62
3.65
RPE (ddce.)
0.28
0.29
1.67
0.21
0.24
0.21
0.21
ATE)mm)
1.45
2223
42.95
L.82
8.51
2.79
2.36
Seg19
E(mm)
RPE R. (die).))
0.11
0.11
2.09
0.09
0.25
0.16
0.15
ATEimm)
8.13
4.3
66.00
1.95
4,30
3.43
4.06
1.mm)
6.20
27:34
3,00
3.06
R.dde.))
ATErmm)
5.08
12.79
43.95
6,90
8.96
4.22
4,00
Moun
RPE (mmm)
3.60
6.55
25.77
3.95
3.79
2.86
2.74
RPE
0.20
2,31
0.22
STD
RPE 1.(m)) RPE (dog.))
2,80
1038
11.30
4.74
2.52
2.22
2.21
0.16
1.01
0.76
0.22
0.13
0.13
0.13
Talle 10: Cameraa accurncy for pess. Wo Show comparison to prosious mathods the prodicted cameem
We
ways cunsing
inference cutputs
0.16
a model that was trained unly cals. Ours (CHBA: Our inference (reeal infermnee time of seconds) for model that was trained only on cats. Churs (C) FT: The ourpuns the model that was trained culy cats ofter fine-tuning losses for each specific videa (total ninning time abour minutes), As coa sern, mone.
cutputs, fellowed by Bundle Adjjuttent
compared the methods, fine-tuning impeoves accursey
RCVD (5)
MiDaS
CasalSAM (SSSMMGG
Ours (C&D)
Our: (C&D)
022
Als Rell
0.14
0.34
0.02
0.06
0.01
Ballaon1
8 1.25
Dya
All
0.62
0.12
0.72
0.39
0.99
0.96
0.98
0.99
0.99
0.56
1.252
22

---

1,00
1.00
1,00
1,00
5 < 1:25
Dye
120
1.00
1.30
1,00
0.88
1.00
1,00
1.00
Abs Relj
Dva
0.30
2.211:005
0.04
0.04
0,03
0.14
4.878405
Ballom2
< 1:25
Dya
099
0.3
0.76
1.00
0.97
1.00
1.25
Dya
1.00
039
1300
1.00
1,00
1,00
1.00
6 < 1.26
Dye
120
0.9
1.00
1.00
1.00
0.93
1.00
1.00
1.00
Abs Rell
Dya
0.14
4.75F404
0.55
0.01
0:15
0.01
DynamicFaze
< 1:25
Dya
038
0.67
1.00
1.00
1.00
1:25
Dye
1200
004
1380
0.35
1.00
1,00
1.00
6 < 1.26
Dva
120
1.00
1.00
0.86
1.00
1.00
1.00
Abs Rell
Dya
0.17
0.41
0315
0:07
0.07
Jumping
5 < 1:26
Dye
0.36
0.22
0.99
039
0.97
0.97
1:252
Dyn
1200
0.30
1300
1.00
0.39
1.00
1,00
1.00
6 < 1.26
Dya
120
1.00
1.00
All
1.00
0.64
1.00
1.00
1.00
Abs Kel)
Dya
0.33
0.32
638
0.16
0.166
Playground
8 < 1:26
1:25?
Dyn
Dyn
0.61
0.44
0.36
0:49
0.72
0.59
0.99
0.96
0.64
0.96
0.78
034
0.94
8 < 1.26
Dyn
0.58
All
0.83
0.87
0.99
0.98
0.98
Abs Kel)
Dyn
0.16
033
0113
0.12
6.10
Skating
< 1:26
1.25
Dya
Dyn
0.93
0.39
1.00
0.39
0.95
0.79
0.97
0.76
0.99
0.33
0.97
1.00
0.3
0.97
0.99
1.00
1.00
8 < 1:25"
Dyn
120
037
1.00
All
1.00
0.53
1.00
1,00
1.00
Abs Kelj
Dyn
0.33
013
0333
0.14
0.08
8 < 1.26
Dyn
All
0.26
0.50
0.81
0.71
1.00
0.95
0.79
1.00
1.00
Truck
1.25
Dva
0.99
1.00
1,00
1.00
8 < 1:25
Dyn
120
1.00
1.00
All
1.00
0.99
1.00
1.00
1.00
Abs Rolj
Dyn
0.08
0.51
0.02
0.03
6.02
Univella
< 1:25
Dya
All
0.91
0.84
0.87
0.68
1.00
1.00
0.99
1.00
1.00
1.00
1:252
1.00
1.00
1.00
1.00
5 < 1.25
Dyn
1.00
0.2
1.000
1,000
1.000
All
1.00
0.72
1.00
1.00
1.00
Abs Rolj
Dya
1.585+05
0.19
2,765-4M
2.705-05
0,15
0.10
0.06
Moan
8 1:25
Dva
All
0.75
0.09
0.58
0.59
0.95
0.99
0.94
0.93
0.97
0.99
1.25
0.5
0.99
0.99
0.9
5 < 1.25
Dyn
0.66
0.33
1.000
1,000
0.59
All
0.9
0.80
1.00
1,00
1.00
Abs Rell
Dva
7,280+05
0.30
7.81E-M4
0.05
0.05
0.05
STD
8 1:25
1.25
Dva
All
0.14
0.20
0.29
0.21
0.37
0.01
0.08
0.01
0.10
0.12
0.02
0.04
0.30
8 < 1.25
Dyn
0.12
0.28
0.00
0.01
0.02
All
0.06
0.16
0.00
0.01
0.01
Talke 11: Depth accuracy for out-of domaann dnta (2) We show compasioon previous metlidss on the predicted depth point trajeciories compuned thair depths. We compare ways running method, (C&D): Our inference time ontputs mindel was trained trained on cats and dogs, after fine-raning our losses for each specifie video, As can be seen, fiae-tuning improves our accuracy
dogs. Ours (C&D) The ompuus our
DROID-SL AM AX
PuttclaSMM M
RCVD 3
CasmiSAM :
Ours
Currs
Ours
(C)BA
(CHFT
AT1[mm)
2147
Balloon!
RPF T.(mm)
4.5%
6.51
2,55402
4.96
27.73
6.76
6.6
RPE R.ddeg))
0.85
0.08
335
0.55
0.40
6477
0.77
ATE(mm)
3.3540
23

---

3.96300
R.dee
3.21
ATE[mm)
2,50
9.71
LTERRE
3.59
32.79
4.11
3.73
DysamicFace
RPE Limm)
1.71
7.93
2,43/0Z
240
48.39
3.20
3.16
Jumping
RPL: V.tmm))
10:25
11.27
2.81:4022
8.09
36.34
11.35
12.13
RPE R.
0,85
0.06
3,09
1.05
0..11
0:07
0.08
ATELman)
7/2
Playgoond
(mmm)
7.68
RPE R.ideg
0.10
0.75
4.60
0.11
0.40
0.10
0.10
ATEI min)
7.21
19.37
78.24
1.28
27.57
9.21
8,98
Skating
T.(mm)
3.255402
(deg
3.91
ATEI min)
1.1E90Z
17.70
42.47
19:29
17.53
Trick
RPF: (mmm)
31.68
3,6F407
30.24
69.22
34.61
28,37
Univella
RPR: (mmm)
8.11
12.08
3.7E402
7.01
39.85
6.98
8.03
RPE R.(deg.
0.04
0.05
3.05
0,03
0.20
0.03
0.83
[enm)
RPE R.ideg
0.07
0.201
3.43
0.06
0.45
0.08
0.08
AIT((mm)
6.25
28.13
T,0ER0
4.25
X.11
4.89
4.30
R.(deg
0.61
out-of-dummin domaan data (2). We show compurison to previous methods on the predicted camera poses. We compure 3 ways of rumning our method. Ours (C): Our inference time ontguts. Ours (CHBA: Our inference followed by Bundle Adjustmntt model that trained specific
15ble 12: Camera poses accuracy for out-of
Ours
outputs
trained only
tuning
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope and are demonstrated in our experiments (Section 3).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes].
25

---

Justification:
The paper discusses the limitations of the work in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All our architecture details and experimental setup provided in Sections 2,3 respectively. Appendix B provides supplementary technical details. This allows repro- ducibility.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the code once the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwt.:/oubbbiiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https CodeeSumiiii www.iisssooooioii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
27

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The training and test details are provided in Section 3, and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The standard deviation numbers for the different experiments are provided in the supplementary tables of Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Inference times are provided in the tables of Section 3. Training times and hardware details are provided in appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We verified that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts discussed in Appendix C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
29

---

We do not see such risks in our paper, that models geometry from video sequences.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the original papers that produce the code packages and datasets. We include the URLs and licenses in Appendix B.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprssii thwodeetehcceeeeooo com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets for now.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
30

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. We only experiments human datasets that have been collected and released by other papers.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. We only experiments human datasets that have been collected and released by other papers.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "5ktehJneYl", "reviewer": "Reviewer_fqXP", "summary": "This paper presents TracksTo4D, a feed-forward approach for estimating 3D structure and camera poses from 2D point tracks. Authors propose a novel architecture that directly processes 2D point tracks, takes into account symmetries present in the data, and assumes movement patterns can be represented by a low-rank approximation. Notably, this approach achieves similar performance to state-of-the-art methods with significantly reduced runtime. Despite training on videos of animals, TracksTo4D generalizes to unseen videos and unseen semantic categories at inference time.", "questions": "Evaluating Robustness to 2D Point Tracking Noise. Although authors use CoTracker in this work, it would be interesting to characterize how this method works when evaluated using a different point tracker at test-time. \n\nEvaluating Robustness to Speed of Dynamic Motion. Although authors highlight that their method does not perform well given rapid motion, it would be useful to quantify this to benchmark performance for future methods. It would be useful to show off performance on automotive datasets that have ground truth depth (via LiDAR) and precise camera poses. \n\nImpact of Data Scaling. Given the speed and generalizability of this approach, it would be interesting to explore how this method scales with more data, particularly since this method doesn't require labeled 3D data. \n\nDefining Out-of-Distribution. Since this paper takes 2D points tracks (instead of raw images) as input, its I would not consider different datasets and semantic classes to be out-of-distribution, since the 2D point tracks don't have any notion of semantics. Instead, it would be useful to benchmark on data with different speed profiles, and provide breakdown analysis by speed buckets.\n\nContextualizing Runtimes. Since this method runs CoTracker before processing, run times should take this into consideration as well. Although the proposed method is quite fast, I believe the 2D point tracker can be a limiting factor on real-world performance. \n\nSupplemental Baseline Comparisons. Although SpaTracker and DuST3R (CVPR 2024) are considered concurrent work, I think this paper can be further strengthen by comparing against such methods. Note that a lack of comparison to these approaches does not influence my ratings.", "limitations": "Yes, authors highlight that their approach can't handle rapid motion and is limited by the tracking noise imputed by CoTracker.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Problem Motivation. This paper clearly describes the problem they are trying to solve and explicitly outlines the inputs and outputs of their method. Learning to predict camera poses and 3D structure from 2D point tracks is a challenging and relevant problem of interest in the community. Few methods currently solve both subproblems using feed forward approaches. \n\nAblations. Authors extensively ablate different losses used in training their method and highlight the impact of training on different subsets of COP3D (e.g. cat vs. dog vs. cat + dog)\n\nClear Writing and Informative Visuals. The paper is well written and easy to follow. The provided visuals in the supplement clearly demonstrate the effectiveness of the proposed method.", "weaknesses": "Limited Evaluation. Although many methods do not tackle jointly predicting 3D structure and camera poses, authors evaluate methods that support a subset of these tasks. However, these evaluations can be supplemented. MiDaS is quite old at this point, and there has been significant improvements in monocular depth estimation in recent years. Instead, one should evaluate multi-view or video depth estimation methods. Other simple baselines can be included to provide more context like Chained RAFT3D [1] and Lifted CoTracker [2].\n\nSmall Test Set Size. Authors evaluate on a subset of COP3D and the NVIDIA Dynamics dataset, both of which only have a limited number of videos. Instead, authors should consider evaluating on synthetic datasets like PointOddessey [3], which can allow authors to explore other dimensions of their method like data scaling (which is unique to this method because it is much faster than competitive methods). \n\n[1] Teed et. al. Raft3D: Scene Flow using Rigid Motion Embeddings. \n[2] Karaev et. al. CoTracker: It Is Better to Track Together.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive feedback on our paper's relevance, novelty, and clarity. We will now address your specific questions and suggestions raised by the reviewer. \n\n\n**Q**: Limited Evaluation,  MiDaS is quite old. Compare to video depth estimation methods. \n\n**A**: Thank you for your feedback. We acknowledge that MiDaS has been around for some time. However, we'd like to clarify that our comparisons were made using MiDaS 3.1, specifically the dpt_beit_large_512 version (Birkl et al. 2023). This is an improved version of MiDaS that utilizes the DPT architecture, representing a more current state of the method. \nTo strengthen this point we also add new experiments with the Marigold depth estimation method (CVPR 2024). Our experiments show that our model achieves higher accuracy compared to both MiDaS 3.1 and Marigold. We've included a table with additional comparison to Marigold in the attached PDF (Table 3) for your reference. To clarify, as the reviewer asked, we performed all these comparisons using lifted CoTracker tracks. We believe these additional evaluations address the concern about limited comparisons and demonstrate our method's effectiveness.\n\n\nRegarding video depth estimation methods, our submission already includes comparisons with CasualSAM and RobustCVD. It's worth noting that while RAFT3D infers 3D scene flow from two depth images, our setup assumes RGB frames as input without depth information, making a direct comparison challenging.\n\n\n**Q**:  Small Test Set Size. Run evaluation on  PointOddessey . \n\n**A**: Following the reviewer's request, we ran an evaluation on multiple test cases from PointOdyssey and compared them with CasualSAM [59], which is the most accurate baseline. The results are presented in the attached PDF (Table 2). We observed that our method generalizes well to these cases while taking much less time than CasualSAM and maintaining high accuracy. We will make an effort to extend the evaluation on PointOdyssey to include more test cases in the final paper. \n\n\n**Q**:  Robustness to 2D Point Tracking Noise. \n\n**A**: We first note that the input point tracks extracted by CoTracker [18] are far from being perfect, and contain noise and outliers. To further evaluate our robustness to noise, we added additional Gaussian noise to CoTracker points and measured the final error. We observed that our method tolerates a significant level of noise. See Table 1 in the attached PDF.\n\n\n**Q**: Using a different point tracker at test time.\n\n**A**: We followed the reviewer's suggestion and tested point tracks extracted by TAPIR [10]. Although we observed some degradation in accuracy, the results are still good especially after finetunning, demonstrating our method's robustness to different point trackers. We added these results to the attached PDF (Table 4) and will add them to the revised paper as well. \n\n\n**Q**:  Robustness to Speed of Dynamic Motion \n\n**A**: To clarify, our limitation to the speed of dynamic motion comes from CoTracker's [18] limitation to track fast motion. In our evaluation set, we did not see such a limitation including in the Nvidia Dynamic Scenes Dataset [58] which contains a variety of motion characteristics. \n\n\n**Q**:  Automotive datasets that have ground truth depth\n\n**A**: We observed that CoTracker [18] often fails when applied to automotive scenes due to high motion blur, especially on the road. This prevents us from running large-scale evaluations on automotive datasets with the current performance of the tracking methods. We will clarify this in the revised paper.\n\n\n\n**Q**: Training with more data\n\n**A**: We observe that our method, though trained only on pet videos, already generalizes well to scenes from the Nvidia Dynamic Scenes Dataset [58], as shown in the main paper. During the rebuttal period, following the reviewer's request, we also tested our model on a variety of samples from the large-scale PointOdyssey dataset (see Table 2 in the attached PDF). Our results demonstrate that the model already performs well on this dataset without additional training. Due to the dataset's large scale, we were unable to train on it during the rebuttal period. For the revised version, we will make an effort to further train our method on PointOdyssey samples and report the effects of this larger-scale training.\n \n\n\n**Q**: Defining Out-of-Distribution data. Benchmark on data with different speed profiles\n\n**A**:The Nvidia Dynamic Scenes Dataset [58] contains several speed profiles, e.g. 'Skating', 'Truck', and 'Umbrella'. Tables 7 and 8 in the Appendix show per-scene errors and demonstrate the robustness to dynamic motion levels. For example, our depth accuracy on 'Skating' is slightly better than on 'Truck'. We will discuss this in the revised paper. \n\n\n\n**Q**: Contextualizing Runtimes. Tracking run times should be taken into consideration as well.\n\n**A**: Our method's runtime in Tables 1 and 2 already includes the point tracking times. Table 4 in the Appendix shows the separation of tracking time that is done as a preprocess by [18] (8.6 seconds), inference time of our network (0.16 seconds), and Bundle Adjustment time (0.24 seconds). We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zoa1KXnNvx", "reviewer": "Reviewer_89V2", "summary": "This paper proposes a feed-forward network that takes a set of 2D TAP curves as input, outputs the 3D curves as well as extracts the camera rigid SE(3) poses.  This paper exploits the permutation and time shift equivariance when designing the encoding network. To train the model, the main loss is the re-projection loss (similar to BA). Experiments show in-category improvement and novel category generalization.", "questions": "Please see weakness", "limitations": "discussed in last paragraph.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The reviewer really likes the paper's style, exploiting the underlying structure of dynamic scenes, although it seems the symmetries exploited here are simple.\n- The problem this paper is solving is very important and meaningful for the community. Although I do see more promising ways to solve the problem (see weakness), I do like this paper's perspective on this problem.\n- The model learned on cats and dogs is somehow generalizable to new categories, verifying their effectiveness.", "weaknesses": "- I do like the equivariance part, but given the current advances in the community, I do find that maybe 3D-track or exploiting depth models may lead to better performance instead of learning everything from scratch.\n- Solving a single scene efficiently: one important loss that supports the model learning is the reprojection error, which seems to conduct an implicit BA when training the model over many sequences, the reviewer is curious whether the same techniques can work on a single video efficiently. This is to justify the necessity of the feedforward network if the same technique can fit a single video efficiently as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive feedback on our paper's style and its significance to the community. Below we answer specific questions and suggestions. \n\n\n**Q**: Maybe 3D-track or exploiting depth models may lead to better performance instead of learning everything from scratch.\n\n**A**: We explored this suggestion by using MiDaS-inferred depth as additional input to our method and applying depth loss relative to MiDaS. Currently, we haven't seen performance improvements from these approaches. However, we agree that exploiting depth models and 3D-tracking techniques is an interesting direction that is worth further investigation in future work. This could potentially lead to enhanced performance compared to learning everything from scratch.\n\n\n\n**Q**: Solving a single scene efficiently. \n\n**A**: Yes, our model can be efficiently fine-tuned on specific scenes using our self-supervised loss terms. We demonstrated in the original paper, in Tables 1 and 2 (Ours+FT) that this approach further improves our accuracies. Training from scratch on a single scene, however, often failed to converge well after 30 minutes of training for some scenes. In contrast, feed-forward inference with our pre-trained model takes only 0.16 seconds. This significant time difference justifies our choice of a feed-forward modeling approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dLHRiU4kM1", "reviewer": "Reviewer_aqo3", "summary": "The paper introduces a method for fast 3D reconstruction of dynamic structures (or 4D reconstruction) from monocular video. The model is a transformer architecture that takes a set of 2D point tracks as input, and lifts them to 3D. It is learned with re-projection losses on 2d tracks without 3D ground-truth. The method is trained on COP3D (containing cat/dog videos) and evaluated on two test datasets: (1) 21 self-collected videos of dogs and cats and (2) NVIDIA dynamic scene dataset, using depth metrics and camera tracking metrics. It is shown to perform on par with an optimization-based (CasualSAM) but much faster.", "questions": "- Is there any mechanism to handle the errors in co-tracker's 2d trajectory?\n- Spatialtracker [B] is another relevant work, although then train on 3D ground-truth and rely on depth inputs. It would be helpful to discuss what are the pros and cons of using unlabelled/3D data to train the model.\n\n[B] Xiao, Yuxi, et al. \"SpatialTracker: Tracking Any 2D Pixels in 3D Space.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The method tries to tackle a challenging and significant problem, and presents a viable approach that works decently well and very fast. \n- It does not need 3D ground-truth, or depth supervision, and shows good generalization ability to novel scenarios. \n- The authors also collected a new test set that has more accurate depth ground-truth than COP3D.", "weaknesses": "- The presentation can be improved. Specifically, I had difficulty understanding the points made in Sec 2.1. \n  - The permutation invariance of point sets has been discussed in early literature (e.g., pointnet), and it might be better to connect to that instead of starting from symmetry analysis. \n  - I also don't really understand what is the relation of \"linear equivariance\" and cyclic group, and how that relates to the proposed architecture. Is there any cyclic operation that moves the end of a sequence to the beginning?\n  - It seemed to me the final architecture is a standard transformer with some modifications. It would be good to highlight which part enforces translation equivariance.\n\n- Since the method does not require 3D ground-truth. In theory, it can be trained using any video. However, the model is only trained on cats and dogs in practice. I'd like to encourage the authors to train it on larger scale dataset and observe whether there is any performance gain on the held-out set.\n- It uses some hand-crafted shape and motion prior (low-rank). I'd be interested in seeing an analysis of how changing the flexibility of the model (by changing the number of basis) affects the results.\n- The evaluation can be further enhanced by reporting 3D end-point-errors, since depth only evaluates one aspect of 4D reconstruction. Tapvid3d [A] or synthetic datasets like Kubrics, PointOdyssey could provide such ground-truth data.\n\n[A] Koppula, Skanda, et al. \"TAPVid-3D: A Benchmark for Tracking Any Point in 3D.\" arXiv preprint arXiv:2407.05921 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of the efficiency and the good performance of our approach. \n\n\n**Q**: The permutation invariance of point sets has been discussed in early literature… and it might be better to connect…\n\n**A**:  We appreciate the reviewer's suggestion. While our architecture is based on a generalization of PointNet to sets of symmetric elements [Maron et al. 2020, ICML], which we reference multiple times in the paper, we acknowledge that PointNet [Qi et al. 2017] and DeepSets [Zaheer et al. 2017] are indeed seminal works in this area. We agree that explicitly discussing these connections would improve the clarity of our presentation. In the revised version, we will add references to PointNet and DeepSets, and briefly discuss how our approach builds upon and extends these foundational ideas in the context of our specific problem of processing point tracks from videos.\n\n\n**Q**: I also don't really understand what is the relation of \"linear equivariance\" and cyclic group?\n\n**A**:\nIn geometric deep learning, it's customary to first analyze the symmetries of the input data and then derive suitable equivariant layers. Linear equivariant layers, such as convolutions and DeepSets [Zaheer et al., 2017], have proven effective and can be derived relatively easily. This approach is exemplified by convolutional layers for images, which are equivariant to translations, modeled as a product of cyclic groups (which formally correspond to an image on a torus). \n\nIn our case, we have a product of two group actions on the input point tracks:\n\n1. Set symmetries (like in PointNet and DeepSets)\n2. Time-shift symmetry (represented by a cyclic group)\n\nModeling temporal signals using cyclic symmetries is standard in the field, analogous to the image case, and forms the algebraic basis for transforms like the Discrete Fourier Transform (DFT).\n\nWe derive the linear equivariant layer structure using the DSS framework [Maron et al. ICML 2020]. However, we use this only as inspiration and replace convolutions with attention mechanisms and positional encoding in our final non-linear transformer-based architecture. This approach aligns with common practices in geometric deep learning [e.g., Bevilacqua et al. ICLR 2022]. We'll clarify this in the revised version.\n\n\n**Q**: It seemed to me the final architecture is a standard transformer…\n\n**A**: While our method incorporates transformer elements, it differs significantly from a standard transformer in its structure and processing:\nOur architecture operates on a 3D tensor of shape N × P × D (time steps × points × features) and processes it using two alternating operations:\n\n1. Temporal Attention: A transformer with temporal positional encoding operates on each point track (N × D slices).\n\n2. Set Attention: A transformer operates across all points at each time step (P × D slices).\n\nThis alternating row-column processing is inspired by the DSS linear architecture [Maron et al. 2020] but as mentioned above, replaces linear operations with more expressive transformer layers.\n\nRegarding translation equivariance, we acknowledge that our final design is not formally equivariant to cyclic translations. Instead, it provides a strong inductive bias for temporal data through the use of temporal positional encoding. Our ablation studies (Table 3, original paper) demonstrate that this design significantly outperforms the strictly linear equivariant layers.\n\n\n\n**Q**:  3D errors vs depth errors for evaluation. \n\n**A**: Our evaluation focuses on depth metrics for lifted point tracks, as current state-of-the-art baselines addressing our case primarily predict dynamic depth maps. For our model, we also assess pixel reprojection error, which evaluates tracking accuracy in the other two dimensions (see Table 3, main paper). \n\n\n**Q**: Is there any mechanism to handle the errors in co-tracker's 2d trajectory?\n\n**A**: We note that the input point tracks extracted by CoTracker [18] are not perfect and contain noise and outliers. Nevertheless, as demonstrated in our paper, our model shows significant robustness to these errors.\nOur model handles imperfect input through two features:\n\n* Static scene modeling: Ensures that only 2D motion that can be represented by a camera motion and truly static points is modeled statically.  This makes our camera estimation robust to errors while pushing the non-modelled errors to the dynamic part.\n\n* Dynamic scene modeling: Using limited basis elements for dynamic parts inherently resists outliers and extreme anomalies.\n\nTo quantify this robustness, we conducted tests by adding Gaussian and uniform noise to CoTracker points. The results (see Table 1 in the attached PDF) confirm that our method tolerates significant noise levels, further validating its effectiveness in handling imperfect input data. We will include this discussion in the revised version of our paper.\n\n\n**Q**: Spatialtracker [B] is another relevant work. \n\n**A**: SpatialTracker [53] is a concurrent work discussed in our related work section. Our approaches differ significantly: unlike SpatialTracker, which relies on depth estimation inputs or 3D ground-truth supervision, our fully self-supervised method requires only 2D point tracks as input, allowing us to train on a wider range of video data and fine-tune on test cases for improved accuracy (see Tables 1,2, Ours+FT in main paper).\nWe have experimented with MiDaS-inferred depth as additional input but this hasn't improved performance. This remains an area for future investigation. We will add a discussion in the revised version.\n\n**Q**:  Training with more data \n\n**A**: Please see the answer to Reviewer fqXP.\n\n**Q**: Changing the number of basis elements\n\n**A**: Please see the answer to Reviewer 6f3c.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uk601ESFsb", "reviewer": "Reviewer_6f3c", "summary": "The paper presents TRACKSTO4D, a fast, encoder-based method for reconstructing 3D structures and camera positions from casual videos with dynamic content. It processes 2D point tracks using a single feed-forward pass, leveraging inherent symmetries and low-rank approximations. TRACKSTO4D is trained unsupervised, reducing runtime by up to 95% compared to state-of-the-art methods while maintaining accuracy. It generalizes well to unseen videos of different categories, offering an efficient solution for 3D reconstruction in applications like robot navigation and autonomous driving.", "questions": "1. How robust is TRACKSTO4D to variations in the quality of 2D point tracks? For instance, how does it perform when the point tracks contain noise, are partially occluded, or have tracking errors? Does it strictly follow the 2D point tracks, or does it try to rectify them to some extent? Understanding this robustness is crucial for assessing its applicability in real-world scenarios where perfect point tracking is often unattainable.\n2. Can the method be extended to handle dense point clouds or full 3D reconstructions rather than sparse point clouds? If so, what modifications would be necessary, and what impact would this have on runtime and accuracy?", "limitations": "1. The method assumes that the movement patterns of the dynamic 3D points can be effectively represented using a low-rank approximation. This assumption might not hold true for all types of motion, particularly those involving complex deformations or highly non-linear dynamics.\n2. There is an assumption on the motion parallax, without which the method fails to generate accurate camera poses.\n3. The accuracy and robustness of the method are more or less dependent on the quality of the 2D point tracks extracted from the videos. Noisy or incomplete point tracks could adversely impact the 3D reconstruction and camera pose estimation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Method:\n\n1. The method considers the symmetry of tracked points and the temporal sequence of the video. By designing an equivariant layer based on transformers and positional encoding, it effectively takes advantage of these properties.\n2. It also uses a low-rank movement assumption to decompose the motion of the 2D tracked points into the global camera motion and the 3D motion of objects in the scene. This converts an originally ill-posed problem into a solvable and intuitive problem.\n\nThe method incorporates hard constraints into the design, resulting in more efficient training and more accurate and constrained results. Overall, I think the method is very solid.\n\n\n\nResults:\n\n1. TRACKSTO4D significantly reduces the runtime for 3D reconstruction from casual videos by up to 95%, making it highly efficient and practical for real-time applications.\n2. The method generalizes well to unseen videos and different semantic categories, demonstrating robustness and versatility.\n3. The use of unsupervised training without the need for 3D supervision minimizes the dependency on annotated datasets, which are often expensive and time-consuming to create.\n\nThe evaluation results are fairly comprehensive.", "weaknesses": "1. The paper primarily focuses on runtime and accuracy compared to state-of-the-art methods, but it lacks a detailed evaluation on other important aspects like robustness to noise or handling of occlusions. There is also limited discussion on how the method scales with increasing video length, number of objects, or more complex scenes. Scalability is crucial for deploying the method in large-scale real-world environments. In particular, I'm very curious how the chosen $K$ would affect the performance since $K$ is like an indirect descriptor of how complicated the point cloud's components are in my opinion.\n2. The method's performance is heavily dependent on the quality of 2D point tracks extracted from videos. Poor quality or incorrect point tracking can adversely affect the 3D reconstruction accuracy. I'm curious how robust the method is against failing 2D point tracks.\n3. While the low-rank assumption helps in reducing complexity, it may not capture the full variability of highly dynamic or complex scenes, potentially limiting the method’s applicability in such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive assessment of our method, particularly the recognition of our consideration of symmetry. We're pleased that the reviewer finds our approach solid and our evaluation results comprehensive. We'll now address the specific points and questions raised in the review.\n\n\n\n**Q**:  Limited discussion on how the method scales with increasing video length, number of objects, or more complex scenes\n\n**A**: Our model shows good adaptability to various scenarios. For video length, we trained on 20-50 frames and successfully tested on both 24 frames (Nvidia Dynamic Scenes Dataset) and 50 frames (Pet test set), with good performance in both cases. The encoder might be able to handle longer sequences, though we haven't tested this. Regarding scene complexity and object count, the Nvidia Dynamic Scenes Dataset includes a range of scenes, from single-object scenes like \"Skating\" to multi-object scenes like \"Jumping,\" and various motion types such as \"Truck\" and \"Umbrella\". Tables 7 and 8 in the Appendix provide per-scene errors, demonstrating our method's robustness across these diverse scenarios, with Figure 4 offering visual examples. We'll include a more detailed discussion of these aspects in the revised paper.\n\n\n\n**Q**: Changing the number of basis elements\n\n**A**: We explored the impact of varying numbers of basis elements in our ablation study (Table 3, main paper), training models from scratch with different K values. Large K (30) led to increased depth error, indicating insufficient regularization, while small K (2) resulted in high pixel reprojection error, suggesting over-regularization and limited 3D representation of 2D motion. Our chosen K (12) balances depth regularization and accurate pixel representation. We found no significant differences with nearby K values (e.g., 11).\n\n**Q**:  Low-rank assumption helps in reducing complexity, it may not capture the full variability of highly dynamic or complex scenes. \n\n**A**: We found K=12 basis elements effective for our evaluation set, balancing complexity reduction and motion representation. However, we acknowledge this fixed number may not capture all possible scene dynamics. Future work could explore automatically inferring the optimal number of bases per scene. We'll add this point to our limitations and future work section.\n\n\n\n\n**Q**:  Robustness to noise or handling of occlusions.  I'm curious how robust the method is against failing 2D point tracks.\n\n**A**: We first note that the input point tracks extracted by CoTracker [18] are far from perfect, containing noise, outliers, and occlusions. To further evaluate our robustness to tracking errors, we performed three experiments by modifying CoTracker point tracks. First, to measure noise robustness, we added Gaussian noise to the tracks with different noise levels and measured overall accuracy. Second, to assess outlier robustness, we replaced a fraction of point tracks with uniformly sampled x,y points. Lastly, we repeated the second experiment but marked outlier points as occluded by setting o=0 (defined in L97 in the main paper). The results, presented in the attached PDF (Table 1), demonstrate that our method tolerates significant tracking errors and occlusions.\n\n\n\n**Q**: Can the method be extended to handle dense point clouds or full 3D reconstructions rather than sparse point clouds? If so, what modifications would be necessary, and what impact would this have on runtime and accuracy?\n\n**A**: Modeling dense point clouds would be an interesting direction for future work. Currently, our network can handle up to about 1000 point tracks in 50 frames in one inference step when running on an NVIDIA RTX A6000 GPU with 48GB of memory. A possible extension to handle denser point clouds could involve querying point tracks iteratively while maintaining a shared state, but this approach remains to be explored. We will include this discussion in the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their thoughtful feedback and constructive suggestions. We were happy to see that all reviewers gave us positive ratings and appreciated the positive comments on our method's efficiency, performance, and generalization ability highlighted by multiple reviewers. As Reviewer aqo3 noted, our approach "presents a viable approach that works decently well and very fast" and "shows good generalization ability to novel scenarios." Reviewer 6f3c stated that “the method is very solid” and appreciated our consideration of symmetry and the comprehensiveness of our evaluation results.

As suggested by the reviewers, we conducted additional experiments whose results can be found in the PDF. We provide here a short summary:

* Tracking Error (Table 1): We experimented with varying levels of Gaussian and uniform noise to CoTracker points, demonstrating our method's tolerance to significant noise levels (in response to the questions raised by Reviewers 6f3c, fqXP, and aqo3 regarding robustness to noise).

* Synthetic data (Table 2): We evaluated test samples from the large-scale PointOdyssey dataset, showing good generalization (as suggested by Reviewer fqXP). 

* Marigold depth (Table 3). We added a comparison to the Marigold depth estimation method (CVPR 2024), to address reviewer fqXP’s concern, and demonstrated that our method is more accurate in terms of depth accuracy.

* Robustness to point tracking method (Table 4):  As suggested by Reviewer fqXP, we evaluated our method with point tracks extracted by TAPIR [10] rather than the point tracks of CoTracker [18] which were used for training our method.  This demonstrates our generalization to the point tracking method.

Meta-review: ## Summary

TracksTo4D is a feed-forward approach for estimating 3D structure and camera poses from 2D point tracks. It uses a novel architecture that directly processes 2D point tracks, considers symmetries, and assumes movement patterns can be represented by a low-rank approximation. The approach achieves similar performance to state-of-the-art methods with significantly reduced runtime. The method generalizes well to unseen videos and unseen semantic categories at inference time. The method is trained on COP3D and evaluated on two test datasets. TRACKSTO4D is a fast, encoder-based method for reconstructing 3D structures and camera positions from casual videos with dynamic content.

## Strengths
- The paper aims to predict camera poses and 3D structure from 2D point tracks. 
- The method considers the symmetry of tracked points and the temporal sequence of the video.
- It uses a low-rank movement assumption to decompose the motion of the 2D tracked points into the global camera motion and the 3D motion of objects in the scene.
- Few methods currently solve these subproblems using feed forward approaches.
- The authors extensively ablate different losses used in training their method.
- The impact of training on different subsets of COP3D is highlighted.
- The paper is well written and easy to follow
- TRACKSTO4D significantly reduces the runtime for 3D reconstruction from casual videos by up to 95%.

## Weaknesses
- The paper primarily focuses on runtime and accuracy compared to state-of-the-art methods.
- The method's performance is heavily dependent on the quality of 2D point tracks extracted from videos.
- The low-rank assumption may not capture the full variability of highly dynamic or complex scenes.
- Authors evaluate methods supporting a subset of 3D structure and camera poses.
- MiDaS is outdated and monocular depth estimation has improved.
- Multi-view or video depth estimation methods should be evaluated.
- Other baselines like Chained RAFT3D and Lifted CoTracker can provide more context.
- Authors evaluate on a subset of COP3D and the NVIDIA Dynamics dataset, which have a limited number of videos.
- Authors should consider evaluating on synthetic datasets like PointOddessey to explore other dimensions of their method.
## Overall
Based on the discussion, the paper should be accepted once all the points raised by the reviewers are addressed in the final version of the paper
Decision: Accept (poster)